The foundations of inference

Lecture 4

Iain R. Moodie

Friday 27th March, 2026

Sampling distributions

Dairy disease

A large dairy farm (\(N\)=500) has a potential outbreak of a disease. Testing for the disease is costly, so only a sample (\(n\)=50) cows are tested to estimate the population’s prevalence of the disease.

Photo from The General Line (1929)

Sampling distributions

Dairy disease: collect a sample

Sampling distributions

Dairy disease: observe your sample statistic

Calculate the proportion of cows that are diseased in the sample:

Sampling distributions

Dairy disease: report your observed statistics

TODO: Whiteboard picture

Sampling distributions

What is a sampling distribution?

The distribution of observed statistics (e.g. proportion) obtained through repeatedly sampling the population
Shows the range of statistics we can observe through sampling
Its central tendency gives us an idea about where the true population parameter value is
Foundation of confidence intervals (today) and hypothesis testing (Monday)

Sampling distributions

What is a sampling distribution?

Sampling distributions

A problem…

We (usually) collect a single sample!
A sampling distribution by definition requires many samples.

Bootstrap resampling

How to construct a sampling distribution from a single sample

Bootstrap resampling

How to construct a sampling distribution from a single sample

If we assume that the sample is representative of the population
- collected randomly and without bias (yesterday’s lecture)
We can re-sample the sample with replacement to create more “samples”
- Process known as the bootstrap (samples are called “bootstrap samples”)
- From English phrase: “To pull one’s self up by their own bootstraps”
- Seems impossible?

Bootstrap resampling

Re-sample your sample (with replacement)

Bootstrap resampling

In each bootstrap sample, calculate the statistic

Bootstrap resampling

Make a bootstrap sampling distribution

Proportions from 10000 bootstrap samples

Bootstrap resampling

Make a bootstrap sampling distribution

Proportions from 10000 bootstrap samples

Bootstrap resampling

Compare to your observed statistic

Proportions from 10000 bootstrap samples
Observed proportion (proportion of original sample) in red
The observed mean is still our best guess at the true mean
Bootstrap sampling distribution allows us to quantify our uncertainty in the mean

Bootstrap resampling

Extremely flexible

Works for almost any statistic!
- Can be applied to new statistics / unique questions where a dedicated test procedure doesn’t exist
Makes no distributional assumptions

Quantifying sampling error

If we collected another sample of the same size, how much would our test statistic be likely to vary?

Quantifying sampling error

What range of observed statistics is plausible?

If we took another sample, it is unlikely that we would get exactly the same observed statistics
How confident are we in our sample statistic?
We want to quantify this (sampling error)
Problem: we (usually) only ever collect one sample
Solution: generate more samples using bootstrap resampling

Quantifying sampling error

The standard error (SE)

The standard error is the standard deviation of the sampling distribution
Use the bootstrap generated sampling distribution as our sampling distribution
Assumptions: your sampling distribution is approximately normally distributed (bell-curve)
If calculated with a bootstrap sampling distribution often written as \(SE_{boot}\)

Quantifying sampling error

The standard error (SE)

Quantifying sampling error

The confidence interval

“Sloppy” definition: the range where we expect the true population parameter to be
Real definition: If we repeated our experiment many times and calculated a X% CI each time, the X% CI’s would include the “true” population value X% of the time

Quantifying sampling error

The confidence interval

For historical reasons in most of biology, we tend to use 95% confidence intervals
- Nothing special about that number

Many methods to calculate:

Percentile method: Middle X% of the sampling distribution
No assumptions, but requires a large (>30 >>14) sample size to be accurate

Quantifying sampling error

The confidence interval: percentile method

Quantifying sampling error

The confidence interval: testing the definition

If we repeated our experiment many times and calculated a 95% CI each time, the 95% CI’s would include the “true” population value X% of the time
Draw your confidence interval on the whiteboard
TODO: Whiteboard picture

Quantifying sampling error

Confidence interval using a bootstrap sampling distribution

Quantifying sampling error

Reporting confidence intervals

Common method:

Observed statistic (X% CI: lower_ci - upper_ci)
84% (99% CI: 70% - 90%)
84% (99% \(CI_{boot}\): 70% - 90%)

Quantifying sampling error

An example

“Breakthrough Drug Reduces COVID-19 Hospitalizations by 12%!”

A new antiviral drug reduced COVID-19 hospitalizations by 12% in a clinical trial involving 1,000 patients.
A new antiviral drug reduced COVID-19 hospitalizations by 12% (95% CI: -11 to 32%) in a clinical trial involving 1,000 patients.

03:00

Quantifying sampling error

Confidence interval for differences between two independent groups

What is the difference in average size of males and females?
Statistic?

Quantifying sampling error

Confidence interval for differences between two independent groups

Quantifying sampling error

Confidence interval using a bootstrap sampling distribution

Quantifying sampling error

Confidence interval for differences between two independent groups

Quantifying sampling error

Confidence interval using a bootstrap sampling distribution

Quantifying sampling error

Confidence interval for differences between two independent groups

02:00

Quantifying sampling error

Confidence interval using a bootstrap sampling distribution

Quantifying sampling error

Confidence interval for differences between two independent groups

Quantifying sampling error

Key ideas

The distribution of observed statistics obtained through repeatedly sampling the population is called the sampling distribution
Since we often only take one sample, we can generate a sampling distribution using data from our sample via resampling our sample with replacement (bootstrapping)
Confidence intervals (the middle X% of the sampling distribution) answer the question: if we collected another sample of the same size, how much would our test statistic be likely to vary?