The first time I came across the term Confidence Level of 95%, I misunderstood that it means there is a 95% probability that the true parameter falls inside this specific interval. This is a common misconception and it took quite a few attempts to get the concept right. So here’s my attempt at explaining it in simple terms using a chai analogy.
Before I get into the details, let’s understand why we have a Confidence Interval and not a confidence value in statistics. I recently read in Practical Statistics for Data Scientists that humans are naturally averse to uncertainty, so we like single, confident numbers. But a single estimate from a sample hides how uncertain it is. A confidence interval gives a range of plausible values instead, which is a more honest answer.
In order to explain the concept of Confidence Intervals, I’ll use a simple example of a Chai dataset I created. There are two columns, personID and their milk-to-water preference ratio in tea. I have 60 records in the dataset – we will call this the original sample set. I’ll first have to explain a crucial factor of sampling.
We take multiple resamples of the sample set with replacement. I’ll explain what replacement means here in a minute. Imagine I pick a chit from a bowl of samples and each one has one person’s ID and milk-to-water ratio, I make a note and put that chit back and pick again, this is how resampling with replacement works. We need many rounds of resamples to be able to confidently give a result.
So say we take the original chai samples (60 records), I make 1000 resamples using this set, each is of length 60 (the same length as the original sample set).
These 1000 resamples we made, they’re called bootstrap samples. Each of these bootstrap samples has a mean of its own. The spread of the bootstrap means shows how much the average would vary from sample to sample.
Fact: If you’re familiar with how Random Forest models work, you must have come across this term Bootstrap aggregation. RF models use this method of drawing samples too.
Next, we choose a confidence level, usually 90% or 95%. So if we choose Confidence Level as 95%, we sort the 1,000 means and cut off the lowest 2.5% and highest 2.5% of the means.
bootstrap_means = []
for _ in range(1000):
resample_df = resample(chai_df, replace=True, n_samples=len(chai_df))
bs_mean = resample_df['Milk_water_ratio'].mean()
bootstrap_means.append(bs_mean)
bootstrap_means = pd.Series(bootstrap_means)
# 95% confidence level: cut off the lowest 2.5% and highest 2.5% of the means
lower_lim = bootstrap_means.quantile(0.025)
upper_lim = bootstrap_means.quantile(0.975)
print(f'Lower limit of Confidence Interval is: {lower_lim}')
print(f'Upper limit of Confidence Interval is: {upper_lim}')
print(f'We are 95% confident (in the sense of the method) that the average milk-to-water ratio lies between {lower_lim} and {upper_lim}')
If you visualise the above result using a histogram, you get something like this:
So what is the 95% referring to?
After taking many (for example, 1,000) resamples, 95% of the bootstrap means lie between two points, the lower and upper limits we found earlier. Those two points form the confidence interval. For my chai data, the Confidence Interval range came out as 1.24 to 1.53 (between the two dashed red lines of the histogram).
So we can say, with 95% confidence, that the average milk-to-water preference of chai drinkers like the ones I surveyed lies between 1.24 and 1.53, even though an individual's preference ranges from 1 to 3.
What are some real-world use cases where you’ve seen or used Confidence Intervals?
This was Day 2 of my 30 Days of Data Science writing challenge. Writing about this topic required a lot of effort and creativity: to understand the concept, come up with a chai related analogy and explain it as plainly as possible.
If you'd like to see the Python code for this post, you can read it at: https://github.com/Rupali-Gauravaram/dev-prompt-generator/blob/main/posts/post_02.ipynb
Top comments (0)