A Non-Technical Introduction to Hypothesis Testing

It's likely that you've had the need to run a hypothesis test at some point, whether you've been aware of it or not. This happens when you have some kind of "numerical claim" (or "hypothesis") that you want to make, and you'd like to determine the validity of this claim by collecting some relevant data. You may be tempted to just look at the data and immediately draw a conclusion, but sometimes your intuition will be wrong if you only rely on a quick glance at the data. This is where the idea of a hypothesis test comes into play.

Even though hypothesis testing is a very technical topic in statistics, it can still be explained at a high level - that is, without all the mathematical details. Let's now jump into the steps of a hypothesis test, while giving an example of what a test may look like:

Conducting a hypothesis test in 3 simple steps:

1. When you have some kind of claim (or hypothesis) that you'd like to assess, it's important to first specify this claim in terms of a formula that can be easily computed.
   
   This is better illustrated with an example: let's suppose we are passionate gardeners that want to grow some new strawberry plants. In our set of gardening tools, we have two different fertilizers that could be applied to the strawberry plants' soil: fertilizer A and fertilizer B. Through reviews that we read online for these two fertilizer products, it looks like customers are happier with using fertilizer A over fertilizer B. We don't have any reason to think otherwise after reading these reviews, so we'll claim that fertilizer A is "better" than fertilizer B.
   
   
   
   

   
   

   Wait...we can't quite move on with this claim yet - the notion of "better" is not specific enough for a statistician. We need to make our claim precise by looking at a quantifiable metric, like the average number of strawberries produced by a plant, or the proportion of strawberry plants that produce at least one strawberry.
   
   We only care about the fertilizer that will produce more strawberries for us to harvest, so let's go with the average number of strawberries produced by a plant as our metric of interest. This gives us the following claim: the average number of strawberries produced by plants grown with fertilizer A is higher than the average number of strawberries produced by plants grown with fertilizer B. We've repeated the same words quite a few times here, but this makes our claim precise enough for a statistician, since the average can be expressed as a formula which can be easily calculated.
   
   
2. Once a claim is stated using a formula, the next steps are to collect some data that will either validate or invalidate this claim, and then calculate values using the formula related to the claim.
   
   With the strawberry plants that we wish to grow, the easiest way to validate our claim is to grow some strawberry plants ourselves using fertilizer A and fertilizer B, and then keep track of the number of strawberries produced by each plant after some period of time. Let's say that we have a huge garden lot that will allow us to grow 30 strawberry plants for each fertilizer type. Then, after we've given the plants enough time to bloom, we can calculate the average number of strawberries produced by all plants grown with fertilizer A. We can also do the same for plants grown with fertilizer B.
   
   
   
   

   
   It's important that you collect enough data to be used in the next step. If we were lazy, or if we only had a small garden lot available, it may be tempting to grow only one or two strawberry plants for each fertilizer type. But this generally won't be enough to provide evidence in favour of/against the claim we proposed. There's always going to be some natural variation in the number of strawberries produced by each plant, regardless of the fertilizer used. So, if we see a higher strawberry yield for only a couple plants with one fertilizer type, it could just be a coincidence because of this natural variation.
   
   Instead, if we grew more and more strawberry plants, there's a lower chance of observing a higher yield for one fertilizer type that is purely by coincidence. It is more likely that a higher yield would actually come from differences in growing conditions among the plants, such as the fertilizer itself. The main takeaway here is that we need to have enough garden lot space and strawberry plant seeds before even starting an experiment like this! Otherwise, using up our fertilizer would just be a waste if we couldn't get any results out of it...
   
   
3. Finally, assess how reasonable it is to observe the collected data, assuming that your claim is true, and draw a conclusion.
   
   The core idea we'll use here is that, if the claim were true, and it seems unreasonable to observe the calculated metrics of interest under this claim, then it is most likely the case that the claim is false. This may sound like a contradictory way of thinking, but a good way to reason about this is similar to when we would suspect that someone is lying. We usually suspect this if it doesn't seem appropriate to believe someone's words, so instead we're more inclined to believe that they're not telling the truth. With hypothesis testing, we'll follow the same thought process and reject our claim if the data does not seem to support it.
   
   Continuing with the strawberry plants that we've grown, let's suppose that the average number of strawberries produced by plants with fertilizer A and fertilizer B were 10 and 20, respectively, which we calculated in the previous step. Right away, this would raise eyebrows, since we've observed that on average, plants with fertilizer B produce double the number of strawberries compared to plants with fertilizer A - which seems to go against the claim that we made in step 1. From this, we can conclude the hypothesis test with our results and say that our claim was likely wrong.
   
   Alternatively, if we had observed that the average number of strawberries produced by plants with fertilizer A and fertilizer B were 20 and 10, respectively, then these averages would actually be in agreement with our claim, and we wouldn't think that our claim was likely wrong.
   
   
   

This YouTube video also gives a quick, beginner-friendly overview of the steps above with a different example. It also reinforces the importance of writing a claim using a formula, along with the thought process in arriving at conclusions:

If you're interesting in learning more about other concepts that show up in hypothesis testing, you can check out this blog post written by Jason Brownlee. Don't worry, he doesn't include too many foreign concepts in statistics, and there aren't any scary long formulas!

At a high level, the idea of testing a hypothesis or claim using the three steps above may seem rather intuitive to you. However, some common mistakes could be made without following the procedure above, such as not collecting enough data in step 2, or incorrectly rejecting the initial claim when the discrepancy between the observed data and the claim itself is not significant enough to do so. The next time you evaluate the truth of a claim, make sure to keep these two points in mind to avoid making the same mistakes. Even though the second mistake is harder to resolve without some more statistics knowledge, simply being aware of it will ensure that you are not too hasty in coming to conclusions. After all, statisticians would rather have nothing to conclude than give conclusions that are entirely wrong.