Hypothesis tests — FAQ

The significance level, $\alpha$, is a threshold you set before you conduct the test. It's your standard for how strong the evidence needs to be. The p-value is calculated from your data. It's the probability of seeing your sample result (or something more extreme) if the null hypothesis were true. You reject $H_0$ if your p-value is smaller than your chosen $\alpha$.

In a two-tailed test, an 'extreme' result could be either significantly higher or significantly lower than the null hypothesis value. The significance level $\alpha$ represents the total probability of making a Type I error. We split this probability between the two possible extremes (the two tails of the distribution), so each tail has a probability of $\alpha/2$.

This is a very rare occurrence in practice. By convention, if the test statistic is in the critical region (e.g., for $Z \le -1.645$, if $Z = -1.645$), you reject $H_0$. However, it highlights that the boundary is arbitrary. In an exam, you should state your comparison clearly (e.g., $-1.645 = -1.645$) and follow the rule to reject $H_0$.

In the 9709 syllabus, continuity corrections are required when approximating a discrete distribution (like Binomial or Poisson) with a continuous one (Normal), especially for finding probabilities of specific ranges. For hypothesis testing of a proportion, where you are testing a single sample proportion $\hat{p}$, a continuity correction is generally not required or expected. The Normal approximation to the sampling distribution of $\hat{p}$ is considered adequate without it, provided $np > 5$ and $n(1-p) > 5$.