Hypothesis testing language: hypotheses, significance and conclusions

A hypothesis test uses sample evidence to assess a claim about a population. It does not prove that a claim is true or false. Instead, it asks whether the observed result would be sufficiently unusual if a stated model were true.

The language matters. In particular:

$$\boxed{\text{not rejecting }H_0\text{ is not the same as proving }H_0.}$$

This lesson develops the reasoning and vocabulary used in binomial, normal and correlation hypothesis tests.

Prerequisites

You should be able to:

• distinguish a population and a sample;
• interpret probabilities and complements from probability;
• understand that a random variable varies from sample to sample;
• use a probability distribution as a mathematical model.

The logic of a test

Begin by temporarily assuming a precise claim, called the null hypothesis, is true. Use that assumption to answer:

If the null hypothesis were true, how surprising would evidence at least this extreme be?

If the evidence is very unlikely under the null hypothesis, reject it. Otherwise, do not reject it.

Worked example 1: the central idea

A manufacturer claims that half of the sweets in a large production process are orange. A random sample of $20$ sweets contains $19$ orange sweets.

Let $p$ be the population proportion of orange sweets. Temporarily assume

$$H_0:p=0.5.$$

If $X$ counts orange sweets, the null model is

$$X\sim\operatorname{B}(20,0.5).$$

Evidence at least as extreme as $19$ in the relevant upper direction means $19$ or $20$:

$$\begin{aligned} P(X\geq19) &=P(X=19)+P(X=20)\\ &=\binom{20}{19}(0.5)^{20}+\binom{20}{20}(0.5)^{20}\\ &=\frac{21}{2^{20}}\\ &\approx0.0000200. \end{aligned}$$

This is possible, but extremely unlikely if $p=0.5$. The observation is therefore strong evidence against $H_0$ in favour of a larger value of $p$.

Unlikely does not mean impossible

A hypothesis test deliberately rejects $H_0$ for some results that can occur when $H_0$ is true. Statistical decisions therefore carry a controlled risk of error.

Parameters, statistics and hypotheses

A parameter describes a population and is usually unknown. A statistic is calculated from a sample and provides evidence about the parameter.

Context	Population parameter	Sample statistic
Success probability	$p$	observed proportion $\hat p$ or count $x$
Population mean	$\mu$	sample mean $\bar x$
Population correlation	$\rho$	sample PMCC $r$

Hypotheses concern the unknown parameter, not the statistic already observed.

The null hypothesis, $H_0$, is the precise claim tested. At A level it normally contains equality so that it specifies the probability model used in calculations.

The alternative hypothesis, $H_1$, states the departure for which evidence is being sought.

Worked example 2: identify the correct symbol

A random sample of bottles has mean volume $\bar x=497.8$ ml. The labelled population mean is $500$ ml, and an inspector suspects underfilling.

The known value $497.8$ is a sample statistic. The hypotheses must concern the unknown population mean $\mu$:

$$\boxed{H_0:\mu=500}, \qquad \boxed{H_1:\mu<500}.$$

Writing $H_0:\bar x=500$ would be wrong because $\bar x$ has already been observed as $497.8$.

Self-check 1

A researcher calculates $r=-0.62$ from a sample and tests whether there is negative linear correlation in the population. Write suitable hypotheses.

Answer

Let $\rho$ be the population PMCC. Then

$$H_0:\rho=0, \qquad H_1:\rho<0.$$

The hypotheses use $\rho$, not the observed sample statistic $r$.

Choosing one tail or two tails

The wording of the investigation determines the alternative hypothesis and hence which results count as extreme.

Question about parameter $\theta$	Alternative hypothesis	Test	Evidence against $H_0$
Has it increased?	$H_1:\theta>\theta_0$	upper tailed	unusually large values
Has it decreased?	$H_1:\theta<\theta_0$	lower tailed	unusually small values
Has it changed?	$H_1:\theta\neq\theta_0$	two tailed	unusually small or large values

A one tailed test has one direction of departure. A two tailed test admits either direction.

Choose the tails from the question before using the data. Looking at the result first and then choosing the favourable tail makes false positive results more likely.

Worked example 3: translate words into hypotheses

A school has historically found that $30\%$ of pupils travel by bus. Write hypotheses for each investigation, where $p$ is the current population proportion.

1. The school suspects the proportion has risen.

   $$H_0:p=0.30, \qquad H_1:p>0.30.$$

   This is upper tailed.

2. The school investigates whether the proportion is different.

   $$H_0:p=0.30, \qquad H_1:p\neq0.30.$$

   This is two tailed. A decrease or an increase could count as evidence.

3. The school suspects fewer pupils now travel by bus.

   $$H_0:p=0.30, \qquad H_1:p<0.30.$$

   This is lower tailed.

Do not use the sample direction

If the question asks whether $p$ has changed, use $H_1:p\neq p_0$ even if the observed proportion happens to be above $p_0$.

Significance level and critical region

The significance level, denoted by $\alpha$, is the maximum intended probability of rejecting $H_0$ when $H_0$ is true. Common choices are

$$\alpha=0.05\quad\text{and}\quad\alpha=0.01.$$

A $1\%$ test requires stronger evidence than a $5\%$ test.

The critical region is the set of test statistic values for which $H_0$ is rejected. Its boundary is a critical value. The complementary set is sometimes called the acceptance region, although “do not reject” is safer language than “accept”.

For a continuous model, a two tailed test at overall level $5\%$ usually places $2.5\%$ in each tail:

$$P(\text{lower critical region}\mid H_0)=0.025,$$ $$P(\text{upper critical region}\mid H_0)=0.025.$$

For a discrete distribution, exact tail probabilities jump between available values. The actual significance level is therefore often less than the stated level.

Worked example 4: decide from a critical region

For a lower tailed test at the $5\%$ level, suppose the critical region is

$$X\leq7.$$

• If $x=6$, the observation is in the critical region, so reject $H_0$.
• If $x=7$, equality is included, so reject $H_0$.
• If $x=8$, the observation is outside the critical region, so do not reject $H_0$.

If $P(X\leq7\mid H_0)=0.037$, the actual significance level is $3.7\%$, not $5\%$.

The p value

The p value is the probability, calculated assuming $H_0$ is true, of obtaining the observed result or a result more extreme in the direction or directions specified by $H_1$.

The decision rule is

$$\boxed{\text{reject }H_0\text{ if }p\text{ value}\leq\alpha.}$$

A small p value means the data are difficult to reconcile with $H_0$. It is not the probability that $H_0$ is true.

Worked example 5: compare a p value with two levels

A test gives a p value of $0.032$.

At the $5\%$ level,

$$0.032<0.05,$$

so reject $H_0$. The result is statistically significant at $5\%$.

At the $1\%$ level,

$$0.032>0.01,$$

so do not reject $H_0$. The result is not statistically significant at $1\%$.

There is no contradiction. A smaller significance level sets a stricter standard of evidence.

Self-check 2

A test has $H_1:\mu>40$, significance level $10\%$, and p value $0.10$. State the decision.

Answer

Reject $H_0$ because

$$0.10\leq0.10.$$

Equality is included in the rejection rule. There is sufficient evidence at the $10\%$ level to suggest that $\mu>40$.

Writing conclusions correctly

A complete conclusion contains:

1. the decision, reject $H_0$ or do not reject $H_0$;
2. the significance level;
3. “sufficient evidence” or “insufficient evidence”;
4. the alternative claim in context.

When the result is significant

Suppose $H_1:p>0.4$ and the result is significant at $5\%$:

Reject $H_0$. There is sufficient evidence at the $5\%$ significance level to suggest that the population proportion is greater than $0.4$.

When the result is not significant

Suppose $H_1:\mu<12$ and the result is not significant at $5\%$:

Do not reject $H_0$. There is insufficient evidence at the $5\%$ significance level to suggest that the population mean is less than $12$.

Avoid “accept $H_0$” and “prove $H_0$”. A non-significant result may arise because $H_0$ is a good model, because the true departure is small, or because the sample is too small to detect it.

Worked example 6: repair an invalid conclusion

A test of $H_0:p=0.6$ against $H_1:p\neq0.6$ gives p value $0.18$. A student writes:

The probability that $H_0$ is true is $0.82$, so $p=0.6$.

This is invalid. The p value is calculated conditional on $H_0$ being true. It does not assign a probability to $H_0$.

At the $5\%$ level, the correct argument is

$$0.18>0.05,$$

so do not reject $H_0$. There is insufficient evidence at the $5\%$ level to suggest that $p$ differs from $0.6$.

Type I and Type II errors

A decision can be wrong because it is based on a random sample.

Reality	Reject $H_0$	Do not reject $H_0$
$H_0$ is true	Type I error	correct decision
$H_0$ is false	correct decision	Type II error

A Type I error is rejecting a true null hypothesis. Its probability is the actual significance level:

$$P(\text{Type I error})=P(\text{reject }H_0\mid H_0\text{ true}).$$

A Type II error is failing to reject a false null hypothesis. If its probability is $\beta$, then

$$\text{power}=1-\beta$$

is the probability that the test correctly rejects $H_0$ for that particular alternative value.

Reducing $\alpha$ makes a Type I error less likely, but with sample size fixed it usually makes a Type II error more likely. Increasing the sample size can improve the test’s ability to detect a real departure.

Worked example 7: errors in context

A screening process tests

$$H_0:\text{a batch meets the standard}$$

against

$$H_1:\text{the batch does not meet the standard}.$$

• A Type I error occurs if a satisfactory batch is rejected.
• A Type II error occurs if an unsatisfactory batch is not rejected.

The definitions come from the hypotheses, not from which practical consequence seems more serious.

Self-check 3

A company tests $H_0:p=0.02$ against $H_1:p>0.02$, where $p$ is the proportion of defective components. Describe both errors in context.

Answer

• Type I error: conclude that the defect proportion has increased above $0.02$ when it is still $0.02$.
• Type II error: fail to conclude that the defect proportion has increased when in fact $p>0.02$.

Statistical significance is not practical importance

A statistically significant result says that the data would be unusual under $H_0$ at the chosen level. It does not say that the effect is large or important.

For example, a very large sample might provide strong evidence that a mean has changed from $50$ to $50.1$. The change can be statistically significant while being negligible in practice. Conversely, a potentially important observed difference can fail to be significant in a small sample because the estimate is imprecise.

Judge practical importance using context, the size of the effect and, where available, an interval estimate, not the p value alone.

Common misconceptions

”$p=0.04$ means there is a $4\%$ chance that $H_0$ is true”

No. It means that, assuming $H_0$ is true, the probability of evidence at least as extreme as that observed is $0.04$.

”A result outside the critical region proves $H_0$”

No. It means only that the evidence is not strong enough to reject $H_0$ at the chosen level.

”A $5\%$ significance level means $5\%$ of the data are wrong”

No. It controls the long run probability of rejecting $H_0$ when $H_0$ is true.

”Two tailed means use $5\%$ in each tail”

No. For an overall $5\%$ test, the total allowance across both tails is $5\%$, normally $2.5\%$ per tail.

”The observed direction chooses the alternative”

No. The research question chooses $H_1$ before the sample result is examined.

Final self-check

A café claims that the mean waiting time is $4$ minutes. A manager suspects that it has changed. A test gives p value $0.047$.

1. Write the hypotheses.
2. Is the test one tailed or two tailed?
3. State the decision at the $5\%$ level.
4. State the decision at the $1\%$ level.
5. Does the result show that the change is important in practice?

Answers

Let $\mu$ be the population mean waiting time.

1. $H_0:\mu=4$ and $H_1:\mu\neq4$.
2. It is two tailed because a change in either direction is relevant.
3. Since $0.047<0.05$, reject $H_0$. There is sufficient evidence at the $5\%$ level to suggest that the population mean waiting time has changed from $4$ minutes.
4. Since $0.047>0.01$, do not reject $H_0$. There is insufficient evidence at the $1\%$ level to suggest that the population mean waiting time has changed.
5. No. Statistical significance alone gives no information about the size or practical importance of the change.

Test procedure summary

1. Define the population parameter in context.
2. Write $H_0$ and $H_1$ using that parameter.
3. Choose one tail or two tails from the question.
4. State the null distribution and significance level.
5. Calculate the test statistic and p value, or find the critical region.
6. Compare the p value with $\alpha$, or the statistic with the critical region.
7. Reject or do not reject $H_0$.
8. Give a cautious conclusion about $H_1$ in context.

Next steps

• Apply this language to exact binomial hypothesis tests.
• Test a population mean using normal hypothesis tests.
• Test population association using hypothesis tests for correlation.