alt="images"/>, which is equal to 20/10 = 2. Suppose the obtained sample mean were equal to 20, and the mean under the null hypothesis, μ0, were equal to 18. The numerator of zM would thus be 20 – 18 = 2. When 2 is divided by the standard error of 2, we obtain a value for zM of 1.0, which is not statistically significant at p < 0.05.
Now, consider the scenario where the standard error of the mean remains the same at 2, but that instead of the sample mean being equal to 20, it is equal to 30. The difference between the sample mean and the population mean is thus 30 – 18 = 12. This difference represents a greater distance between means, and presumably, would be indicative of a more “successful” experiment or study. Dividing 12 by the standard error of 2 yields a zM value of 6.0, which is highly statistically significant at p < 0.05 (whether for a one‐ or two‐tailed test).
Having the value of zM increase as a result of the distance between and μ0 increasing is of course what we would expect from a test statistic if that test statistic is to be used in any sense to evaluate the strength of the scientific evidence against the null. That is, if our obtained sample mean turns out to be very different than the population mean under the null hypothesis, μ0, we would hope that our test statistic would measure this effect, and allow us to reject the null hypothesis at some preset significance level (in our example, 0.05). If interpreting test statistics were always as easy as this, there would be no misunderstandings about the meaning of statistical significance and the misguided decisions to automatically attribute “worth” to the statement “p < 0.05.” However, as we discuss in the following cases, there are other ways to make zM big or small that do not depend so intimately on the distance between and μ0, and this is where interpretations of the significance test usually run awry.
Consider the case now for which the distance between means, is, as before, equal to 2.0 (i.e., 20 – 18 = 2.0). As noted, with a standard error also equal to 2.0, our computed value of zM came out to be 1.0, which was not statistically significant. However, is it possible to increase the size of zM without changing the observed distance between means? Absolutely. Consider what happens to the size of zM as we change the magnitude of either σ or n, or both. First, we consider how zM is defined in part as a function of σ. For convenience, we assume a sample size still of n = 100. Consider now three hypothetical values for σ: 2, 10, and 20. Performing the relevant computations, observe what happens to the size of zM in the case where σ = 2:
The resulting value for zM is quite large at 10. Consider now what happens if we increase σ from 2 to 10:
Notice that the value of zM has decreased from 10 to 2. Consider now what happens if we increase σ even more to a value of 20 as we had originally:
When σ = 20, the value of zM is now equal to 1, which is no longer statistically significant at p < 0.05. Be sure to note that the distance between means has remained constant. In other words, and this is important, zMdid not decrease in magnitude by altering the actual distance between the sample mean and the population mean, but rather decreased in magnitude only by a change in σ.
What this means is that given a constant distance between means , whether or not zM will or will not be statistically significant can be manipulated by changing the value of σ. Of course, a researcher would never arbitrarily manipulate σ directly. The way to decrease σ would be to sample from a population with less variability. The point is that decisions regarding whether a “positive” result occurred in an experiment or study should not be solely a function of whether one is sampling from a population with small or large variance!
Suppose now we again assume the distance between means to be equal to 2. We again set the value of σ at 2. With these values set and assumed constant, consider what happens to zM as we increase the sample size n from 16 to 49 to 100. We first compute zM assuming a sample size of 16:
With a sample size of 16, the computed value for zM is equal to 4. When we increase the sample size to 49, again, keeping the distance between means constant, as well as the population standard deviation constant, we obtain:
We see that the value of zM has increased from 4 to 6.9 as a result of the larger sample size. If we increase the sample size further, to 100, we get
and see that as a result of the even larger sample size, the value of zM has increased once again, this time to 10. Again, we need to emphasize that the observed increase in zM is occurring not as a result of changing values for or σ, as these values remained constant in our above computations. Rather, the magnitude of zMincreased as a direct result of an increase in sample size, n, alone. In many research studies, the achievement of a statistically significant result may simply be indicative that the researcher gathered a minimally sufficient sample size that resulted in zMfalling in the tail of the z distribution. In other cases, the failure to reject the null may in reality simply indicate that the investigator had insufficient sample size. The point is that unless one knows how n can directly increase or decrease the size of a p‐value, one cannot be in a position to understand, in a scientific sense, what the p‐value actually means, or intelligently evaluate the statistical evidence before them.
2.28.2 The Make‐Up of a p‐Value: A Brief Recap and Summary
The simplicity of these demonstrations is surpassed only by their profoundness. In our simple example of the one‐sample z‐test for a mean, we have demonstrated that the size of zM is a direct function of three elements: (1) distance , (2) population standard deviation σ, and (3) sample size n. A change in any of these while holding the others constant will necessarily, through nothing more than the consequences of how the significance test is constructed and functionally defined, result
