Statistical significance means that a result in your data is unlikely to have happened by random chance alone — conventionally, when the probability of a fluke (the p-value) is less than 5% (p < .05). If you've run a test, seen "p = .03," and had no idea whether to celebrate or panic, you're in the right place — this is the single most misunderstood number in research.
Key Takeaways
- Statistical significance tells you whether an effect is likely real or likely a random fluke — it does not tell you whether the effect is large or important.
- The p-value is the probability of seeing your result (or a more extreme one) if there were truly no effect. A p-value below your threshold (usually .05) is called "statistically significant."
- p < .05 is a convention, not a law of nature — it means you accept roughly a 1-in-20 chance of a false alarm.
- A significant result with a tiny effect size can be practically meaningless; always report an effect size alongside significance.
- Statistical significance ≠ practical importance ≠ proof — it's evidence, weighed against a threshold you set before collecting data.
What Does "Statistically Significant" Actually Mean?
A result is statistically significant when it is unlikely to have occurred by random chance if there were no real effect in the population you're studying. In practice, researchers set a cutoff (the alpha level, usually .05) before running their analysis. If the test returns a p-value below that cutoff, the result clears the bar and you reject the idea that "nothing is going on."
The intuition: imagine you flip a coin 10 times and get 9 heads. That could happen with a fair coin — but it's rare. At some point, "rare enough" makes you doubt the coin is fair. Statistical significance formalises that gut feeling with a number.
What Is a p-value, in Plain English?
A p-value is the probability of getting a result at least as extreme as the one you observed, assuming the null hypothesis (no real effect) is true. A small p-value means your data would be surprising in a world where nothing was happening — so you doubt that "nothing is happening" world.
Here's the trap almost everyone falls into: the p-value is not the probability that your hypothesis is true, and it's not the probability your result was due to chance. It's a conditional statement — "if there were no effect, how weird is my data?" A p-value of .03 means: if the null were true, you'd see data this extreme only 3% of the time.
Why Is p < .05 the Magic Number?
The .05 threshold means you're willing to accept a 5% chance of a false positive — declaring an effect exists when it doesn't. This cutoff traces back to statistician Ronald Fisher in the 1920s, who called 1-in-20 a convenient standard. It stuck, but it was always a convention, not a discovery.
Different fields use different thresholds. Genome-wide studies often demand p < .00000005 because they run millions of tests. Physics uses "5 sigma" (about p < .0000003) to declare a discovery. Your psychology thesis almost certainly uses .05 — but knowing why helps you defend it in a viva.
A Worked Example With Real Numbers
Suppose you want to know whether a new study-skills workshop improves exam scores. You compare two groups of students:
- Workshop group (n = 30): mean score = 74, SD = 8
- Control group (n = 30): mean score = 69, SD = 9
You run an independent-samples t-test and get:
t(58) = 2.27, p = .027, d = 0.59
Let's decode each piece:
- t(58) = 2.27 — the test statistic, with 58 degrees of freedom (roughly the total sample minus 2). Bigger absolute values mean a clearer difference relative to the noise.
- p = .027 — since this is below .05, the difference is statistically significant. In a world where the workshop did nothing, you'd see a gap this large only 2.7% of the time.
- d = 0.59 — the effect size (Cohen's d). This is the part significance alone hides: 0.59 is a medium effect, so the workshop isn't just real — it's meaningfully sized (about a 5-point score bump).
The takeaway: p = .027 tells you the effect is probably real; d = 0.59 tells you it's worth caring about. You need both. If you're deciding which test fits your own two-group comparison, our guide on the independent t-test vs Mann-Whitney U walks through the choice.
Significance vs Effect Size: What's the Difference?
This is where careers are made and dissertations are defended. Statistical significance answers "is it real?"; effect size answers "how big is it?" With a large enough sample, even a trivial difference becomes statistically significant — this is why huge studies can report p < .001 for effects too small to matter.
| Concept | Question it answers | Depends on sample size? | Example |
|---|---|---|---|
| p-value / significance | Is this likely a real effect or chance? | Yes — big samples make tiny effects significant | p = .027 |
| Effect size (d, r, η²) | How large is the effect? | No — it's scale-free | d = 0.59 |
| Confidence interval | What's the plausible range for the true effect? | Yes — narrows with bigger samples | 95% CI [0.6, 8.9] |
A p-value with no effect size is half a finding. APA 7 explicitly requires you to report both — reviewers will send it back if you don't.
How Do I Report Statistical Significance in APA 7?
APA 7 has strict formatting rules that trip up most students. Report the test statistic, degrees of freedom, exact p-value, and an effect size — italicising the statistics and dropping the leading zero on p.
Correct APA 7 examples:
- t-test: t(58) = 2.27, p = .027, d = 0.59
- ANOVA: F(2, 42) = 4.31, p = .019, η² = .17
- Correlation: r(48) = .34, p = .016
- Chi-square: χ²(1, N = 60) = 4.82, p = .028, V = .28
Key rules: italicise t, F, r, p, and d; write p = .027 (not p = 0.027); and if p is below .001, report p < .001 rather than a string of zeros. Getting this wrong is the most common reason results tables come back marked up.
Common Mistakes People Make With Significance
The biggest error is treating p = .05 as a wall between "true" and "false." A result at p = .049 and one at p = .051 are essentially identical evidence — the cliff-edge is artificial. Report exact p-values and let readers judge.
Other frequent traps: running many tests and reporting only the significant ones (p-hacking), interpreting a non-significant result as "proof of no effect" (absence of evidence isn't evidence of absence), and forgetting that significance says nothing about causation — that depends on your study design, not your p-value.
StatRyx flags these automatically: it checks your test assumptions, applies corrections for multiple comparisons when needed, and writes the significance statement in correct APA 7 so you don't have to second-guess the notation.
Let StatRyx Do the Significance Testing for You
Choosing the right test, checking assumptions, calculating the statistic, and formatting it in APA 7 is exactly the workflow that makes SPSS painful. StatRyx is an AI-powered statistical analysis tool that replaces manual SPSS workflows with automated, APA 7-formatted reporting — you upload your data, it picks the correct test, and it returns the p-value, effect size, and a ready-to-paste results paragraph. For a $100+/year SP