Distinguish statistical significance from practical importance

A result can be statistically significant and still be too small to change a practical decision. Another result can miss a conventional significance threshold while remaining compatible with effects that would matter.

Those are not contradictions. Statistical significance and practical importance answer different questions.

A careful interpretation keeps four elements separate: the estimated magnitude, its uncertainty, the statistical test and a justified threshold for what would count as meaningful in the study context. The threshold cannot be supplied by the p-value, and it should not be chosen after seeing the result.

Separate the questions before writing the conclusion

A significance test usually asks how incompatible the observed data are with a specified statistical model, often a model that sets one parameter to zero. Practical importance asks whether effects of a particular size would change understanding, policy, practice, cost, benefit or harm.

The American Statistical Association’s statement on p-values is direct: statistical significance does not measure the size of an effect or the importance of a result, and decisions should not be based only on whether a p-value crosses a threshold.

Keep the questions visible.

On a narrow screen, scroll the table sideways to read every column.

Interpretation element	Question it answers	What it does not establish
Point estimate	What magnitude and direction were estimated?	Whether the estimate is precise, unbiased or important
Confidence interval	Which parameter values remain compatible with the data and stated model at the selected confidence level?	A probability that the fixed parameter lies inside this one interval
P-value	How incompatible are the data with the tested model, under the calculation's assumptions?	The probability that the hypothesis is true, or the size of the effect
Practical threshold	What difference would be large enough to matter for the defined decision?	A universal boundary that transfers automatically to another context
Decision	What action or interpretation is justified when all four elements and the study limits are considered?	A guarantee that the result will replicate or generalize

A small p-value can accompany a small estimated effect when the data are highly precise. A larger p-value can accompany a potentially important estimate when the interval is wide. Sample size, variability, design and model all influence statistical uncertainty; importance still depends on the substantive problem.

Define what would matter before reading the result

A practical threshold is a value on the effect scale that separates differences too small to affect the intended decision from differences that would matter. It may be called a smallest effect size of interest, minimally important difference, decision threshold or another field-specific term.

The label is less important than the justification. Possible bases include:

a theoretical prediction about the smallest effect the explanation requires;
a validated minimally important difference for the outcome;
a measurement-resolution or just-noticeable boundary;
the benefits, costs and harms attached to a decision;
prior evidence from genuinely comparable studies;
a policy, professional or regulatory threshold that applies to the setting;
informed stakeholder judgment recorded before outcome inspection.

The ASA President’s Task Force statement notes that decision thresholds should be explicitly defined from study goals and the consequences of incorrect decisions. Conventions differ across disciplines and purposes.

A threshold chosen after seeing the estimate can become a way to manufacture the preferred conclusion. Record who set it, when it was set, what scale it uses and why that value would change the decision.

If no defensible threshold exists, say so. You can still report the estimate, interval and p-value, discuss plausible consequences and identify what evidence would be needed. Do not invent a precise cutoff merely to make the interpretation look complete.

Prefer a meaningful scale

An unstandardized effect often supports a practical decision more directly than a standardized label. Five minutes, three percentage points, two errors per 1,000 cases or four points on a validated scale can be connected to consequences. A standardized effect can help compare results measured on different scales, but its meaning depends on the standardizer, population, design and outcome variability.

Lakens’s effect-size primer explains why the effect-size measure must fit the research design and question. Schäfer and Schwarz’s analysis of effect magnitudes shows why global small, medium and large benchmarks can mislead: typical magnitudes differ across subfields and designs, and published effects can be distorted by selection and publication bias.

Do not translate a standardized effect into “small and unimportant” without a substantive argument. A numerically small effect may accumulate over time or affect many people. A numerically large effect may concern an outcome with little decision value. The practical question belongs to the outcome and setting, not to the adjective.

Read five fields together

Create a short interpretation record before drafting prose.

On a narrow screen, scroll the table sideways to read every column.

Field	Record	Check
Effect definition	Parameter, contrast, direction, unit and analysis population	Is the sign and reference group unambiguous?
Point estimate	Estimated magnitude on the chosen scale	Is it reported with its unit rather than only an adjective?
Uncertainty	Interval limits, confidence level and method	Which negligible, beneficial or harmful values remain compatible?
Test result	Tested model, p-value and any multiplicity adjustment	Is the test being interpreted as model compatibility rather than importance?
Practical threshold	Value, direction, source, timing and justification	Was it defined independently of the observed result?
Bounded conclusion	What the output supports, remains uncertain and cannot decide	Does the language preserve the design and measurement limits?

If you need to revisit the interval itself, use the guide to interpreting a confidence interval without overstating certainty. The present task begins after the effect and uncertainty have been calculated with a suitable method.

Worked fictional outputs

The study, threshold and numerical outputs below are wholly fictional. They were constructed to teach interpretation and do not report participant data or recommend a universal five-point boundary.

A fictional randomized study compares two versions of a planning worksheet. The prespecified contrast is version B minus version A on a 0–100 planning score. Before the outcomes are examined, the project team records that differences smaller than 5 points in either direction would not change the implementation decision. The fictional practical boundaries are therefore −5 and +5 points.

Output A: statistically detectable, but below the stated practical threshold

The fictional output is:

estimated mean difference: +1.8 points;
95% confidence interval: +1.2 to +2.4 points;
p-value for a zero difference: p < .001.

The estimate is positive and the interval excludes zero, so the output is statistically detectable under the stated model. Every value inside this 95% interval is also between the fictional −5 and +5 practical boundaries.

A bounded interpretation is:

Version B was associated with an estimated 1.8-point higher mean score (95% CI 1.2 to 2.4; p < .001). Although the estimate was statistically distinguishable from zero under the prespecified analysis, the entire 95% interval remained below the project’s pre-justified five-point threshold for changing implementation.

This does not prove that a 1.8-point difference is universally trivial. It states what the result means under this fictional decision rule. A formal claim of statistical equivalence would require an analysis and interval level designed for that question.

Output B: not conventionally significant, yet practically unresolved

The second fictional output is:

estimated mean difference: +3 points;
95% confidence interval: −7 to +13 points;
p-value for a zero difference: p > .05.

The interval contains zero, but it also crosses both fictional practical boundaries. It remains compatible with meaningful harm below −5, negligible values between −5 and +5, and meaningful benefit above +5.

A bounded interpretation is:

The estimated difference was +3 points (95% CI −7 to +13; p > .05). The result was not statistically significant at the conventional threshold, but the interval remained compatible with practically meaningful harm, negligible differences and practically meaningful benefit. The study therefore did not resolve the implementation question.

“Not statistically significant” is not a synonym for “no effect,” “the versions are equivalent” or “the intervention does not matter.”

The contrast is easier to audit in one matrix.

On a narrow screen, scroll the table sideways to read every column.

Output	Estimate and 95% interval	Zero-difference test	Relation to fictional ±5 threshold	Bounded conclusion
A	+1.8; +1.2 to +2.4	p < .001	Entire interval inside the practical bounds	Statistically detectable; below this project's stated threshold, without claiming universal triviality
B	+3; −7 to +13	p > .05	Interval crosses harmful, negligible and beneficial regions	Practically unresolved; non-significance does not establish no meaningful effect

Do not claim equivalence from a non-significant test

A conventional null-hypothesis test is usually designed to ask whether the data are sufficiently incompatible with a specified null value. Failing to reject that model does not show that all effects large enough to matter are absent.

An equivalence test asks a different question. Lakens, Scheel and Isager’s equivalence-testing tutorial explains how bounds based on a smallest effect size of interest can be used to test whether effects at least that extreme can be rejected. The bounds should be justified independently of the result, and the method should be planned and reported correctly.

Do not retrofit the phrase “statistically equivalent” onto an ordinary p-value above .05. Report the estimate and interval. If the interval remains wide enough to include meaningful benefit or harm, the honest conclusion is uncertainty.

The same caution applies in the other direction. A p-value below .05 does not prove the effect clears a practical threshold. Output A demonstrates why the estimate and interval must remain visible.

Carry design and measurement limits into the decision

A precise, threshold-crossing estimate can still be misleading if the effect measure is poorly defined or the analysis is biased. Statistical output does not repair a weak design.

Audit at least these areas:

whether assignment, sampling or adjustment supports the intended causal or population claim;
whether the outcome measures the construct and has enough resolution for the chosen threshold;
whether missing data, attrition or non-adherence could change the estimate;
whether the analysis, outcome and threshold were selected before or after inspecting results;
whether multiple tests or repeated looks were accounted for;
whether the effect persists for relevant groups, time periods and implementation conditions;
whether costs, harms and opportunity costs fall on the same people who receive the benefit.

Greenland and colleagues’ guide to statistical misinterpretations emphasizes that estimates, intervals and tests depend on the full analysis model and its assumptions. Practical interpretation should name the limits that could change the decision rather than treating a neat numerical matrix as complete evidence.

Write results and discussion sentences with different jobs

In the results section, identify the contrast, estimate, interval and test without announcing that the effect is important.

The estimated difference in mean planning score was +1.8 points for version B relative to version A (95% CI +1.2 to +2.4; p < .001).

In the discussion, compare the plausible range with the pre-justified threshold and state the decision consequence.

The interval was entirely below the project’s prespecified five-point implementation threshold, so the statistically detectable difference would not change the fictional implementation decision. This judgment depends on that context-specific threshold and does not make 1.8 points universally unimportant.

For Output B:

The interval crossed both practical boundaries, so the study could not distinguish negligible differences from effects large enough to change the decision. Additional evidence or a more precise design would be needed.

Use the results-versus-discussion guide to keep reporting separate from contextual explanation. If the study is still being planned, the sample-size justification guide can help align precision or power with the smallest effect the project needs to detect or rule out.

A practical 12-step workflow

Name the parameter, contrast, direction and unit.
Confirm that the effect measure fits the design and research question.
Record the point estimate without a qualitative label.
Record the interval, confidence level and method.
Record the tested model and exact p-value where available.
Identify any multiplicity adjustment or selection of analyses.
Retrieve the practical threshold and its documented source.
Confirm that the threshold was set independently of the observed result.
Compare the estimate and both interval limits with the threshold.
State which harmful, negligible and beneficial values remain compatible.
Add design, measurement, missingness and generalization limits.
Write a bounded decision statement and remove probability, certainty and “no effect” shortcuts.

The purpose is not to replace statistical analysis with a single practical cutoff. It is to prevent one statistical cutoff from impersonating a substantive decision.

Statistical significance can be useful when its model and assumptions are clear. Practical importance becomes defensible when the outcome scale, uncertainty, consequences and decision threshold are equally visible. Statistics and data-analysis support can help you examine effect measures, uncertainty and context-specific interpretation using your own data while you retain responsibility for the research, analysis and reporting.