Two options have been tested. One scores three points higher, the report marks the gap with a significance test, and the team would like a winner. The test is part of the answer.
A research difference is big enough to act on when it has been measured precisely enough, would be worth enough to the business if it is real, and leaves an acceptable cost if the choice proves wrong. A significance test speaks to the first of these, and only in part.
What three points can tell you
Consider a hypothetical packaging test. Each of two designs is shown to its own randomly allocated group of 400 category buyers: 34% would probably or definitely buy Design A, and 31% Design B.
Treating each group as a simple random sample, the textbook 95% margin of error is about 4.5 to 4.6 points on each score. On the gap between them it is wider, about 6.5 points, because the gap inherits uncertainty from both scores.
A 95% confidence interval for the gap runs from Design B ahead by three and a half points to Design A ahead by nine and a half. A conventional test would not find this gap significant. That does not show the designs perform equally. It shows that this sample cannot tell them apart at that standard of evidence.
Now give each design 4,000 people. The margin on the gap shrinks to about two points, and the same three-point difference passes a conventional test. Nothing about what three points would be worth has changed.
The design of the sample matters too. If the same people rated both designs, a different calculation applies. If the sample is weighted or drawn from an opt-in panel, the textbook margin rests on further assumptions and can understate the uncertainty.
What a significance test establishes
A significance test asks how surprising a gap at least this large would be if the designs truly performed the same and the test’s assumptions were met.
That protects a team from reading too much into sampling noise. It does not give the probability that Design A is better, and it says nothing about whether three points matter.
The American Statistical Association made the point in its statement on p-values: “A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.” Testing remains valuable. The question is what it can settle.
What was measured also shapes the reading. Stated purchase intent is a survey response, and three points of intent are not three points of sales.
Because each person saw one design, allocated at random, a real gap can be attributed to the design, for these people and this measure. A gap between two customer groups is an observation about groups that may differ in many other ways.
The number of comparisons matters as well. If a report sets the two designs against each other in twenty subgroups, and they truly perform the same in every one, about one comparison would be expected to pass a 5% threshold by chance.
When scores come from a model, such as a driver analysis or a simulator, the model’s assumptions add their own uncertainty.
The same gap can matter a great deal or very little
Imagine both designs are new, cost the same to produce and fit the brand equally well. Choosing the higher scorer is then reasonable even when the evidence cannot separate them.
Little is lost if that choice proves wrong, so a gap that misses the threshold can still serve as a tie-breaker.
Now imagine Design A would replace a pack customers have recognized for years, with new tooling and a retail changeover. A plausible range that includes losing ground is a serious problem. And even if the gap were certainly real, three points of stated intent might not repay the cost of the change.
A task force set up by the association’s president put the principle directly in its statement on significance and replicability: “If thresholds are deemed necessary as part of decision-making, they should be explicitly defined based on study goals, considering the consequences of incorrect decisions.”
That cuts both ways. A cheap, reversible choice can proceed on modest evidence. A costly one may need a stricter standard, or a test closer to the real decision, such as the new design shown beside the existing pack. No single sample size or error rate suits every decision; the standard follows from what is at stake.
What a clear recommendation explains
Good quantitative research reports the size of a difference with its uncertainty, in units the business recognizes, and says which assumptions matter most to the reading.
The recommendation then says what the difference might be worth and what being wrong would cost. It also names what would change the advice: a larger gap in a follow-up test, a cheaper way to reverse the choice, or evidence that the measure tracks behavior in this category. Those conditions tell the team what to watch after it acts.
Sometimes the plain answer is that the research cannot separate the options, and the choice should rest on cost, brand fit or ease of production. Said clearly, that is useful. It stops a team from treating a ranked chart as a verdict, and from treating a missed threshold as proof that nothing differs.