When Your Sample Size Is Too Small to Trust a Result¶
How Many Panel Respondents Do You Need for a Reliable Result covers sizing a wave before you launch it. This one is about the other side of the same question: you've already got your responses, and now you need to judge how much confidence the results actually deserve.
Small Samples Move Around More¶
The core intuition, without needing formal statistics: a result built from a small number of responses can shift substantially if you'd happened to survey a slightly different set of people. A result built from a large number is more stable - individual respondents' quirks average out more reliably. This is true regardless of whether you're looking at a topline number or a cross-tab segment.
Where This Shows Up Most¶
Overall survey results are usually the most stable number you have, since they're built from your full response count. The moment you start segmenting - a cross-tab, a specific sub-group, a rare answer option - your effective sample size for that specific slice shrinks, sometimes dramatically. See Reading Cross-Tab Results: What to Look for Beyond the Percentages for how this plays out specifically in cross-tabs.
No Significance Testing, So Use Judgment Instead¶
Opionate doesn't offer formal statistical significance testing (t-tests, chi-square, and similar) as a built-in feature, so there's no automated flag telling you a result is or isn't reliable. In its place, a few practical habits help: be more cautious the smaller a segment gets, look for a finding to be corroborated by more than one question or cut of the data before treating it as solid, and when in doubt, report a smaller-sample finding as directional rather than definitive.
A Rough Rule of Thumb¶
There's no universal minimum that guarantees reliability, but as a practical habit: treat anything under roughly 30 responses in a specific segment or answer group with real caution, and treat anything under 10 as anecdotal rather than a finding you'd act on with confidence. These aren't formal thresholds - they're a sanity check, not a certified cutoff.
What to Do With a Thin Result¶
A thin-sample finding isn't worthless - it can be a useful early signal worth investigating further, especially if it's surprising or actionable. The responsible move is presenting it with an honest caveat about its sample size, and where possible, following up with more targeted data collection (a Panel Recruitment wave aimed at that specific segment, for instance) before treating it as settled.
FAQ¶
Does this apply to Text Analytics category counts too?
Yes - a category built from only a handful of classified responses deserves the same caution as a thin cross-tab segment. See Reading a Category Frequency Chart Without Overinterpreting It.
Is there a way to see my effective sample size for a specific cross-tab segment directly?
Check your Reporting widget for whether segment-level response counts are shown alongside the percentages.
Should I avoid reporting results from small segments entirely?
Not necessarily - just be transparent about the sample size behind them rather than presenting a thin result with the same confidence as a robust one.
How does this relate to sizing a Panel Recruitment wave in advance?
See How Many Panel Respondents Do You Need for a Reliable Result for the pre-launch planning side of this same underlying issue.
For the pre-collection planning version of this question, see How Many Panel Respondents Do You Need for a Reliable Result.