Reading Cross-Tab Results: What to Look for Beyond the Percentages¶
A cross-tab showing satisfaction at 72% for one segment and 61% for another practically invites you to say "the first group is happier." Sometimes that's a fair read. Sometimes it's a gap that would disappear if you asked a slightly different set of respondents the same question again. Here's how to tell the difference without leaning on statistical significance testing, which isn't part of how Opionate's Reporting or Text Analytics tools are built to work.
Check the Segment Sizes First¶
A percentage built from 15 responses moves around a lot more than one built from 300. Before reading anything into a gap between two segments, check how many responses are actually behind each one - see Cross-Tabs and Combined Survey Analysis in Reporting for setting up the comparison itself. A big-looking percentage gap built on small segments deserves much more skepticism than the same-sized gap built on large ones.
Look for a Pattern, Not a Single Number¶
One cross-tab showing a gap is a data point. The same gap showing up consistently across a related question, or holding steady as you look at the same comparison a different way, is a much stronger signal than any single cross-tab in isolation. Before reporting a finding, see if it's corroborated somewhere else in your data.
Consider Whether the Segments Are Really Comparable¶
A gap between two segments can reflect a real difference in opinion, or it can reflect that the two segments differ in some other way that happens to correlate with the question you're analyzing - tenure, how they were recruited, or something else entirely. Be cautious about attributing a gap directly to the segmenting variable without considering what else might differ between the groups.
What "Meaningful" Actually Means Here¶
Without formal significance testing available, "meaningful" is a judgment call informed by segment size, consistency across related questions, and how large the gap actually is - not a mathematically certified threshold. Treat a cross-tab as suggestive evidence worth investigating further, not as proof on its own, especially for a finding that will drive a real decision.
When in Doubt, Say So¶
If a cross-tab result is genuinely close or built on a thin segment, it's more honest to present it with that caveat ("directional, based on a smaller sample") than to report it with the same confidence as a robust, well-supported finding. A caveated finding that turns out to be real is more credible in hindsight than a confidently reported one that turns out to have been noise.
FAQ¶
Does Opionate offer statistical significance testing for cross-tabs?
No - significance testing (like t-tests or chi-square tests) isn't a current feature. Cross-tab results should be read as descriptive comparisons, not statistically tested findings.
Is there a minimum segment size I should require before trusting a cross-tab?
There's no universal number - the right threshold depends on how much the finding matters and how much variability you'd expect in your specific population. As a general habit, treat anything under a few dozen responses per segment with real caution.
Should I avoid cross-tabs on small segments entirely?
Not necessarily - a small-segment cross-tab can still be a useful early signal worth investigating further. Just don't present it with the same confidence as one built on a large, robust sample.
How does this apply to Combined Survey comparisons across waves?
The same caution applies - see Comparing Results Across Survey Waves: A Methodology Guide for wave-specific considerations.
Next: see When Your Sample Size Is Too Small to Trust a Result for the broader picture of sample size and confidence.