Reading Pairwise Comparison Results: What the Rankings Actually Mean¶

A Pairwise Comparison widget in Reporting summarizes how each item performed across its head-to-head matchups - see Pairwise Comparison Testing: Setting Up a Head-to-Head Tournament for how the underlying tournament works. Here's how to read what comes out the other end.

Wins Are Relative, Not Absolute¶

An item's performance in a Pairwise Comparison is entirely a function of what it was matched against - it reflects how often it won when compared directly to other items in your pool, not an independent quality score. Swap out the pool of items being compared, and the same image could plausibly perform differently, since it's being judged relative to a different set of competitors.

Close Results Deserve Caution¶

An item that narrowly edges out another isn't necessarily meaningfully preferred - a near-even split in head-to-head wins is closer to "no strong preference either way" than to "item A is the clear winner." Treat a close margin with the same caution you'd apply to a small cross-tab difference (see Reading Cross-Tab Results: What to Look for Beyond the Percentages) rather than reporting a narrow edge as a decisive result.

How Many Comparisons Back a Result Matters¶

An item's win rate is only as reliable as the number of head-to-head matchups it was actually part of. Fewer items in your comparison pool, or a Randomized setup that doesn't guarantee every possible pairing, can mean some items accumulated fewer total comparisons than others - worth checking before treating every item's win rate as equally well-supported.

Randomized vs. League-Based Affects the Read¶

Since matchup structure differs between the two configuration options (see Randomized vs. League-Based in the setup guide), how confidently you can compare win rates across items can differ too - a League-based structure designed to surface a clear overall ranking reads somewhat differently than a Randomized one built around broader, less systematic coverage.

Pairing Results With Qualitative Context¶

A win rate tells you which item won more often; it doesn't tell you why. If you're testing images or concepts and want to understand the reasoning behind a preference, pairing Pairwise Comparison with an open-ended follow-up question elsewhere in the survey - and running it through Text Analytics - fills in the "why" a ranking alone can't answer.

FAQ¶

Can I trust a small difference in win rate between two items?
Treat it with the same caution as any close, small-sample result - a narrow margin often means "roughly tied" rather than a real preference.

Does item order in the comparison pool affect results?
The comparison is built around head-to-head matchups, not the order items were originally added - check your specific results for any positional patterns if you're concerned about order effects.

Can I export Pairwise Comparison results the way I can Text Analytics results?
Check your Reporting export options for what's available for Pairwise Comparison widgets specifically.

Is a Pairwise Comparison ranking the same as a Sequence Flow Route Flow result?
No - they're built from different kinds of respondent interactions. See Ranking Questions vs. Sequence Flow: When Simple Rank Order Isn't Enough for how Sequence Flow's ranking data is analyzed differently.

For the setup side of this section type, see Pairwise Comparison Testing: Setting Up a Head-to-Head Tournament.