Published research on AI-human coding agreement - covered in our companion guide on what the research says - is useful for setting general expectations, and it's answering a question about someone else's data, coded with someone else's category definitions, on someone else's topic. The single most credible way to answer whether AI-assisted coding is trustworthy for your specific project is the same one qualitative researchers have used for decades to check two human coders against each other: run an independent inter-rater reliability check, just with one of the two "raters" being the AI-assisted process instead of a second human.
Table of Contents¶
- The Classic Inter-Rater Reliability Setup, Adapted
- Step by Step: Running the Check
- Reading the Result
- Watch for the Kappa Paradox
- What to Do With a Disappointing Result
- FAQ
The Classic Inter-Rater Reliability Setup, Adapted¶
In traditional qualitative research, two coders independently apply the same set of category definitions to the same batch of responses, without seeing each other's work, and their results are then compared to see how often they agree - a check that validates the category definitions and the coding process itself, not just the individual coders' judgment. Adapting this for AI-assisted coding changes almost nothing about the logic: a human coder codes a sample independently, AI-assisted coding is run on the same sample using the same category definitions, and the two sets of results are compared exactly the way two human coders' results would be. The human coder should genuinely not see the AI's output before coding their own sample, for the same reason a second human coder wouldn't be shown the first coder's results in advance - seeing the other coder's answers first would inflate agreement artificially and defeat the purpose of an independent check.
Step by Step: Running the Check¶
Start by pulling a random sample from your full dataset - our companion guide on validating an AI classifier covers sizing this sample in more depth, but a few hundred responses is a common practical range for a meaningful check without hand-coding your entire dataset. Have a human coder, using the exact same category definitions that will be given to the AI-assisted process, code that sample independently, ideally without having looked at any AI-generated output for those specific responses beforehand. Run the AI-assisted classification on the same sample, using the same category definitions. Then line up the two sets of results side by side, response by response, and calculate agreement - either a straightforward percent-agreement figure (what share of responses got the same category from both), or, for a more rigorous read, Cohen's Kappa, which adjusts for the agreement you'd expect from chance alone given how common each category is in your data.
If your project involves more than one human coder normally - a research team, rather than a single analyst - it's worth having each human coder run their own independent AI-comparison check separately, which also gives you a human-to-human reliability figure to compare the AI-to-human figure against, a more complete and more defensible picture than either comparison alone.
Reading the Result¶
A kappa value in the "substantial" range (roughly 0.61 to 0.80) or higher generally indicates the AI-assisted process is agreeing with your specific human coder's judgment about as well as two well-trained human coders typically agree with each other on comparable qualitative tasks - a reasonable bar for treating the AI-assisted result as reliable for ordinary reporting purposes, with the spot-checking discipline from our guide on when to trust AI categorization still worth applying to catch individual edge cases. A "moderate" result (0.41 to 0.60) suggests real, systematic disagreement worth investigating before trusting the classification at face value - not necessarily a sign to abandon AI-assisted coding, but a clear signal to look closely at where the disagreement is concentrated before treating the output as final. Anything lower is worth treating as a serious flag that either the category definitions need real revision, or the specific task is one where AI-assisted coding, for whatever reason, isn't currently performing well for this particular kind of data.
Watch for the Kappa Paradox¶
One statistical quirk is worth knowing before reading too much into a single kappa number: when one category is much more common than the others in your sample, kappa can come out surprisingly low even when the raw percentage of matching answers between the two coders is genuinely high. This happens because kappa specifically corrects for the agreement you'd expect from chance, and when one category dominates, both coders can happen to agree on it a lot just because they're both, individually, likely to pick the common category most of the time - kappa is built to discount that kind of coincidental agreement, which can make an otherwise solid result look weaker than the raw numbers suggest. Reporting both kappa and the plain percent-agreement figure side by side avoids being misled by this in either direction, and is worth doing as standard practice whenever your category sizes are noticeably uneven.
What to Do With a Disappointing Result¶
A weaker-than-hoped agreement figure is genuinely useful information, not just a discouraging one - it's telling you specifically where to look, if you look at which responses the human and AI disagreed on rather than just the summary number. Pulling the disagreement cases and reading them together, the same pattern-finding approach covered in our guide on catching AI miscategorization, often reveals a specific, fixable cause: two category definitions that overlap more than intended, a category whose boundary needs a sharper example, or a genuine edge case in the data that would have been hard for any coder - human or AI - to place confidently. Revising the category definitions based on what the disagreement pattern reveals and rerunning the check on a fresh sample is a more productive response than either abandoning AI-assisted coding outright or ignoring a weak result and proceeding anyway.
FAQ¶
Do I need statistics software to calculate Cohen's Kappa?
The calculation itself is straightforward enough to do in a spreadsheet with a documented formula, though dedicated statistics tools make it faster, especially for datasets with more than two or three categories. Plain percent agreement, while less rigorous, is calculable with nothing more than a simple count and requires no special tooling at all.
How does this differ from the confidence scores AI classification already provides?
A confidence score reflects how certain the AI was about one specific assignment, generated at the same time as the classification itself. An inter-rater reliability check is an independent, after-the-fact validation against real human judgment on the same data - a different, more rigorous kind of evidence, and the two are complementary rather than substitutes for each other.
Should I run this check every time, or just once?
For an ongoing, tracked survey using a stable set of category definitions, running it once when the categories are first finalized is usually sufficient, with a lighter re-check if the category definitions change meaningfully. For a one-off, high-stakes analysis, running it on that specific dataset is worth doing regardless of past results on different data.
What if my human coder and the AI disagree on genuinely ambiguous responses that don't have a clean right answer?
That's a normal, expected part of qualitative data, not a failure of either coder - the same genuine ambiguity would likely produce disagreement between two human coders too. It's worth distinguishing this from a systematic pattern in your review, since genuine ambiguity doesn't need a definition fix, just an documented acknowledgment that some responses don't cleanly resolve either way.
For the research context behind this practice, see Can You Trust AI to Analyze Qualitative Data? What the Research Says and How Many Human-Coded Responses Do You Need to Validate an AI Classifier?.