Auditing AI Classification Accuracy on Your Own Data¶

"The AI classified it" isn't an answer that holds up when a client or stakeholder asks how confident you are in a finding. Before you report Text Analytics results as a headline number, it's worth running a deliberate audit - not because the tool is unreliable, but because you should be able to say specifically how you checked, not just that you trust it.

Step 1: Start with the Confidence Distribution¶

Look at how classified responses are spread across confidence levels for the question you're auditing. A question where most responses landed at high confidence is a different starting point than one where a large share landed low - see Understanding AI Confidence Scores in Text Analytics for what these numbers represent. A heavily low-confidence question deserves more manual review before you trust it, not less.

Step 2: Pull a Random Sample, Not Just the Flagged Ones¶

Reviewing only the low-confidence responses tells you about the hard cases, but not whether the easy ones are actually easy. Pull a genuine random sample across all confidence levels - including high-confidence ones - and check them against the category definitions yourself. If high-confidence responses are consistently correct in your sample, that's real evidence you can cite. If they're not, that's worth knowing before you report anything.

Step 3: Check the Category Definitions Themselves¶

A lot of what looks like a classification error is actually an ambiguous or overlapping category definition - two categories close enough that a genuinely borderline response could reasonably go either way. If your audit turns up several "wrong" classifications that all trace back to the same fuzzy category boundary, the fix is tightening the category (see Editing and Refining Categories in Text Analytics), not just recoding responses one at a time.

Step 4: Correct What Needs Correcting¶

For any response your audit found misclassified, correct it directly - see Reviewing and Correcting Individual AI Classifications. Keep a rough count of how many corrections you made relative to your sample size; that ratio is useful context to have on hand if anyone asks how much manual review went into the final numbers.

Step 5: Document What You Checked¶

For a client-facing or high-stakes deliverable, a short note on your audit process - sample size, what you checked, what you corrected - is worth keeping even if you don't publish it. It's the difference between "we used AI" and "we used AI and verified it," which is a materially stronger claim if anyone pushes back on a finding.

FAQ¶

How big should my audit sample be?
There's no universal number - it should be large enough that you'd trust a pattern found in it, and larger for a question where the stakes of getting it wrong are higher.

Should I audit every question, or just the ones going into a final report?
At minimum, audit anything that will directly support a conclusion you're reporting. A question you're using only for internal exploration can reasonably get a lighter check.

What if my audit finds a high error rate?
Revisit your category definitions first - a high error rate is more often a sign of ambiguous categories than of the classification step failing outright. Tighten the categories, then re-run classification and re-audit before trusting the results.

Does auditing and correcting responses cost credits?
No - reviewing and manually correcting classifications is free. Only category generation and classification runs themselves are credit-based.

For the underlying workflow this audit process sits on top of, see How Text Analytics Works: From Open-Ends to Categories.