Ask an AI classifier to sort a batch of open-ended responses into themes with nothing more than "find the themes in this data," and it will absolutely do that - it just won't necessarily find the themes that matter to your specific question. A support-ticket dataset handed a vague goal might come back split by which product area each ticket mentions, when what you actually needed was a split by how urgent or resolvable each issue was. Nothing about that result is wrong, exactly. It's just answering a different, more generic question than the one in your head, and the gap between the two is almost entirely down to how specifically you described what you were looking for.
Table of Contents¶
- Why a Vague Goal Produces Generic Categories
- What a Good Classification Goal Actually Contains
- Before and After: Five Rewrites
- How Broad or Narrow Should Your Categories Be
- Iterating After the First Pass
- FAQ
Why a Vague Goal Produces Generic Categories¶
When you let AI propose a starting set of categories from your actual responses, it's working from two things: the text of the responses themselves, and whatever goal or intent you gave it to guide the sorting. Without a specific goal, it defaults to the most statistically obvious way to split the data - usually topic (what's being talked about) rather than the more specific angle you actually need, like severity, sentiment, root cause, or which team should own the follow-up. Responses about a product's checkout flow, sorted with no real goal, tend to come back grouped as "checkout issues" broadly, when what a product team usually needs is checkout issues split by whether they're a bug, a confusing design choice, or a missing feature - three categories that call for three completely different fixes.
This isn't a shortcoming of the classification step - it's just what happens when a system that's good at finding patterns is given an underspecified question. The fix isn't a smarter AI, it's a clearer goal.
What a Good Classification Goal Actually Contains¶
A classification goal that reliably produces useful categories tends to include three things, even in a single sentence. First, the angle - not just "what are people talking about" but specifically what dimension matters: root cause, urgency, sentiment, feature area, decision stage, whatever the actual business question is. Second, a sense of granularity - roughly how many categories you're expecting, or how fine- versus coarse-grained they should be, since "find the themes" with no granularity hint can come back as three broad buckets or twenty narrow ones depending on how the data happens to cluster. Third, anything already known worth anchoring against - if you already suspect pricing and onboarding are likely themes, saying so helps the categories align with language your team already uses internally, rather than generating a parallel vocabulary that needs translating before anyone can act on it.
None of this needs to be more than a sentence or two. "Categorize by the underlying reason for dissatisfaction, not just the topic mentioned - I want to know if it's a bug, a missing feature, confusing design, or something else, roughly five to eight categories" does more work than "find the themes" ever will, despite being barely longer.
Before and After: Five Rewrites¶
Vague: "Find themes in these employee engagement comments."
Specific: "Categorize by the underlying driver of engagement or disengagement mentioned - management quality, workload, growth opportunity, compensation, culture, or recognition - even if the comment also touches on a specific incident."
Vague: "Sort these NPS verbatims."
Specific: "Split promoters' and detractors' comments separately, and within each, categorize by which product area or experience the comment is actually about, aiming for categories specific enough that a team lead would know who owns the follow-up."
Vague: "Categorize this product feedback."
Specific: "Categorize by feedback type - bug report, feature request, usability complaint, or praise - rather than by which part of the product is mentioned, since the team needs to route these to different owners."
Vague: "Analyze these exit interview responses."
Specific: "Categorize by the primary reason for leaving, distinguishing reasons the company could realistically have prevented (management, workload, career growth) from reasons it couldn't (relocation, career change, retirement)."
Vague: "Find patterns in this customer support feedback."
Specific: "Categorize by urgency and resolvability - clearly resolved satisfactorily, resolved but respondent still frustrated by the process, or unresolved - rather than by which product feature the ticket concerned."
In every case, the specific version isn't longer because it's trying harder - it's longer because it's naming the actual axis of interest instead of leaving the system to guess at it from the topic alone.
How Broad or Narrow Should Your Categories Be¶
A goal that produces three sprawling categories and a goal that produces twenty narrow ones can both come from the same dataset, depending on how the granularity is framed - and neither is wrong in isolation, but one is probably wrong for your specific report. Broad categories (four to seven, say) are easier to chart, easier to present to leadership, and more stable wave over wave, but they can blur together distinct problems that deserve separate attention. Narrow categories (fifteen or more) preserve more nuance and are more useful for a team that's going to act on individual findings directly, but they're harder to summarize at a glance and more prone to a handful of categories ending up with only a handful of responses each, too thin to say much with confidence.
A reasonable default is to ask for a number in the general range you actually plan to report on - if the result is going into a five-bar chart for an executive summary, say so; if it's feeding a detailed team-level action list, say that instead. The category count is one of the easiest things to steer directly, and doing so up front saves a full re-run later.
Iterating After the First Pass¶
Even a well-written goal rarely produces a perfect category list on the very first attempt, and that's fine - the review step exists specifically because the first pass is a strong starting point to react to, not a finished product to accept blindly. If two proposed categories turn out to be measuring close to the same thing, merge them and rerun, or just relabel one before classifying. If a category is a grab-bag that's too broad to be useful once you see actual responses sorted into it, split it with a more specific follow-up goal, focused just on that subset. Treating the first classification as a draft - one you're allowed to argue with and refine - produces meaningfully better results than treating whatever comes back on the first try as final, especially for a dataset you're planning to use as an ongoing, wave-over-wave tracked category set.
FAQ¶
How long should a classification goal be?
One to three sentences is usually enough - the goal is to name the angle, rough granularity, and any known anchoring themes clearly, not to write an exhaustive brief. A longer goal isn't automatically better if it's still vague about the actual dimension you care about.
Should I mention specific category names I expect to see?
Yes, if you already have a strong hunch - naming a couple of expected categories helps anchor the language to terms your team already uses, without forcing every response into only those categories, since the system still has room to propose others it finds in the data.
What if my goal produces categories that don't match what I expected at all?
That's useful information in itself - either the data genuinely doesn't split the way you assumed, or the goal needs to be more specific about the angle you actually meant. Reviewing a handful of responses in an unexpected category usually clarifies which one it is quickly.
Can I change my classification goal after reviewing the first result?
Yes - rerunning with a refined goal, or building a second, more targeted classification scoped to a subset of the first, is a normal and expected part of the process rather than a sign anything went wrong the first time.
For the full workflow this fits into, see our introduction to Text Analytics.