How I design evaluation systems for AI outputs
This is how I design a system that asks a question of a model many times and gets an answer you can trust twice. I built and ran it on a private measurement platform, and the method generalizes past that use case. No client specifics appear here.
What is being measured?
Whether an AI engine, asked a question a real customer would ask, produces an answer that mentions a given entity and cites a given source. Everything below exists to make that question answerable the same way twice.
How are prompts constructed?
Prompts are grouped into topic clusters rather than written as a flat list. A cluster is a set of questions a buyer would ask at the same moment of intent, and grouping them is what makes an aggregate number interpretable: "visibility went up" is meaningless, while "visibility went up in the comparison cluster and not the problem-awareness cluster" tells a comms team what to do next.
Three rules I would keep on any implementation. Write prompts in the language a customer uses, not the language the brand uses about itself. Include prompts the entity should lose, because a set the entity always wins measures the set, not the entity. And version the set, because an unversioned prompt library makes week-over-week comparison silently unreliable.
Which engines, and how often?
All six of ChatGPT, Claude, Perplexity, Gemini, Google AI Overviews, Google AI Mode, on every run. Partial engine coverage does not give a smaller version of the right answer, it gives a different answer, because share of voice computed across an incomplete engine set is a different quantity from share of voice.
Cadence on the client platform was 3 runs a day. The reason for more than one is variance: these systems do not return the same answer to the same question, so a single daily sample confuses noise with movement. Multiple runs a day let you distinguish a real shift from a resampling artefact.
Why collect from the browser rather than the API?
Because the API does not return what a person sees. The consumer-facing surfaces produce different answers, with different citations, than their API equivalents, and the citation set is the entire point. Measuring the API means measuring a product nobody uses.
That is why collection runs through headless browser sessions on Playwright and Apify. It is also the most fragile part of any system in this category: engines change their answer surfaces without notice, and every change is a silent break. Any honest description of this work says that collection maintenance, not analysis, is the recurring cost.
How is a competitor set defined?
Explicitly, and in advance. A competitor set assembled after the fact from whoever happened to appear is a description of the results, not a control on them. The set should be the competitors the business actually names in its own strategy, plus any entity that repeatedly appears in answers to the tracked prompts, added deliberately and dated.
What do the metrics mean?
- Response presence
- How often an AI engine returns any answer to a tracked prompt in the brand's topic area.
- Citation rate
- How often the engine's answer cites the brand's own or earned content.
- Share of AI voice
- The brand's share of mentions across a competitor set within answers to the same prompts.
- Citation and source mapping
- Which publishers, owned pages, and third-party sources each engine pulls from for brand-relevant prompts, and how that mix shifts after an earned media push.
- Brand descriptor shift
- Change over time in the language models attach to the brand.
Each definition is written to survive being extracted alone, because that is how it will be encountered: quoted in a slide, without the paragraph around it.
Where does this methodology strain?
Non-determinism sets a floor on precision. The same prompt to the same engine on the same day can return different answers. Everything here is a sampling estimate, and any product presenting a single number without a sense of its variance is overstating what it knows.
Personalization is unobservable from outside. Collection runs from a session that is not any particular customer's session. What a logged-in user with history sees may differ, and no external measurement system can see that.
Citation is not attribution. An engine citing a source does not establish that the source shaped the answer, and an engine not citing a source does not establish that it was not used. The metric is what is shown, which is the right thing for a comms team to act on, but it is not a claim about the model's internals.
Entity disambiguation is unsolved at small scale. The method assumes the tracked entity resolves cleanly. For a household-name brand it does. For a person, or a brand sharing a name with a common noun, the competitor set silently fills with unrelated entities and the resulting share figures mean less than they appear to.
What would I build differently next time?
Version prompt sets from the first run, not the fiftieth. Instrument run integrity before scaling cadence, so a partial failure announces itself rather than surfacing as an odd number in a client report. And keep metric definitions separate from their computation, so improving a definition is a configuration change rather than a deploy.