Test
Quality reports
Version 1.0 · Last updated 2026-05-27
Anecdotes tell you the agent is doing well or doing badly. Numbers tell you whether it is getting better or worse. Quality reports give you the numbers, with the same questions every time, so improvement is measurable.
Key concepts
Control questions
Questions you would expect any customer to ask, with the answer you would expect the agent to give. Build the list once, run it many times.
Control answers
The expected answers. Used as the benchmark when scoring.
Report runs
Each time you generate a report, the agent runs every control question. Scores compare each answer to the control answer.
Report score
Average of all question scores. Useful as a quick health indicator over time.
Per-question score
The signal you actually act on. Low-scoring questions point straight at the Source that needs work.
What you can do here
- Add control questions, manually or pulled from top questions in Analyze
- Set the expected answer for each question
- Run a report and view per-question scores
- Compare reports over time to track quality trends
- Inspect which Sources the agent used for a low-scoring answer
When to use it
- After any change to Living Knowledge, to confirm scores held or improved
- On a weekly cadence, as a regular health check
- Before showing the agent’s results to leadership or auditors
How it works
Each control question runs against the live agent. Scores compare the generated answer to the control answer using semantic similarity, not exact match. Small differences in wording get high scores. Real disagreements get low scores.
Frequently asked questions
How do I add a control question?
Open "Test > Quality reports > Control questions" and click "Add". Enter the question, the expected answer, and optionally a topic.
How do I pull questions from real customer activity?
Open "Analyze > Performance" and look at the top common questions. Click the plus icon next to a question to add it to your control set.
How do I run a report?
Open "Test > Quality reports > Reports" and click "Run report". The report appears in the list when it finishes.
What does a low score mean?
The agent's answer disagreed substantially with the control answer. Open the question to see the agent's answer, the Sources it used, and the control answer side by side.
How do I improve a low score?
Find the Source the agent used. Either fix that Source, add a new Source that has the right information, or update the control answer if the agent was actually correct.
How often should I run reports?
Once a week as a baseline. Plus on demand after any major change.