Agent Evaluation Set Starter
A case file template, coverage matrix, grader rubric with grader instructions, and release gate for evaluating an AI agent.
Decide whether a prompt, model or tool change ships from graded cases and a written gate, not from a few manual tries.
Template · AI Agents · PDF, 14 pages · Free · Updated 2026-11-22
What you will accomplish
What is inside
Written for: AI product teams; Agent and LLM engineers; Engineering leads.
Preview
Evaluation case template
Each case states the input, the fixtures it runs against, what correct behavior is, and how it is graded. Shown expanded for reading; store each case on one line of the .jsonl file.
{
"id": "tool-001",
"category": "tool_use",
"severity": "blocker",
"source": "spec:[REQUIREMENT OR TICKET ID]",
"input": [{"role": "user", "content": "[REQUEST THAT NEEDS ONE SPECIFIC TOOL]"}],
"fixtures": {"user": "[TEST USER A]", "tools": "[TOOL STUB FILE]"},
"expected": "Calls [TOOL NAME] once with the correct arguments and reports the tool result accurately.",
"must": ["tool_call:[TOOL NAME]", "arg:[ARGUMENT NAME]=[EXPECTED VALUE]", "end_state:[RECORD AND FIELD]=[EXPECTED VALUE]"],
"must_not": ["tool_call:[SIDE-EFFECTING TOOL THAT IS NOT NEEDED]", "more than one call to [TOOL NAME]"],
"grader": {"type": "code", "check": "[CHECK NAME]"}
}
Coverage matrix
Eight categories of cases, what belongs in each, a starting minimum, and the grading method that usually fits.
Release gate checklist
Run this on every prompt, model, tool or retrieval change. It compares the candidate with the current production baseline and ends in a recorded decision.
Tools covered
Related reading
How to launch an AI customer support agent that works
AI agents that are safe to leave running