Agent Evaluation Set Starter

A case file template, coverage matrix, grader rubric with grader instructions, and release gate for evaluating an AI agent.

Decide whether a prompt, model or tool change ships from graded cases and a written gate, not from a few manual tries.

Template · AI Agents · PDF, 14 pages · Free · Updated 2026-11-22

What you will accomplish

  • Write a versioned case file where every case names its input, fixtures, expected behavior, assertions, grader and source.
  • Cover core tasks, edge inputs, refusals, tool use, prompt injection, data boundaries, long conversations and past incidents with graded cases.
  • Run a release gate that compares each prompt, model or tool change with the production baseline and records a ship or hold decision.
  • What is inside

  • Evaluation case template
  • Coverage matrix
  • Grader rubric and grader instructions
  • Release gate checklist
  • Written for: AI product teams; Agent and LLM engineers; Engineering leads.

    Preview

    Evaluation case template

    Each case states the input, the fixtures it runs against, what correct behavior is, and how it is graded. Shown expanded for reading; store each case on one line of the .jsonl file.

    {
      "id": "tool-001",
      "category": "tool_use",
      "severity": "blocker",
      "source": "spec:[REQUIREMENT OR TICKET ID]",
      "input": [{"role": "user", "content": "[REQUEST THAT NEEDS ONE SPECIFIC TOOL]"}],
      "fixtures": {"user": "[TEST USER A]", "tools": "[TOOL STUB FILE]"},
      "expected": "Calls [TOOL NAME] once with the correct arguments and reports the tool result accurately.",
      "must": ["tool_call:[TOOL NAME]", "arg:[ARGUMENT NAME]=[EXPECTED VALUE]", "end_state:[RECORD AND FIELD]=[EXPECTED VALUE]"],
      "must_not": ["tool_call:[SIDE-EFFECTING TOOL THAT IS NOT NEEDED]", "more than one call to [TOOL NAME]"],
      "grader": {"type": "code", "check": "[CHECK NAME]"}
    }

    Coverage matrix

    Eight categories of cases, what belongs in each, a starting minimum, and the grading method that usually fits.

  • Tool-use correctness
  • Prompt injection via retrieved content
  • Data boundaries between users
  • Regressions from real incidents
  • Release gate checklist

    Run this on every prompt, model, tool or retrieval change. It compares the candidate with the current production baseline and ends in a recorded decision.

  • Write the thresholds before the run
  • Start every trial from a clean state
  • Read every newly failing transcript
  • Hold the release on any blocker failure
  • Download the free PDF

    Tools covered

    Claude

    ChatGPT

    Related reading

    How to launch an AI customer support agent that works

    AI agents that are safe to leave running

    Related resources

    AI Product Launch Checklist: A launch gate for AI features covering evaluation, fallbacks, cost, safety, support, and rollback.

    AI Agent Production Safety Checklist: Controls for tool permissions, data handling, budgets, approvals, observability, and shutdown.

    AGENTS.md Template: A portable instruction template for coding agents working across nested directories.

    Browse the site

  • Home
  • About
  • Story
  • Work
  • Expertise
  • The AI practice
  • AI Product Development
  • AI Agents
  • AI Automation
  • AI Consulting
  • Arabic-first AI Products
  • AI in Kuwait
  • Web Development
  • Claude
  • Lovable
  • Notion
  • Webflow
  • Shopify
  • WordPress
  • AI Solutions
  • Services
  • Business Strategy
  • Growth Planning
  • Tools
  • Resources
  • Blog
  • Listening
  • Books
  • Dubai
  • Stack
  • Contact
  • Get a quote
  • Privacy
  • Terms