How do you know an AI agent is ready? A practical evaluation framework

A demonstration is not deployment evidence

Demos are usually selected, observed and retried. Real work is not. Inputs arrive incomplete, sources disagree, systems time out and users express the same need in different ways. The question is not whether the agent has succeeded once, but how reliably it succeeds across the conditions that define the intended workflow.

Agent evaluation is particularly difficult because tool calls and intermediate decisions create multiple places for error. Mistakes can propagate across turns, while probabilistic outputs mean the same task may not follow the same path twice (Anthropic, 2026). A release decision based on a handful of favourable examples provides little information about that variation.

The minimum evidence is a repeatable evaluation: defined tasks, controlled test conditions, explicit acceptance criteria and recorded results. This converts “it looks good” into a claim that can be examined and challenged.

Define the job and its failure boundaries

Evaluation begins with a bounded job. “Conduct research” is too broad. A more useful specification might be: produce a decision brief from authorised public sources, support material claims with traceable citations, identify conflicting evidence and escalate when reliable evidence is insufficient.

Define failure with equal care. For a Research Agent, failures include an unsupported conclusion, a citation that does not support its claim, omission of material contrary evidence, use of an inaccessible source, or confident completion when the brief cannot be answered. These are not merely model-quality problems; they are failures of the whole system, including instructions, tools, retrieval, permissions and interface.

NIST recommends that testing reflect intended purpose, deployment context and identified risks, and that test methods and results be documented (NIST, 2023). The practical implication is simple: readiness criteria should be written before the team becomes attached to the prototype.

Evaluate outcomes, not polish

A fluent answer is not necessarily a successful outcome. Test the state that matters after the agent finishes. If the job is research, ask whether the brief answers the question, covers the required evidence, represents uncertainty accurately and lets a reviewer verify important claims. If the agent is allowed to change a system, inspect whether the correct change occurred—not whether the agent said it occurred.

OpenAI describes contextual evaluations as a way to translate business objectives into measurable expectations specific to a workflow (OpenAI, 2025). That is more useful than a generic benchmark. A model may score well on broad capability tests and still fail because an organisation’s sources, policies or acceptance rules are different.

Outcome measures should combine hard checks with judgement. For the Research Agent, hard checks can confirm that every citation resolves and that required sections exist. Domain review is still needed to assess whether the synthesis is balanced, material evidence is missing or a conclusion overreaches.

Evaluate the trajectory

The final result tells you what happened; the trajectory helps explain why. Record the tools called, sources retrieved, intermediate states, errors, retries and stopping decision. Microsoft’s agent evaluators similarly distinguish end-to-end task completion from the quality and efficiency of workflow steps (Microsoft, 2026a).

This does not mean showing private model reasoning to users. It means retaining an inspectable execution record. A correct brief reached through repeated failed searches and an unauthorised source is operationally different from the same brief reached through an efficient, compliant route. Trajectory review can reveal tool-selection errors, loops, ignored evidence and accidental success that an outcome score misses.

Build a representative evaluation set

An evaluation set should reflect the distribution of work the agent will face, not a collection of convenient prompts. Begin with normal tasks, then add edge cases, ambiguous requests, missing inputs, conflicting sources, tool failures and attempts to exceed the agent’s boundaries. Anthropic suggests that 20–50 tasks drawn from real requirements and failures can provide a useful starting point in early development, with larger sets needed as the system matures (Anthropic, 2026).

For the illustrative Research Agent, the set might include:

• a normal question with sufficient, consistent evidence;
• a question where credible sources disagree;
• a question with important missing evidence;
• a relevant source that is inaccessible or cannot be parsed;
• a draft containing an unsupported conclusion;
• a citation that points to a source but not to supporting evidence; and
• a case where the correct outcome is escalation rather than an answer.

Run important tasks more than once. A single successful trial measures possibility; repeated trials begin to measure consistency. Keep a separate regression set for behaviours that must not deteriorate when prompts, tools or models change.

Use the right evaluation methods

No single grader is sufficient. Deterministic checks are appropriate for objective conditions such as valid URLs, required fields, tool permissions and correct database state. Model-based graders can apply a detailed rubric to coverage, groundedness or instruction-following at scale, but they need calibration. Human reviewers remain important where expertise, materiality or nuanced judgement defines quality.

Use each method where it is strongest. A program can verify that a citation exists; a reviewer may need to decide whether it genuinely supports the claim. A model grader can flag possible unsupported statements; a domain expert should sample its decisions and resolve disagreement. Anthropic recommends combining code-based, model-based and human grading rather than treating one layer as conclusive (Anthropic, 2026).

Evaluation data also needs maintenance. Record the system version, model, instructions, tools and test-set version. Otherwise, a changed score may reflect changed conditions rather than a better or worse agent.

Include operational quality

Accuracy alone does not determine whether an agent is usable. Measure completion rate, human correction, latency, cost per completed task, tool errors, retries and escalation rate. Track distributions, not only averages: an acceptable median can conceal a small number of very slow or expensive runs.

The target depends on the workflow. A research brief may tolerate minutes of latency but not unverifiable claims. A high-volume internal triage task may accept bounded uncertainty but require predictable cost. Operational thresholds should therefore express the service the organisation is prepared to run, not an abstract standard of model performance.

Scale controls to consequence

Controls should reflect what can go wrong and how easily it can be reversed. Read-only access, narrow tool permissions, step limits and spending limits reduce the available failure surface. High-impact or irreversible actions need explicit human approval, while uncertainty should trigger an escalation path rather than improvised completion.

Microsoft recommends least-privilege access and human gates for actions affecting people, money, compliance or other difficult-to-reverse outcomes (Microsoft, 2026b). OpenAI likewise recommends human intervention for high-risk actions or repeated failures (OpenAI, n.d.).

Make a defensible release decision

A readiness review should end with an explicit decision: release, release with restrictions, continue the pilot, or stop. A single composite score can hide a critical failure, so define non-negotiable gates separately from improvement targets. An agent that usually writes excellent briefs but occasionally invents citations has not passed a citation-integrity gate.

Do not turn an early score into a universal promise. State the evaluated population, the number of trials and the confidence appropriate to the evidence. Compare the agent with the current workflow as well as the previous system version. If a narrower configuration performs the job more consistently, costs less and needs fewer interventions, it may be the better release even when a more autonomous version appears more capable in selected demonstrations.

The record should state what was tested and what was not. It should identify who accepts the residual risk and when the decision will be revisited. This turns readiness from a subjective impression into a bounded management judgement.

Continue evaluation after deployment

Pre-release tests cannot cover every real input or system interaction. NIST calls for AI systems to be tested before deployment and regularly in operation, with production behaviour monitored and post-deployment response plans maintained (NIST, 2023).

Monitor the same core measures used before release, sample trajectories, collect structured user feedback and add confirmed failures to the regression set. Re-evaluate after changes to the model, instructions, tools, data sources or operating policy. Production monitoring detects new conditions; evaluation determines whether a proposed fix improves the system without breaking established behaviour.

An agent is ready when evidence supports a specific, controlled use—not when uncertainty disappears. The responsible release is the smallest scope in which the agent’s job is clear, its important failures are tested, its controls match the consequence and its performance can be observed after launch.

A demo asks, “Can the agent do this?” Readiness asks, “What evidence justifies letting it do this work under real conditions?”