A regression threshold should compare a candidate with an accepted baseline by behavior and severity, while preserving uncertainty and automatic hold conditions.
Define the baseline and blocking behaviors
The baseline must be a named prompt, model, retrieval snapshot, tool version, and configuration that the team has already accepted. Record its results on the same versioned case set used for the candidate. Then identify behaviors that block release regardless of aggregate score, such as an unauthorized tool action, loss of required citations, a schema violation, or a severe known failure returning.
Less severe behaviors can use per-category tolerances. An overall average can hide a large decline in a small but important category, so report each behavior separately and include the number of cases behind the rate. When a category has only a few examples, the report should show that limited evidence rather than imply precise confidence.
Repeat stochastic cases and preserve variance
One run can overstate improvement or regression when the system samples outputs or tools. Repeat selected cases enough times to reveal instability, use the same run policy for baseline and candidate, and store each result. Anthropic's guide discusses repeated trials and multiple graders for agent evaluation because success on one attempt does not establish reliable behavior.
The hold rule should state what happens when the result is close to the threshold. A small apparent gain with wide disagreement may warrant human review rather than automatic approval. The release report should identify the cases driving the decision so a product owner can inspect the underlying behavior.
Store the threshold and related rules as a versioned policy record
Store the threshold, blocked behaviors, weighting, judge versions, human-review sample, and approval record beside the suite. Changing the policy is a separate reviewed action from changing the product. This prevents a team from lowering a threshold inside the same pull request that fails it without making the policy change visible. Store the exact aggregation formula and treatment of missing runs alongside the threshold, because a changed denominator or skipped case can alter the decision without any model behavior changing.
Release Measure prepares the comparison through Reality Contact, LLC. The customer approves the threshold, interprets business severity, and makes the release or hold decision.
Where the service stops
Reality Contact, LLC implements and calibrates the evaluation system, but does not decide product policy, certify model safety, label regulated outcomes, approve deployment, or replace the buyer's domain experts and risk owners. The buyer approves the gold cases, scoring policy, threshold, and hold rules, then replaces the informal release check with the accepted suite for the next AI change. This is technical evaluation implementation; it does not replace domain-expert, legal, regulatory, security, safety, or product-policy review. We do not promise coverage of every behavior, a defect-free model, evaluator agreement, successful deployment, or approval of any release.
Sources: Anthropic guidance on agent evaluation trials and graders; Braintrust platform pricing for evaluation scores.