A useful gold dataset represents the failures and successful behavior that matter to a release, records how each label was decided, and keeps evaluation cases separate from development examples.
Start from decisions the product must support
Begin the gold set from the release question rather than from whichever traces are easiest to export. List the behaviors that can block release, the acceptable variations, and the user situations where those behaviors appear. Then select successful, failed, and ambiguous cases for each behavior. Include short and long inputs, common traffic, difficult edges, and cases that previously fooled the evaluator.
Each row needs the input or replay reference, expected properties, prohibited properties, failure category, source, collection date, labeler, review state, and any dependency on a tool or retrieval snapshot. Use property-level expectations rather than one preferred output for generative systems. This lets multiple valid responses pass while keeping the behavior under test precise.
Adjudicate disagreement instead of hiding it
Two reviewers should inspect a calibration subset independently. Where they disagree, the team should repair the criterion, mark the case as genuinely ambiguous, or choose an authority for that domain. Forcing every case into a clean label makes the dataset look consistent while moving uncertainty into the release result. Keep the original ratings and the adjudicated label so later reviewers can see how the standard changed.
Anthropic's agent-evaluation guidance separates outcome checks from transcript inspection and recommends inspecting traces to understand how an agent achieved a result. A gold row for an agent can therefore include both the final-state property and process constraints such as prohibited tools, excessive turns, or an action that required approval.
Version the split and protect it from leakage
Separate development examples from the held release set, record the split rule, and version every case change. A prompt author who repeatedly sees the held cases can tune to the exam rather than the product behavior. New production failures should enter a review queue, then move into the appropriate split through a normal dataset change with provenance intact.
Release Measure prepares and reviews this dataset through Reality Contact, LLC. The customer approves domain labels, release significance, and any policy judgment that determines whether a case can block deployment.
Where the service stops
Reality Contact, LLC implements and calibrates the evaluation system, but does not decide product policy, certify model safety, label regulated outcomes, approve deployment, or replace the buyer's domain experts and risk owners. The buyer approves the gold cases, scoring policy, threshold, and hold rules, then replaces the informal release check with the accepted suite for the next AI change. This is technical evaluation implementation; it does not replace domain-expert, legal, regulatory, security, safety, or product-policy review. We do not promise coverage of every behavior, a defect-free model, evaluator agreement, successful deployment, or approval of any release.
Sources: Anthropic engineering guide to evaluating AI agents; LangSmith evaluation documentation for datasets and human feedback.