Golden set

• You need fast, repeatable checks before shipping changes.
• Support or compliance demands proof that quality isn’t slipping.
• You are benchmarking providers or prompt variants.

Golden sets anchor quality discussions. PMs decide composition (edge cases, critical intents), ownership, and refresh cadence. A too-small or stale set gives false confidence; a well-maintained one reduces incidents and accelerates approvals.

Collect real user queries and failure cases; anonymize and label with expected outcomes and rationales. Keep size lean (50–200 per feature) and refresh monthly. In 2026, semi-automate refresh with sampled production traffic reviewed by humans.

A support bot’s 120-case golden set catches a regression where refunds above $500 are denied incorrectly. Fix ships before rollout, avoiding a spike in tickets and refunds backlog.

• Letting the set age past major product changes.
• Focusing only on common cases and missing rare but risky ones.
• Not storing rationales, making labels hard to audit or update.