Regression testing (LLM)

• Every prompt/model/corpus change going to production.
• Rolling out new tool integrations or schemas.
• Upgrading model versions or providers.

Regression tests protect reliability and reduce firefighting. PMs decide thresholds for blocking releases and how to weigh quality versus latency/cost impacts. Strong regression suites increase shipping cadence and stakeholder confidence.

Reuse your golden set, add assertions for safety, format, and latency. Automate in CI; fail builds on critical regressions. In 2026, add differential tests that compare old vs. new paths and show deltas on cost and speed alongside accuracy.

A model upgrade improves accuracy but adds 300 ms. Regression tests flag the latency hit; PM approves because SLA is 1.8 s and quality gain is worth it, avoiding a rollback loop post-launch.

• Testing only accuracy, ignoring latency or cost regressions.
• Allowing flaky tests that teams start ignoring.
• Not updating tests after major UX or policy changes.