Deployment and AI CI/CD
Ship AI systems with preview checks, evaluations, canaries, kill switches, and rollback.
Key takeaways
- AI deployment adds quality and safety gates to normal software checks because a passing build does not prove acceptable model behavior.
- Release gates span typecheck and build, tests, prompt and schema evals, retrieval evals, safety checks, preview review, and canary on limited traffic.
- Version prompts, tools, and evaluation datasets, and keep kill switches for high-risk AI flows.
- Separate provider config from application code and attach release notes to behavior changes.
- Monitor quality and cost immediately after each release.
AI deployment needs normal software checks plus quality and safety checks. A passing build does not prove that model behavior is acceptable.
AI Release Gates
| Gate | Purpose |
|---|---|
| Typecheck and build | Software correctness |
| Unit and integration tests | Deterministic behavior |
| Prompt and schema evals | Output quality and format |
| Retrieval evals | Grounding and permission filtering |
| Safety checks | Policy and injection resistance |
| Preview review | UX and stakeholder validation |
| Canary | Real traffic with limited blast radius |
Deployment Rules
- Version prompts, tools, and evaluation datasets.
- Keep kill switches for high-risk AI flows.
- Separate provider config from application code where possible.
- Attach release notes to behavior changes.
- Monitor quality and cost immediately after release.