Skip to content

Release & Rollback Policy

Rules governing when and how releases are cut and reverted.

Release Cadence

Environment Trigger Approvals Required
dev Every push to main None
staging Manual trigger after CI passes 1 reviewer
production Manual trigger + quality gates pass 2 reviewers

Quality Gates Before Release

All of the following must pass before a production deploy:

  1. All unit and integration tests green (pytest)
  2. ruff linting and formatting check passes
  3. DVC pipeline reproduces deterministically (dvc repro --dry)
  4. Model metrics meet or exceed champion (see Baseline & Success Metrics)
  5. No HIGH severity findings in container image scan

Rollback Process

Rollbacks are performed manually across three independent layers. Each layer can be rolled back independently as needed.

Service rollback (Helm)

Revert to the last known good Helm release:

helm rollback soccer-api

See Deployment Recovery Runbook for the full procedure.

Model rollback (MLflow)

Revert the champion alias to the previous registered model version:

# Via MLflow UI or CLI — reassign champion alias to prior version

See Model Rollback & Recovery Runbook for the full procedure.

Data rollback (DVC)

Restore DVC-tracked artifacts to a prior commit:

dvc checkout <commit>

Notes

  • Rollbacks are not automated.
  • All rollback decisions require human review.
  • After any rollback, the CI pipeline should be re-run to confirm system state.