Skip to content

Audit Cycle Summary — SoccerPredictAI (Baseline)

Date: 2026-04-24 Auditor: GitHub Copilot (Claude Sonnet 4.6) — full audit cycle Cycle scope: Audits 00 → 11 (audit 12_docs_validation did not yet exist in this cycle) Baseline reference: None — this cycle is itself the first reference baseline. One-line verdict: Working end-to-end stack with known operational gaps — 8 system-level risks open; smoke-mode training params block production deployment.

This summary was produced retroactively on 2026-04-28 to provide a baseline scorecard for delta comparisons. The methodology follows the version of SKILL.md step 7 in force on that date. The 11 audit reports themselves were not modified.

1. Per-audit reports

# Audit Report One-line outcome
00 System 00_system_audit_v2.md 12-layer architecture confirmed; 8 system risks (R1–R8) opened
01 Data 01_data_audit.md Ingestion + GE gates working; manual export; validate_interim contract drift
02 Features 02_feature_audit.md Train↔inference parity for offline; online predict relies on client features
03 Training & Eval 03_training_evaluation_audit.md Smoke training params (fracs=[0.001, 0.002], n_trials=2)
04 Pipeline DVC + Hydra 04_pipeline_dvc_hydra_audit.md 15-stage DAG; Hydra unused; missing validation deps
05 MLflow Registry 05_mlflow_registry_audit.md matches_clf_smoke active; no champion gate, no rollback
06 Train ↔ Serve 06_train_serve_consistency_audit.md Offline parity ✅; online client-supplied; no hot-reload, no staleness guard
07 Serving 07_serving_audit.md 13 endpoints; /predict/* unauthenticated; no model hot-reload
08 Orchestration 08_orchestration_audit.md 5 ETL DAGs; no DAGs for dvc repro / batch_inference / export
09 UI 09_ui_audit.md Only Livescores page; pages/ empty; no prediction UI
10 Ops / Security / Obs 10_ops_security_observability_audit.md All replicas=1; CORS *; Grafana/Evidently not deployed
11 Docs & Tests 11_docs_testing_audit.md ~294 tests; contract test references missing validate_interim
12 Docs Validation — (not run this cycle) Audit introduced in cycle 20260428

2. Best-practices compliance scorecard

Methodology per SKILL.md §7 (Summary-table mapping ✅=1.0, ⚠ / partial=0.5, ❌=0.0).

# Audit Compliance %
00 System 91.7
01 Data 57.1
02 Features 50.0
03 Training & Eval 57.1
04 Pipeline DVC + Hydra 66.7
05 MLflow Registry 62.5
06 Train ↔ Serve 50.0
07 Serving 61.1
08 Orchestration 60.0
09 UI 60.0
10 Ops / Security / Obs 44.4
11 Docs & Tests 41.7
12 Docs Validation n/a
— Overall (12 audits) 58.5

3. Consolidated risk register (P0 / P1, deduplicated)

Rank Severity ID Title Owning audits
1 P0 R1 Smoke fracs_for_train + n_trials=2 in params.yaml 00, 03
2 P0 R2 No automated retrain trigger; manual dvc repro 00, 08
3 P0 R5 No DAG for batch_inference → stale serving features 00, 06, 08
4 P0 D-01 validate_interim in contract test absent from dvc.yaml (CI-red) 01, 11
5 P0 D-03 Manual etl_export_matches_to_source 01, 08
6 P0 UI-01 src/ui/app/pages/ empty — no prediction UI 09, 11
7 P1 R3 No model hot-reload on champion change 00, 05, 06, 07
8 P1 R6 No champion-vs-challenger gate 00, 05
9 P1 R8 All replicas=1; HPA disabled 00, 10
10 P1 SRV-01 /predict/* and /monitoring/* unauthenticated 07, 10
11 P1 OPS-03 CORS allow_origins=["*"] 10
12 P1 R7 No drift detection (Evidently not wired) 00, 10
13 P1 OPS-04 Grafana not deployed in K8s 10
14 P1 OR-04 No DAG-level alerting 08
15 P1 OR-05 No automatic retrain trigger 08
16 P1 TR-03 Holdout used for model selection 03
17 P1 TR-04 ablation_study isolated from selection path 03
18 P1 F-01 No server-side feature contract on online predict 02, 06
19 P1 F-02 No runtime check classification.window_sizes ⊆ features.window_sizes 02
20 P1 D-02 match.parquet dead artifact 01, 04
21 P1 D-04 MinIO ETag (multipart) ≠ content hash 01
22 P1 R4 stats.py router not registered 00
23 P1 P-01 split_data dep misnamed 04
24 P1 P-02 ablation_study lacks test_ids.parquet dep 04
25 P1 P-03 validate_* stages lack src/data_quality/*.py deps 04
26 P1 P-04 Hydra conf/ shipped but unused 04
27 P1 ML-01 matches_clf_smoke is active production experiment 05
28 P1 UI-02 APIClient lacks /predict/* methods 09

4. Top must-fix items (≤ 10, ordered)

  1. Fix CI-red contract test (D-01) — remove validate_interim from EXPECTED_STAGES or add the missing DVC stage.
  2. Restore production training params (R1) — real fracs_for_train and tuning.n_trials.
  3. Implement prediction UI page or downgrade docs/status.md (UI-01).
  4. Add freshness guard on match_features.parquet (R5).
  5. Add model hot-reload on champion change (R3).
  6. Add champion-vs-challenger gate (R6).
  7. Automate the retrain loop (R2 / OR-01..03).
  8. Add auth + tighten CORS on inference endpoints (SRV-01 / OPS-03).
  9. Fix DVC validation deps (P-01, P-03).
  10. Refresh docs/status.md with corrected status labels.

5. Open questions / unverified areas

  • Current MLflow champion (soccer_clf@champion) — actual run, metrics, lineage.
  • Last batch_inference run timestamp on match_features.parquet.
  • Evidence directory placeholder check.
  • Celery worker liveness probe verification.