Skip to content

Data & Model Monitoring (Evidently)

Status: 📋 Planned — architecture designed; Evidently not yet integrated or deployed.

No drift detection is active today. The service is monitored via Prometheus metrics only — see Metrics.

Intended usage

Evidently will be used to: - detect feature distribution drift against the training baseline, - monitor prediction distribution changes over time, - generate offline batch reports as a first step.

Planned monitored signals

Data drift

  • Feature distribution shift (PSI / KS test) against training reference.
  • Missing or unexpected values.
  • Schema changes.

These connect to the data pipeline in src/data/ and the feature engineering layer in src/features/.

Prediction drift

  • Output probability distribution shift.
  • Class balance changes.

Ground truth evaluation

  • Match results arrive ~90 minutes after kick-off.
  • Full accuracy evaluation requires a ground truth feedback loop — planned separately from drift detection.

Offline-first plan

  1. Batch Evidently report on recent predictions vs training baseline (initial phase).
  2. Export key drift signals as Prometheus metrics (future phase).
  3. AlertManager rule on confirmed drift signal (future phase).