Data & Model Monitoring (Evidently)¶

Status: 📋 Planned (Architecture designed, implementation pending)

Why model monitoring is required¶

A model can fail silently even when: - the service is healthy, - latency is acceptable.

Monitoring must therefore extend beyond infrastructure.

Planned Evidently Usage¶

Evidently will be used to: - detect data drift, - monitor prediction distribution changes, - track proxy quality metrics.

Monitored Signals (Planned)¶

Data drift¶

• feature distribution drift (PSI / KS),
• missing or unexpected values,
• schema changes.

Prediction drift¶

• output distribution shift,
• confidence score degradation,
• class balance changes.

Offline vs near-real-time monitoring¶

• Offline reports (planned):
• generated on batches of recent predictions,

• reviewed periodically or on alerts.

• Near-real-time checks (future):

• lightweight drift signals,
• exported as metrics.

Limitations¶

• ground truth labels may arrive with delay,
• monitoring will focus on early warning signals, not full accuracy evaluation.

Delayed labels will be evaluated separately when available.