Experiment Tracking (MLflow)¶

Purpose¶

Document what MLflow is responsible for in this system, how runs are structured, what each training stage logs, and how traceability back to data and code is maintained.

MLflow is the observability and traceability layer for all ML training stages. It is not used for serving, feature storage, or monitoring.

Responsibilities¶

Experiments¶

Two experiments are used across all pipeline stages:

Ablation variants (elo_only, stats_only, full_no_h2h, full) are not separate experiments. They run inside the same experiment and are identified by run-level tags (pipeline.variant, features.profile). This keeps the experiment list short and makes cross-variant comparison straightforward from a single experiment view.

The experiment name is controlled by params.yaml → classification.experiment_name (shared with ablation.experiment_name).

Run structure¶

Run hierarchy¶

All pipeline stages use a two-level hierarchy where applicable:

• Parent run (pipeline.scope=parent): groups a set of related child runs; holds summary params and pipeline context tags.
• Child run (pipeline.scope=child): one run per model or Optuna trial; holds metrics, model artifact, and full lineage tags.

Standalone runs (final_train) use pipeline.scope=parent with no children.

Standard run tags¶

Every run (parent and child) carries:

Data lineage tags (child runs):

Run name format¶

classification_models stage¶

One parent run per frac/variant combination, with nested child runs per model. Selects the best model by holdout log-loss and writes the winning run ID to data/models/run_id.json.

ablation_study stage¶

Runs inside the same matches_clf experiment as classification_models. Each feature subset gets its own parent run tagged with pipeline.variant and features.profile for filtering.

tune_xgb stage¶

One parent run ({run_kind} | tuning | frac={frac}), with one nested child run per Optuna trial. Each trial child run logs: - XGBoost parameters (xgb.*) - Mean CV log-loss across walk-forward folds (cv.logloss_mean) - Trial number

Parent run logs the best parameters (best.*) and best CV log-loss at study completion.

final_train stage¶

The definitive training run. This is the only run that evaluates on the held-out test set.

Tags logged:

Parameters logged:

Metrics logged:

• Holdout log-loss, accuracy, Brier score
• ECE (calibration error) — before and after calibration when calibration is enabled
• Precision, recall, F1 per class

Artifacts logged:

• model/ — the serialised sklearn pipeline (or calibrated wrapper) as MLflow pyfunc
• Calibration curves
• Confusion matrix (multi-class)
• Feature importance plot

Dataset input logged:

• MLflow dataset object pointing to the training parquet with source path

Lineage chain¶

Each final_train run is traceable through:

MLflow run
  ├── tag: data.version = DVC hash of data/processed/dataset.parquet
  ├── tag: pipeline.git_sha = git short SHA
  ├── tag: pipeline.params_hash = SHA-256 of params.yaml
  ├── MLflow dataset input: source = filesystem path to dataset.parquet
  └── params: best.* = tuned hyperparameters from xgb_best_params.json

Given an MLflow run ID, you can recover: - The exact dataset version (via data.version tag → dvc checkout). - The exact code version (via pipeline.git_sha tag). - The exact configuration (via logged params + params.yaml in git).

Filtering runs in the UI¶

To find all ablation runs: filter by tags.pipeline.stage = 'ablation'

To find elo-only runs: filter by tags.pipeline.variant = 'elo_only'

To find all smoke runs: filter by tags.pipeline.run_kind = 'smoke'

To find the winning final model runs: filter by tags.pipeline.stage = 'final_train'

What MLflow is not responsible for¶

• Serving: models are loaded from the registry by PredictionService; MLflow tracking is not in the hot path.
• Feature computation: features are DVC-tracked Parquet files, not MLflow datasets.
• Monitoring / drift detection: Evidently is the planned monitoring tool; not yet integrated.
• A/B testing: not implemented. Promotion is manual via the registry.

Running the MLflow UI¶

mlflow ui --port 5001

Navigate to the matches_clf experiment to see all training and tuning runs.

Related¶

• Training Pipeline
• Tuning
• Model Registry
• Status