Skip to content

Container View (C4 — Level 2)

This view shows deployable containers/services and their responsibilities.

flowchart LR subgraph DataPlane[Data Plane] Airflow[Airflow Scheduler/Workers] DB[PostgreSQL] S3[MinIO / S3] end subgraph MLEngineering[ML Engineering Plane] DVC[DVC Pipelines] MLflow[MLflow Tracking and Registry] end subgraph ServingPlane[Serving Plane] API[FastAPI Inference Service] MQ[RabbitMQ] Worker[Celery Workers] Cache[Redis] UI[Web UI] end subgraph Observability[Observability Plane] Prom[Prometheus] Graf[Grafana] Evid[Evidently] end Airflow --> DB DB --> S3 S3 --> DVC DVC --> MLflow UI --> API API --> MLflow API --> Cache API --> MQ MQ --> Worker Worker --> Cache API --> Prom Worker --> Prom MQ --> Prom Prom --> Graf DVC --> Evid API --> Evid

Container responsibilities

Airflow

  • schedules and runs ingestion/ETL workflows (scraping, normalization, exports)
  • ensures raw data is consistently produced and exported

PostgreSQL

  • stores scraped/normalized structured data
  • provides stable extraction source for raw export jobs

MinIO (S3)

  • stores raw parquet exports and (optionally) ML artifacts
  • acts as DVC remote storage backend

DVC pipelines

  • orchestrate reproducible offline ML steps (features/train/eval)
  • tie outputs to specific data versions and git commits

MLflow

  • tracks experiments: params, metrics, artifacts
  • manages model versions and promotion workflow

FastAPI Inference Service

  • exposes /predict endpoint (sync) and async job submission (optional)
  • loads model via stable model_uri from MLflow registry

RabbitMQ + Celery

  • supports async inference or heavy feature/materialization tasks
  • provides retries and backpressure

Prometheus + Grafana

  • collects and visualizes operational metrics and SLO compliance

Evidently

  • generates drift and quality reports (offline and/or near-real-time)