Skip to content

Security & Secrets Management

SoccerPredictAI follows a secure-by-design approach: secrets are never stored in plaintext, access is bounded by namespace isolation, and every boundary is an explicit trust zone.

Trust Boundaries Summary

Zone Components Trust level Access policy
Public Internet End users, external clients Untrusted TLS only; Pydantic schema validation; no auth today
External operator hosts Selenoid host, Streamlit VPS, host-level Nginx Trusted (operator-controlled) Separate operational boundary; no K8s network membership
Delivery boundary GitLab CI/CD Trusted for delivery only Access to encrypted secrets; no runtime cluster access
K8s cluster — internal All namespaces (ds, soccer-api, monitoring, ingress-nginx) Trusted Namespace-scoped secrets; NetworkPolicy where defined
Untrusted data input WhoScored scraped content Untrusted GE validation at raw stage; parameterized DB inserts

Threat Model

Threat Vector Mitigation
Secret exposure in repository Plaintext .env or credentials committed to git SOPS + age encryption; all secret files stored encrypted
Secret exposure in CI logs Env vars printed during build/test steps GitLab masked variables; decrypted values never echoed
Secret exposure in Docker images Credentials baked into image layers Secrets injected at runtime via K8s Secrets; not in Dockerfile or image
Unauthorized API access Public /predict endpoint accepts any request Pydantic schema validation; rate limiting 📋 planned; no auth today (known gap)
Namespace escape Compromised pod accessing secrets from another namespace K8s namespace isolation; secrets are namespace-scoped
MITM on external traffic Traffic between client and API intercepted TLS termination at host-level Nginx; HTTPS enforced
MITM on internal traffic Intra-cluster traffic intercepted K8s cluster networking (no mTLS today; 📋 planned for sensitive paths)
Stale / leaked age key age private key compromised Key stored only in GitLab protected CI variable; rotation procedure documented
WhoScored data injection Malicious content in scraped HTML/JSON affecting downstream GE validation at raw stage; parameterized queries for DB inserts

Secret Lifecycle

Secret created (e.g., PostgreSQL password)
    ↓
Encrypted with SOPS using age public key
    ↓
Encrypted file committed to git (*.enc.yaml, .env.enc)
    ↓
CI pipeline: age private key from protected CI variable
    ↓
CI decrypts secret in scoped deployment step only
    ↓
Kubernetes Secret manifest applied to cluster (namespace-scoped)
    ↓
Pod environment variable injected from K8s Secret at runtime
    ↓
Application reads secret from env var — never from file or code

Encrypted asset types in the repository: - .env.enc — application environment variables - k8s/secrets/*.enc.yaml — Kubernetes Secret manifests - k8s/helm/*/values-*.enc.yaml — Helm values with credentials - gitlab-registry-token.enc.yaml — container registry pull secret

Access Boundaries

Public Internet → System

The only publicly exposed endpoint is the FastAPI service, reachable via:

Internet → host-level Nginx (TLS) → K8s NodePort 31390 → Ingress Controller → FastAPI

All other services (ds namespace, monitoring namespace) are internal-only. No direct public access to PostgreSQL, MinIO, MLflow, Prometheus, Grafana, or RabbitMQ.

K8s Namespace Isolation

Namespace Internal access External access
ingress-nginx Routes to other namespaces Yes — inbound traffic router
ds Internal only No
soccer-api Internal; FastAPI exposed via Ingress FastAPI via Ingress only
monitoring Internal only No

K8s Secrets are namespace-scoped; a pod in soccer-api cannot read secrets from ds.

External Selenoid Host

Traffic from celery-worker-api to Selenoid crosses the cluster network boundary. The Selenoid host is operator-controlled and trusted, but this connection is not protected by mTLS today.

CI/CD Secret Handling

  • The age private key is stored exclusively as a GitLab protected variable (visible only to protected branches/tags).
  • During deployment jobs, the key is exported to the CI environment for SOPS decryption only.
  • Decrypted manifests are applied to K8s and immediately discarded; they are never persisted as job artifacts.
  • CI logs are scanned for accidental secret leakage via GitLab masked variable rules.

What CI can access: - Encrypted SOPS files (committed to repo — safe) - age private key (protected CI variable — restricted) - K8s cluster credentials (protected CI variable — restricted)

What CI cannot do: - Print decrypted secret values (masked) - Persist decrypted files as downloadable artifacts

Secret Rotation Procedure

When a secret must be rotated (e.g., database password changed):

  1. Generate new credential value.
  2. Update the relevant K8s Secret manifest (plaintext) locally.
  3. Re-encrypt with SOPS: sops --encrypt --age <public-key> secret.yaml > secret.enc.yaml.
  4. Commit the updated encrypted file.
  5. CI decrypts and re-applies the K8s Secret on next deployment.
  6. Restart affected pods to pick up the new environment variables.

Full procedure: see Runbook — Secret Rotation.

Public Exposure Summary

Endpoint Public? Auth? Notes
POST /predict Yes No Schema-validated; no rate limit today
GET /predict/{match_id} Yes No Read-only lookup
POST /predict/async/ Yes No Returns task_id
GET /healthcheck/ Yes No Health probe only
GET /metrics Yes No Prometheus scrape; no sensitive data
PostgreSQL No — Internal-only
MinIO No — Internal-only
MLflow No — Internal-only
Grafana No — Internal-only; 📋 auth to be configured
RabbitMQ management No — Internal-only

Best Practices Checklist

  • [x] SOPS/age private key stored only in protected CI variables
  • [x] CI logs do not print decrypted values (masked variables)
  • [x] Kubernetes secrets are namespace-scoped
  • [x] No credentials in Dockerfiles or container images
  • [x] No plaintext .env files in the repository
  • [ ] Rate limiting on public API endpoints (📋 planned)
  • [ ] mTLS for intra-cluster sensitive paths (📋 long-term)
  • [ ] Grafana authentication configured (📋 planned)