Model Monitoring & Drift Detection in Production

Deploying a model to production is not the finish line — it is the starting gun for a continuous monitoring challenge. Unlike traditional software, ML models can degrade silently: no exceptions are thrown, no alerts fire, but the model is quietly making worse predictions because the world has changed.

This article covers the full monitoring stack for production ML systems: what to measure, how to detect drift, how to set up alerting, and when to trigger automated retraining.

What Can Go Wrong in Production?

ML models fail in production for three distinct reasons:

Data drift (covariate shift): The distribution of input features changes. The model was trained on one distribution but is now scoring a different one. Example: a fraud detection model trained on pre-pandemic transaction patterns encounters pandemic-era spending.
Concept drift: The relationship between features and the target changes. The model's assumptions about the world become invalid. Example: a credit scoring model trained on a period of economic growth encounters a recession.
Infrastructure failure: Upstream pipeline changes break feature computation. A feature that used to be float now arrives as null. A join that used to match starts missing 40% of records.

Lesson learned: In practice, infrastructure failures are the most common cause of model degradation and the hardest to detect if you're only monitoring model output metrics. Always monitor data quality at the pipeline entry point.

The Monitoring Stack

A complete monitoring setup has four layers:

Layer 1

Infrastructure

Latency, throughput, memory, CPU, error rates

Layer 2

Data Quality

Null rates, out-of-range values, schema violations

Layer 3

Feature Drift

Distribution shift vs training baseline

Layer 4

Model Performance

Accuracy, AUC, precision, business KPIs

Data Quality Monitoring

Before checking if the model is drifting, check if the data pipeline is healthy. Great Expectations and Pydantic are excellent tools for defining and enforcing data quality contracts.

from great_expectations.core import ExpectationSuite
import great_expectations as gx

context = gx.get_context()

# Define expectations on the incoming feature dataset
suite = context.add_expectation_suite("features.production")

# Validate expected schema and value ranges
validator.expect_column_values_to_not_be_null("customer_age")
validator.expect_column_values_to_be_between("transaction_amount", min_value=0, max_value=100000)
validator.expect_column_values_to_be_in_set("country_code", ["BR", "US", "DE", "FR"])
validator.expect_column_mean_to_be_between("credit_score", min_value=500, max_value=750)

# Run validation
results = validator.validate()
if not results.success:
    # Alert and halt pipeline
    raise DataQualityException(f"Validation failed: {results.statistics}")

Detecting Feature Drift with Statistical Tests

Feature drift detection compares the current serving distribution against the training baseline using statistical tests. The most commonly used are:

Kolmogorov-Smirnov (KS) test: For continuous features. Tests whether two samples come from the same distribution. p-value below 0.05 indicates drift.
Population Stability Index (PSI): Widely used in financial ML. PSI < 0.1: stable. 0.1–0.2: monitor. > 0.2: significant drift, investigate.
Chi-squared test: For categorical features. Tests whether observed vs expected frequencies differ significantly.
Jensen-Shannon Divergence: A symmetric, bounded version of KL-divergence. Good for monitoring overall distribution shift.

from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, DataQualityPreset
import pandas as pd

# reference_data = training dataset or recent stable window
# current_data   = last 24h of production data

report = Report(metrics=[
    DataDriftPreset(drift_share=0.3),  # alert if >30% features drift
    DataQualityPreset(),
])

report.run(reference_data=reference_df, current_data=current_df)
report.save_html("drift_report.html")

# Extract drift summary
result = report.as_dict()
drifted_features = [
    feat for feat, info in result["metrics"][0]["result"]["drift_by_columns"].items()
    if info["drift_detected"]
]

if len(drifted_features) / total_features > 0.3:
    send_alert(f"Drift detected in {len(drifted_features)} features: {drifted_features}")

Model Performance Monitoring

When Labels Are Available (Low Latency)

For models where ground truth labels arrive quickly — fraud detection (chargeback within hours), click prediction (feedback within seconds) — you can compute real-time performance metrics and compare against the champion model's baseline.

# Prometheus metrics for model performance (Python)
from prometheus_client import Gauge, Counter, Histogram

PREDICTION_LATENCY = Histogram(
    "model_prediction_duration_seconds",
    "Time to generate a single prediction",
    buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5]
)

MODEL_AUC = Gauge("model_auc_score", "Rolling 24h AUC", ["model_version"])
DRIFT_SCORE = Gauge("feature_drift_psi", "PSI score per feature", ["feature_name"])
PREDICTION_ERRORS = Counter("model_prediction_errors_total", "Total prediction errors")

# In the serving endpoint
@PREDICTION_LATENCY.time()
def predict(features):
    return model.predict(features)

When Labels Are Delayed (High Latency)

For models where ground truth is delayed — loan default (months), churn (quarterly) — you cannot wait for labels to know if the model is degrading. Instead, use proxy metrics:

Prediction distribution shift: Compare the distribution of model output scores today vs training baseline. A shift in score distribution often precedes a shift in performance.
Business metrics: Conversion rate, click-through rate, revenue per prediction. These reflect real-world model impact even without labels.
Calibration: For probability models, check if predicted probabilities match observed frequencies (reliability diagram).

Alerting and Thresholds

Not all drift requires immediate action. Define tiered alert levels:

# Alerting thresholds (Prometheus alerting rules)
groups:
  - name: model_monitoring
    rules:
      - alert: FeatureDriftWarning
        expr: feature_drift_psi{feature_name!=""} > 0.1
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Feature {{ $labels.feature_name }} PSI={{ $value | humanize }}"

      - alert: FeatureDriftCritical
        expr: feature_drift_psi{feature_name!=""} > 0.25
        for: 30m
        labels:
          severity: critical
        annotations:
          summary: "Critical drift — retraining required"

      - alert: ModelAUCDegraded
        expr: model_auc_score < 0.78
        for: 2h
        labels:
          severity: critical
        annotations:
          summary: "Model AUC {{ $value | humanize }} below threshold 0.78"

Automated Retraining Triggers

Manual retraining is reactive and slow. Build retraining triggers into the monitoring pipeline:

Schedule-based: Retrain weekly or monthly. Simple, predictable, but may retrain when unnecessary or fail to retrain fast enough.
Performance-based: Trigger retraining when AUC drops below threshold. Requires labeled feedback loop.
Drift-based: Trigger retraining when PSI exceeds threshold for a sustained period. Works without labels — suitable for high-latency feedback models.

Best practice: Combine all three triggers with an AND logic for high-stakes models: retrain only when drift is detected AND it has persisted for more than 48 hours AND the weekly schedule is due. This prevents unnecessary retraining on transient anomalies.

The Monitoring Dashboard

A Grafana dashboard for ML models should surface at a glance:

Prediction volume over time (sudden drops = upstream pipeline failure)
Prediction score distribution heatmap (shifts indicate drift)
Per-feature PSI scores with threshold bands
Rolling performance metrics (AUC, precision@K, calibration error)
Data quality pass rate by feature and pipeline stage
Retraining history and model version deployment timeline

Key Takeaways

Monitor in four layers: infrastructure, data quality, feature drift, and model performance.
Data quality monitoring is your first line of defense — pipeline failures are the most common cause of degradation.
Use PSI and KS tests for drift detection; set tiered alert thresholds to avoid alert fatigue.
For delayed-label models, monitor prediction distribution and business KPIs as proxy metrics.
Automate retraining triggers based on drift persistence, not just point-in-time anomalies.
Track every model version, deployment date, and retraining event in a central registry.

What Can Go Wrong in Production?O Que Pode Dar Errado em Produção?

The Monitoring StackA Stack de Monitoramento

Data Quality MonitoringMonitoramento de Qualidade de Dados

Detecting Feature Drift with Statistical TestsDetectando Drift de Features com Testes Estatísticos

Model Performance MonitoringMonitoramento de Performance do Modelo

When Labels Are Available (Low Latency)Quando os Rótulos Estão Disponíveis (Baixa Latência)

When Labels Are Delayed (High Latency)Quando os Rótulos São Atrasados (Alta Latência)

Alerting and ThresholdsAlertas e Limites

Automated Retraining TriggersGatilhos de Retreinamento Automático

The Monitoring DashboardO Dashboard de Monitoramento

Key TakeawaysPontos Principais

What Can Go Wrong in Production?

The Monitoring Stack

Data Quality Monitoring

Detecting Feature Drift with Statistical Tests

Model Performance Monitoring

When Labels Are Available (Low Latency)

When Labels Are Delayed (High Latency)

Alerting and Thresholds

Automated Retraining Triggers

The Monitoring Dashboard

Key Takeaways