drift monitorPJM · demand
ALERT
PSI9.07
error vs baseline15.9×
Portfolio project · MLOps

Driftwatch

A self-monitoring electricity-demand forecasting service that flags its own decay.

Most ML apps stop at serving a prediction. The dangerous failure in production isn't a crash — it's a model that keeps returning confident, wrong answers after the world shifts underneath it. Driftwatch forecasts regional grid demand and watches its own inputs and error, so it can say when it has become untrustworthy instead of failing silently.

≈2.0%
forecast error (MAPE)
60.7%
better than a seasonal-naive baseline
9.07
drift PSI caught (alert ≥ 0.25)
16×
error spike detected
50
passing tests · 0 lint errors

What it is

A forecasting service that knows when it's wrong

Driftwatch pulls hourly electricity demand for a US balancing authority (PJM), forecasts the next 24 hours with a gradient-boosted model, and serves it behind an API with a live dashboard. The differentiator is the layer on top: it continuously compares incoming demand and its own prediction error against what it saw at training time, and raises a graded flag — ok / warn / alert — the moment it starts to drift.

forecast

Gradient boosting

scikit-learn on calendar + demand-lag features. Scored on a time-ordered hold-out against a "same hour yesterday" baseline — so the reported skill is real, not leaked.

serve

FastAPI + Docker

Typed endpoints for health, model metadata, and forecasts. One container, self-bootstrapping, deployable to any free tier in a click.

monitor

The drift layer

PSI + Kolmogorov–Smirnov on the demand distribution, plus prediction-error tracking. Thresholds turn silent decay into a visible, paging signal.

How it works

One pipeline, from raw feed to a red banner

Data flows left to right; the monitoring layer branches off the model to compare live windows against the training reference.

EIA API
hourly demand feed
→
Ingest
paginated, retried, idempotent
→
SQLite
observations + run log
→
Features
calendar + lags, leakage-safe
→
Model
gradient boosting
FastAPI
/health · /model · /predict
→
Drift layer
PSI · KS · error ratio
→
Flag + reports
ok / warn / alert, stored
→
Dashboard
served by the API itself

Training also snapshots a reference — the demand distribution the model learned on — so every later window has something honest to be compared against.

Proof

We fed the running system shifted data. The flag fired.

The whole point, tested end to end: train on 45 stable days, then inject a +35% demand level shift into the most recent week — a heat-wave-scale jump to a level the model never trained on. Both independent detectors caught it and the banner flipped from green to red.

OK · stable window

Inputs and error look like training

Input drift — PSI0.04
KS test (p-value)0.08 (p=0.40)
Recent error vs baseline1.01×
Alert · +35% shift

Distribution moved; the model can't keep up

Input drift — PSI9.07
KS test (p-value)0.81 (p≈0)
Recent error vs baseline15.9×
alert ≥ 0.25 stable · PSI 0.04 drift · PSI 9.07 36× past the line →
Population Stability Index, stable vs. shifted. driftwatch drift --fail-on-alert exits non-zero on alert, so a scheduled job pages instead of the model failing quietly. Reproduce it in ~1 min with scripts/drift_demo.sh.

How it was built

Five weekly increments, shipped and verified

Each week is a self-contained commit that leaves the system runnable — plumbing first, model second, the monitoring differentiator third, then the human view, then deploy and stress.

Week 1

Ingestion

A typed EIA API client with pagination and retry/backoff, into an idempotent SQLite store keyed on (respondent, type, hour) with a run log. Scheduled via GitHub Actions.

Week 2

Forecast & serve

A gradient-boosted forecaster on calendar + lag features, scored honestly against a seasonal-naive baseline, served behind FastAPI and containerized.

Week 3

The drift layer

PSI and KS on the demand distribution plus prediction-error monitoring, against a reference captured at training time — with the thresholds that raise the flag.

Week 4

Dashboard

A live monitoring page — the flag, model quality, the forecast, and the drift-history timeline — served by the API itself, no second process.

Week 5

Deploy & stress

A self-bootstrapping container that runs on any free tier, and the reproducible experiment above that proves the flag fires under a deliberate shift.

Engineering decisions

The choices that signal "operates a system"

No train/serve skew

Training and inference build features through the same code path, and every feature reads only history older than the target hour — forecasts up to 24h ahead are leakage-free by construction.

Honest evaluation

Scored on a time-ordered hold-out (never a shuffled split) against a seasonal-naive baseline, so the reported ~61% skill is a real improvement, not an artifact.

Idempotent by design

Ingestion upserts on a natural key, so overlapping scheduled runs never duplicate rows; the deploy bootstrap only seeds and trains when the store is empty.

Meaningful drift, not noise

PSI tracks demand level, not calendar features — whose distribution is fixed by the window, not by drift — so alerts are signal, not false alarms.

Stack

Python 3.11+ FastAPI + uvicorn scikit-learn HistGBR pandas + numpy SciPy KS test SQLite Docker GitHub Actions cron pytest · 50 tests ruff