A self-monitoring electricity-demand forecasting service that flags its own decay.
Most ML apps stop at serving a prediction. The dangerous failure in production isn't a crash — it's a model that keeps returning confident, wrong answers after the world shifts underneath it. Driftwatch forecasts regional grid demand and watches its own inputs and error, so it can say when it has become untrustworthy instead of failing silently.
Driftwatch pulls hourly electricity demand for a US balancing authority (PJM), forecasts the next 24 hours with a gradient-boosted model, and serves it behind an API with a live dashboard. The differentiator is the layer on top: it continuously compares incoming demand and its own prediction error against what it saw at training time, and raises a graded flag — ok / warn / alert — the moment it starts to drift.
scikit-learn on calendar + demand-lag features. Scored on a time-ordered hold-out against a "same hour yesterday" baseline — so the reported skill is real, not leaked.
Typed endpoints for health, model metadata, and forecasts. One container, self-bootstrapping, deployable to any free tier in a click.
PSI + Kolmogorov–Smirnov on the demand distribution, plus prediction-error tracking. Thresholds turn silent decay into a visible, paging signal.
Data flows left to right; the monitoring layer branches off the model to compare live windows against the training reference.
Training also snapshots a reference — the demand distribution the model learned on — so every later window has something honest to be compared against.
The whole point, tested end to end: train on 45 stable days, then inject a +35% demand level shift into the most recent week — a heat-wave-scale jump to a level the model never trained on. Both independent detectors caught it and the banner flipped from green to red.
driftwatch drift --fail-on-alert
exits non-zero on alert, so a scheduled job pages instead of the model failing quietly.
Reproduce it in ~1 min with scripts/drift_demo.sh.Each week is a self-contained commit that leaves the system runnable — plumbing first, model second, the monitoring differentiator third, then the human view, then deploy and stress.
Ingestion
A typed EIA API client with pagination and retry/backoff, into an idempotent SQLite store keyed on (respondent, type, hour) with a run log. Scheduled via GitHub Actions.
Forecast & serve
A gradient-boosted forecaster on calendar + lag features, scored honestly against a seasonal-naive baseline, served behind FastAPI and containerized.
The drift layer
PSI and KS on the demand distribution plus prediction-error monitoring, against a reference captured at training time — with the thresholds that raise the flag.
Dashboard
A live monitoring page — the flag, model quality, the forecast, and the drift-history timeline — served by the API itself, no second process.
Deploy & stress
A self-bootstrapping container that runs on any free tier, and the reproducible experiment above that proves the flag fires under a deliberate shift.
Training and inference build features through the same code path, and every feature reads only history older than the target hour — forecasts up to 24h ahead are leakage-free by construction.
Scored on a time-ordered hold-out (never a shuffled split) against a seasonal-naive baseline, so the reported ~61% skill is a real improvement, not an artifact.
Ingestion upserts on a natural key, so overlapping scheduled runs never duplicate rows; the deploy bootstrap only seeds and trains when the store is empty.
PSI tracks demand level, not calendar features — whose distribution is fixed by the window, not by drift — so alerts are signal, not false alarms.