Problem
Equipment sensor streams generate terabytes of telemetry, but excursions are still caught after the fact, when yield has already been lost. Static thresholds either alarm constantly or miss slow drift entirely.
Solution
A streaming anomaly-detection service that learns per-chamber baselines and flags drift before it crosses spec limits, with a review UI that lets engineers confirm or dismiss anomalies, feedback that continuously recalibrates the detectors.
Architecture
- Kafka ingestion of tool sensor streams with schema-enforced topics
- Online drift detection with per-chamber seasonal baselines
- Time-series store for replay and offline detector evaluation
- React review console where engineer feedback tunes detector sensitivity
Outcome
- Drift caught excursions earlier than static SPC limits in backtesting
- False-positive rate driven down by the engineer feedback loop
- Replay infrastructure made every detector change testable against history
Lessons
- “Streaming systems fail at the boundaries: schema discipline saves you”
- “An anomaly detector without a feedback loop trains engineers to ignore it”
- “Backtesting against replayed history is the only honest evaluation”
Stack
- Python
- Kafka
- TimescaleDB
- React
- Grafana