Design an ML Feature Store
HardSystem design · Machine Learning · Databricks
Design a feature store that serves machine learning features for both training and online inference. Data scientists define features once; the platform computes them in batch and from streams, builds point-in-time-correct training sets from history, and serves the latest values to models in milliseconds. Out of scope: model training itself and the model serving layer.
Requirements
- Keep offline and online features consistent: one definition, no separate reimplementation
- Produce point-in-time correct training data with no leakage of future values
- Serve whole feature vectors for an entity at low latency; support per-feature freshness from seconds to daily
- Support backfills of new features, ownership, versioning and detection of stale or drifting features