Design a Model Serving Platform
HardSystem design · Machine Learning
Design a platform that deploys ML models and serves predictions to applications over an API. Teams register new model versions; the platform loads them onto accelerator-backed workers, routes traffic, batches requests, rolls out gradually and monitors quality. Assume a mix of small models and a few large ones, with bursty traffic. Out of scope: training and the feature store itself.
Requirements
- Support model versioning, shadow, canary and instant rollback
- Scale inference workers despite slow model load, and batch requests to use accelerators efficiently
- Meet a latency budget that includes feature fetch, preprocessing and postprocessing
- Monitor latency, errors, input drift and outcome quality, not just availability