Design a Data Ingestion Pipeline
HardSystem design · Data · Snowflake
Design a data ingestion platform that accepts events from hundreds of producing services (and database change feeds) and lands them reliably in a data lake and warehouse for analytics.
In scope: the producer contract and schema registry, a durable log, transformation and the dead-letter path, a raw archive for replay, delivery semantics, backfill, and observability. Out of scope: the analytical queries on top and real-time stream-processing jobs (see the streaming problems).
Requirements
- Validate against a versioned schema with enforced compatibility, rejecting bad events at the edge
- Decouple producers from sinks with a durable partitioned log sized for the worst realistic outage
- Keep an unmodified raw archive; support replay, backfill and idempotent sinks keyed on event id
- Quarantine unparseable records without blocking partitions, and monitor lag, freshness and counts end to end