Design Distributed Tracing
HardSystem design · Observability
Design a distributed tracing platform for a microservice estate generating about 1M spans/s (roughly 25K traces/s). Engineers look up a trace by ID and search for slow or failing requests by service, operation, duration and tags. Cover context propagation, head versus tail sampling, collection and assembly, storage and indexing, 7-day retention, and how aggregate service metrics stay accurate when only about 1% of traces are kept.
Requirements
- Preserve parent-child relationships across processes, queues and thread pools
- Sampling must not lose errors and slow requests
- Instrumentation overhead on the request path must be negligible and must never block the application
- Store and query traces with retention controls