Design a Chatbot Platform
HardSystem design · AI Products
Design a platform where users chat with AI assistants that can retrieve from knowledge sources and call tools. The platform calls third-party or hosted language models, streams responses, and has to control cost and safety.
In scope: conversation storage, context assembly and retrieval, streaming and cancellation, tool execution, cost and rate controls, and safety. Out of scope: training or fine-tuning models and building the vector database itself.
Requirements
- Persist conversation state, retrieved context and tool results; resume a stream after disconnect
- Stream partial responses and cancel upstream generation when the user stops or disconnects
- Apply per-tenant rate limits, spend caps, model routing and prompt caching to control cost
- Screen input and output, treat tool calls and retrieved content as untrusted, and require confirmation for side effects