Design a chatbot fallback for unknown questions
Overview
This question evaluates a candidate's competency in ML system design, focusing on handling model uncertainty, retrieval-augmented components, tool integration, decision policies, and operational evaluation within the ML System Design domain.
OpenAI
Dec 15, 2025, 12:00 AM
Machine Learning Engineer
Onsite
ML System Design
0
0
Scenario
You run a ChatGPT-like assistant. Users sometimes ask questions the model cannot answer reliably (unknown/uncertain/needs up-to-date facts).
Task
Design a system that improves user experience and answer quality when the assistant "doesn't know".
Requirements
- The assistant should avoid hallucinations and be transparent about uncertainty.
- It should still help the user make progress (clarify, retrieve sources, route to tools, etc.).
- Support both factual queries (need citations) and personal/workflow queries (need clarification).
- Low latency for common queries; allow slower tool usage when needed.
Deliverables
- High-level architecture (LLM, retrieval, tools, policies).
- Decision policy for: answer directly vs. ask clarifying questions vs. retrieve vs. refuse.
- How you evaluate success (offline and online metrics) and handle failure modes.
Solution
Login required
Show