# Review and Debug an AI Chatbot Review the design of an AI chatbot and propose improvements. Then explain how you would investigate reports of incorrect, unsafe, slow, or inconsistent responses without assuming every failure comes from the model itself. ### Constraints & Assumptions - The chatbot uses a language model plus application prompts, retrieval, optional tools, safety controls, and a user-facing service. - Conversation data may contain sensitive information. - Model calls are probabilistic, expensive, and dependent on external services. - Product quality includes task success, safety, latency, reliability, and user trust. ### Clarifying Questions to Ask - Who are the users, what tasks should the chatbot complete, and which failures are unacceptable? - Does it retrieve documents or call tools, and what authorization boundaries apply? - Which models, prompts, indexes, and policies are currently deployed? - What traces and user feedback are retained, and under what privacy policy? - How are quality, safety, cost, and latency measured today? ### Hints - Reconstruct one failing request across the entire pipeline. - Build an error taxonomy before changing prompts or models. - Use a representative evaluation set and controlled comparisons. ### What a Strong Answer Covers - Request flow through policy, prompt construction, retrieval, model, tools, post-processing, and presentation. - Traceability with versioned configurations, privacy-aware logs, correlation IDs, and replayable fixtures. - Error categories such as retrieval miss, stale data, prompt conflict, tool failure, authorization error, model behavior, and UI misrepresentation. - Offline and online evaluations, human review, adversarial tests, cohort analysis, and regression gates. - Improvements to grounding, tool contracts, uncertainty, escalation, latency, caching, and safety. - Staged experiments, rollback, cost controls, incident response, and explicit residual risk. ### Follow-up Questions 1. How would you determine whether a wrong answer came from retrieval or generation? 2. What data would you log without retaining sensitive conversation content? 3. How would you evaluate a prompt change whose average score improves but one safety cohort worsens? 4. When should the chatbot refuse, ask a clarifying question, or escalate to a human?