Generate Customer-Support Instructions from Company Policies

Design a RAG system that helps customer-support personnel resolve user issues. It reads company policies and produces step-by-step instructions for the support agent to follow. ### Constraints & Assumptions - The output is an evidence-grounded procedure for a human support agent, not autonomous execution of account changes. - Policies can have versions, effective dates, and applicability conditions. Clarify which issue context is available and which facts the support agent must confirm. - Steps must not invent permissions, exceptions, or remedies absent from the applicable policies. - Include data, retrieval, model behavior, evaluation, and online debugging. No scale, model vendor, or latency SLA is specified. ### Clarifying Questions to Ask - Which issue types are supported, and how are conflicting policies prioritized? - Are policies restricted by region, product, account type, or support role? - Should the assistant ask for missing facts before producing an executable procedure? - What is the escalation path when evidence is missing, conflicting, or outdated? ### Part 1 — Policy ingestion and retrieval Design policy storage, chunking, metadata, and retrieval for the issue being handled. #### What This Part Should Cover - Version/effective-date metadata, access controls, and preservation of procedural context. - Retrieval of prerequisites, exceptions, and related rules, not only a superficially similar paragraph. - Handling of changed, deleted, or conflicting policy sources. ### Part 2 — Step-by-step generation Define how retrieved evidence becomes an ordered support procedure with references and uncertainty handling. #### What This Part Should Cover - Applicability checks and required customer facts before taking a step. - Ordering, branching conditions, citations, and explicit escalation when unsupported. - A boundary between untrusted policy text, model instructions, and authorized human actions. ### Part 3 — Evaluation and debugging Explain how to test the system and diagnose an incorrect instruction sequence in production. #### What This Part Should Cover - Retrieval coverage and procedural correctness, including missing prerequisites and invalid exceptions. - Test cases involving stale policies, conflicting clauses, missing context, and no supported answer. - Traceability from each generated step to retrieved versions and the decision path. ```hint Retrieve the rule around the action A policy sentence naming a remedy may be unsafe to apply without its eligibility conditions, exceptions, and approval requirements. Keep those dependencies visible to both retrieval and evaluation. ``` ### What a Strong Answer Covers - A policy-aware data and retrieval design tied to the support workflow. - Useful ordered instructions that preserve policy conditions and distinguish missing information from a supported action. - Evaluation and online debugging that can detect a plausible but procedurally invalid answer. ### Follow-up Questions - How would you handle a policy update while a support conversation is already in progress? - What would you do if the retrieved documents disagree about an exception?