Design LLM Summarization of E-commerce Reviews with Prompt-Injection and PII Defenses

Design a platform that ingests customer product reviews from an e-commerce site and uses a large language model (LLM) to summarize them for an internal dashboard. The interviewer focused on trust and safety: prompt injection, data sanitization, personally identifiable information (PII), and related risks. ### Constraints and Clarifications - Reviews are untrusted user-generated text. Any review may contain spam, abuse, personal data, or text written to manipulate the LLM. - The summaries are shown on an internal dashboard, not to shoppers. - Scale, freshness requirements, languages and the choice of LLM are not given. Ask for them or state your assumptions. ### Clarifying Questions - Who uses the dashboard, and what decisions do they make from it? - At what granularity are summaries produced (per product, per seller, per category, per time window), and how fresh must they be? - How many reviews arrive per day, and does ingestion include a backfill of historical reviews? - Is the LLM a third-party hosted API or a self-hosted model, and may review text leave the company? - May dashboard users drill down to the original reviews, and may they see who wrote them? - Which privacy regulations and retention requirements apply? ### Part 1 — Ingestion and summarization pipeline Design the end-to-end flow from a review being submitted to a summary appearing on the dashboard: the components, the storage, and how summaries are produced and refreshed. ```hint Not every review needs a new summary Think about when a summary must be regenerated, and how to avoid sending every review to the LLM again each time one arrives. ``` ```hint More text than fits A popular product may have far more review text than fits into one model request. Decide how to summarize it anyway. ``` #### What This Part Should Cover - Components and a data model for raw reviews, processed reviews and summaries - Asynchronous processing, and how summaries are triggered and refreshed - Products whose reviews exceed the model's context window - Control of LLM cost and throughput, and behavior when the LLM fails ### Part 2 — Trust and safety, sanitization and PII How does the platform keep spam, fake and abusive reviews, unsafe content and personal data out of the summaries and the dashboard? ```hint What the model never needs Some of what a review contains is not needed to produce a useful summary. Sort the contents of a review by whether the summary needs them. ``` ```hint Order the checks Some checks are cheap enough to run on every review before anything else; others are expensive. Decide the order. ``` #### What This Part Should Cover - Sanitization and normalization of text before any processing - Detection and redaction of PII, and protection of the raw data that still contains it - Filtering of spam, fake and abusive reviews so that they do not distort summaries - Access control, retention and audit for stored data and the dashboard ### Part 3 — Prompt injection and output safety A review may contain text addressed to the model, such as instructions to ignore the task or to praise the product. How do you prevent reviews from manipulating the LLM, and how do you make sure that what the dashboard shows is safe and faithful to the reviews? ```hint Instructions versus data The model receives your instructions and the review text in the same input. Think about how to keep it from treating review text as instructions, and why no single technique is enough. ``` ```hint Limit the blast radius Assume an injection sometimes succeeds. Ask what the model is able to do, and where its output can go. ``` #### What This Part Should Cover - Separation of trusted instructions from untrusted review content - Detection of injection attempts, and the model's limited capabilities - Validation and grounding of the output before it reaches the dashboard - Safe rendering on the dashboard, and testing and monitoring for attacks ### What a Strong Answer Covers - A coherent asynchronous pipeline whose data model supports incremental summaries and traceability from each summary to its source reviews - Defense in depth: sanitization, PII redaction, abuse filtering, injection resistance and output validation as separate layers - Clear trust boundaries around the LLM, with untrusted input and untrusted output on both sides - Deterministic numbers computed outside the LLM, with the summary limited to text grounded in cited reviews - Trade-offs among cost, freshness, coverage and safety, and measurement of summary quality and safety over time ### Follow-up Questions - A coordinated group posts hundreds of reviews containing the same hidden instruction. How do you detect and contain it, and what happens to summaries already produced? - A product manager wants to click a summary sentence and see the reviews behind it. How do you support that without exposing PII? - How would you evaluate a new prompt or model for faithfulness and injection resistance before rolling it out? - A customer deletes their review under a privacy regulation. What must happen to stored data and to summaries that used it?