Prompt Injection And Data Exfiltration Defenses

What's being tested

Candidates must show they can model attacker capabilities against a language-driven system, then design pragmatic engineering controls that reduce prompt injection and data exfiltration risk while preserving service utility. Interviewers probe ability to define a clear threat model, pick concrete isolation and sanitization mechanisms, reason about tradeoffs (latency, recall, developer ergonomics), and propose measurable detection/mitigation instrumentation. OpenAI cares because production systems must balance safety, availability, and developer productivity at scale.

Core knowledge

Threat model: clearly separate attacker goals (read secrets, pivot to internal tools, induce privileged actions) and attacker surface (user prompt, uploaded files, RAG contexts, tool outputs). Define attacker capabilities and success criteria before designing controls.
Principle of least privilege: grant each subsystem only the permissions it needs—e.g., retrieval subsystem can read indexed docs but cannot access secrets store; runtime tokens scoped and short-lived.
Prompt canonicalization & framing: normalize user input (strip control characters, base64-encode binary), then prepend a trusted system prompt that is immutable on the server to reassert policy and provenance; avoid letting user-controlled text overwrite system instructions.
Input-level defenses: apply syntactic sanitization (remove nested instruction blocks), semantic heuristics (detect "ignore previous" patterns), and ML-based classifiers to flag adversarially-crafted prompts. Combine rule + model for better recall/precision.
Retrieval/Context hardening: in RAG, tag each retrieved chunk with provenance and trust score; use strict source filters and similarity thresholds (e.g., cosine similarity > θ) to avoid returning highly irrelevant or adversarial fragments. Partition vector spaces by tenant/namespace.
Tool & capability gating: treat external actions (DB writes, code execution, HTTP calls) as capabilities behind an authorization layer; require explicit server-side allowlists, rate limits, and human approvals for high-risk actions.
Output sanitization & redaction: run post-generation filters (regex + classifiers) that redact high-confidence secrets, internal endpoints, or credential patterns before returning text. Maintain a denylist and use fuzzy matching for rotated secrets.
Rate limiting & quotas: use a token-bucket model (refill R tokens/sec, capacity B) to limit exfiltration throughput; exfiltration potential ≈ burst × avg_secret_bytes_per_token.
Isolation patterns: use process-level sandboxing or dedicated service boundaries (e.g., separate inference and tooling services), so compromises in one layer can't reach secret stores.
Auditability & detection: log prompts, retrieved contexts, model responses, and tool requests with immutable IDs. Build anomaly detectors on patterns like unusually long outputs, repeated retrievals of sensitive namespaces, or high similarity to secret templates.
Testing & red-team validation: perform automated adversarial fuzzing, unit tests for sanitizers, and simulated exfiltration scenarios to validate defenses; measure false negative/positive rates.
Tradeoffs: aggressive filtering reduces risk but increases false positives and degrades UX; instrumentation and ML classifiers introduce latency and maintenance burden. Quantify impact in SLOs (e.g., p95 latency, false-positive rate).

Worked example — "Design defenses for prompt injection and data exfiltration in an assistant that uses RAG and tool calls"

First 30 seconds: clarify scope (single-tenant vs multi-tenant, what secrets exist: API keys, PII, DB rows), required latency/SLOs, and what actions the assistant may perform (read-only, write, external HTTP, code execution). Skeleton answer pillars: (1) threat model and success definition, (2) containment/isolation (capability gating, scoped creds), (3) input/output sanitization and retrieval hardening, (4) monitoring & incident response. A concrete design decision: prefer server-side immutable system prompts and deny user-supplied system messages entirely, because user-supplied instructions are high-risk; tradeoff is less flexible agent prompting for power users. Close by proposing measurable tests (fuzzing suite, red-team scenarios) and incremental rollout: start read-only RAG with conservative retrieval thresholds, gather telemetry, then enable tool calls behind stricter checks. If more time: implement provable provenance headers for each retrieved chunk (signed IDs) and build a feedback loop to retrain injection-detection models.

A second angle — "Secure an LLM that can execute developer-submitted scripts (sandboxed code execution)"

Same core concept applies but constraints differ: code execution increases blast radius and requires strong runtime isolation. Emphasize ephemeral, highly-scoped credentials for any external API access, container sandboxing with kill-switch, syscall whitelisting (seccomp-like), and capped resource usage (CPU, memory, disk). Because malicious outputs may try to exfiltrate via network, either disable outbound network entirely or mediate all outbound connections through a proxy that enforces allowlists and inspects payloads. Instrument code-exec events for rapid rollback and require human approval for any action that escalates privileges. Here, detection focuses more on behavioral signals (unexpected system calls, long-tail network destinations) rather than pure text patterns.

Common pitfalls

Pitfall: Assuming string filters are sufficient. Text-only regexes miss semantic injections (e.g., paraphrased instructions) and can be bypassed by encoding tricks; always pair with semantic classifiers and server-side framing.

Pitfall: Overclaiming "perfect" security. Saying "we'll just block user system messages" without addressing retrieval contamination, tool gating, or leaked logs underestimates real-world attack paths; demonstrate layered defenses.

Pitfall: Ignoring telemetry and measurement. A solution that lacks logging, replayability, and test harnesses cannot be iterated on; interviewers penalize designs that have hard-to-evaluate efficacy.

Connections

This topic frequently pivots to adjacent engineering problems: secret management (rotating, scoped credentials, Vault patterns) and observability (structured logging, distributed tracing for prompt/response lineage). Interviewers may also ask about performance impacts of security controls (SLO design) or privacy-preserving techniques like differential privacy when protecting aggregated data.