LLM Evaluation And Guardrails

What's being tested

Interviewers are probing your ability to design reliable, observable, and enforceable runtime protections for a production LLM-driven system: measurable evaluation pipelines, automated safety guardrails, deployment/rollout patterns, and operational controls that an engineer can implement. Expect to justify tradeoffs between latency, cost, false-positive filtering, and developer velocity while showing how you instrument and iterate on model quality in production.

Core knowledge

• Evaluation pipeline: separate offline (batched annotations, synthetic tests) and online (production sampling) evaluation flows; store prompt/output pairs, metadata, and hashed user IDs for repro and privacy.

• Factuality / hallucination metrics: define hallucination rate = false factual assertions / total factual assertions; estimate with binomial margin: $n = p(1-p)*(1.96/m)^2$ for 95% confidence.

• Quality & safety SLIs: common SLIs include p99 latency, request error rate, throughput, hallucination rate, and toxic-content-rate; bind these to observable SLOs with alert thresholds and runbooks.

• Annotation tooling: use standard schemas, inter-rater agreement (e.g., Cohen's kappa) to validate label quality; reduce label drift with periodic rater calibration.

• Guardrail patterns: input validation via JSON Schema, allowlist/blocklist checks, rule-based sanitizers, staged classifiers (fast filter → LLM), and output filters (regex, HTML escaping, profanity lists).

• Fallback & routing: implement circuit breakers, graded fallbacks (rule response, template reply, smaller model), and traffic splits/feature flags for canarying new models or prompts.

• Adversarial testing: automated prompt fuzzing, property-based testing (e.g., Hypothesis-style), and red-team scenarios; log adversarial prompts to a replayable corpus.

• Observability & logging: log structured artifacts (prompt, response, model version, embedding id, latency, trace id) to a long-tail store with TTL and sampling; enable deterministic replay for debugging.

• Confidence & verification: use provenance (citations), retrieval-augmented generation, or lightweight verifier classifiers for high-stakes responses; avoid trusting raw token probabilities alone.

• Privacy & compliance: redact PII at ingress, hash identifiers for metrics, and avoid logging raw sensitive text unless explicitly allowed in policy and with access controls.

• Performance tradeoffs: batching and caching embeddings reduce cost but increase tail latency; set p99 SLOs explicitly and pick batch-size vs latency tradeoff per endpoint.

• Deployment hygiene: automate canary rollouts, use idempotency keys for retries, implement backpressure and rate limits, and provide feature flags to disable LLM paths quickly.

Worked example — "Design an evaluation and guardrail pipeline for a customer-facing LLM chat service"

First 30 seconds you clarify: what are the high-risk failures (hallucination, toxicity, PII leakage), acceptable latency SLOs, and sampling budget for human review. Organize your answer into three pillars: (1) offline test-suite (unit tests, adversarial corpus, synthetic assertions), (2) online sampling + automated filters (fast classifier → block/fallback), and (3) ops (logging, canaries, alerts). Concretely, propose sampling 1–2% of production queries for full human review, keep a streaming classifier to catch high-risk outputs with a tuned threshold that prioritizes recall for toxicity, and store structured logs for repro with trace ids and model version tags. Flag the main tradeoff: stricter filters reduce risky outputs but increase false positives and user friction — propose metrics to track both. Close with rollout steps: start with internal-only canary, expand via feature flags to gradually increasing traffic, and iterate on annotation schema and thresholds. If you had more time, add simulated adversarial generation and automated A/B experiments measuring downstream user satisfaction.

A second angle — "How would you monitor hallucination and safety regressions at scale?"

Same core concepts but the framing emphasizes continuous detection: implement a lightweight online verifier that detects factual assertions and routes a sampled subset to fact-checkers; compute daily aggregated SLIs (e.g., hallucination rate per intent) and apply statistical change detection (e.g., control charts or simple proportion tests) to trigger alerts. Instrument histograms of output lengths, token-likelihood anomalies, and classifier score distributions to detect distributional shifts. Use canaries and shadow deployments to compare new model versions without user exposure. This angle forces tradeoffs: you must choose what to sample (uniform vs risk-weighted), how much human labeling budget you allocate, and which automated signals to trust for noisy pre-alarms.

Common pitfalls

Pitfall: Over-relying on proxy metrics like BLEU/ROUGE. These surface-level NLG scores rarely correlate with factuality or safety; pair them with human annotations or task-specific verifiers instead.

Pitfall: Logging everything without privacy controls. Dumping raw prompts/responses to logs violates compliance and creates attack surfaces; redact PII, hash identifiers, and apply strict access controls.

Pitfall: Designing one-off manual filters as the only guardrail. Rule-based filters rot quickly under adversarial input and scale badly; combine classifiers, staged fallbacks, and an iterative red-team corpus to keep defenses current.

Connections

This topic naturally pivots to model serving & autoscaling, feature-flag driven rollouts / canarying, and observability for ML (traceability, lineage, and metric alerting). Interviewers may probe deployment orchestration (autoscale, batching), or how you'd pair evaluation with CI/CD for models.