Large Language Models

What's being tested

These prompts test whether you can turn an LLM from a research artifact into a measurable, deployable, and improvable production component. The interviewer is probing for transformer fundamentals, evaluation design, groundedness and safety measurement, model-serving tradeoffs, and disciplined iteration rather than vague claims that a larger model is “better.” Microsoft cares because production assistants must remain useful, safe, observable, and cost-controlled across changing prompts, models, retrieval corpora, and user populations.

Core knowledge

• Transformer architecture uses token embeddings plus positional information, then repeated self-attention and feed-forward blocks. For sequence length $n$ and hidden dimension $d$, standard attention has approximately $O(n^2d)$ compute and memory pressure, making long-context serving expensive.

• Scaled dot-product attention is
  $\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V.$
  Causal masking sets future-token logits to $-\infty$; padding masks prevent attention to nonexistent tokens.

• Autoregressive decoding generates one token at a time. KV caching stores prior keys and values so each new token attends without recomputing the full prefix; it improves latency but increases memory roughly with batch size, layers, sequence length, and head dimension.

• Decoding strategy changes both quality and reproducibility: greedy decoding is deterministic, beam search is more useful for constrained generation, and temperature/top-$p$ sampling improves diversity. Evaluation must fix seeds or report confidence intervals when comparing stochastic versions.

• Offline evaluation should use a versioned, stratified dataset containing representative requests, adversarial cases, long contexts, multilingual examples, and known regressions. Track model, prompt, retrieval configuration, evaluator version, and dataset hash so results are reproducible.

• LLM-as-a-judge can score relevance, correctness, style, or safety, but judge scores require calibration against human labels. Measure agreement such as Cohen’s $\kappa$ or Spearman correlation, test position and verbosity bias, and retain human review for high-impact decisions.

• Groundedness asks whether each claim is supported by supplied evidence, not merely whether the answer sounds plausible. A practical sentence-level score is
  $\text{groundedness}=\frac{\#\text{supported factual claims}}{\#\text{factual claims}},$
  with abstention or citation requirements when support is absent.

• Retrieval-augmented generation (RAG) quality decomposes into retrieval recall, context precision, answer faithfulness, and answer usefulness. Diagnose retrieval separately from generation: a correct answer is impossible when the required evidence is not retrieved, while irrelevant context can induce distraction or contradictions.

• Online monitoring should combine quality proxies and operational metrics: groundedness, refusal rate, task success, user feedback, token usage, throughput, p50/p95/p99 latency, timeout rate, and safety-violation rate. Segment by language, task, tenant, model version, and traffic cohort to expose localized regressions.

• Drift monitoring covers input, retrieval, and output distributions. Track embedding-distance or population-stability changes, prompt-length shifts, retrieval hit rates, citation coverage, and judge-score distributions; alert thresholds should be calibrated to false-positive cost rather than chosen arbitrarily.

• Safe deployment uses shadow traffic, canaries, feature flags, automatic rollback, rate limits, content filtering, prompt-injection defenses, and least-privilege tool access. Canary decisions should use minimum sample sizes and guardrails for safety, latency, cost, and quality—not only average user preference.

• Serving optimization balances quality against cost and latency through quantization, batching, continuous batching, speculative decoding, prompt-prefix caching, smaller routing models, and distillation. Quantization can reduce memory and improve throughput, but validate degradation separately for rare languages, long contexts, tool calls, and safety behavior.

Worked example

For Design an Evaluation and Improvement Loop for a Production LLM Application, I would first clarify the application’s task, user population, risk level, latency and cost targets, available feedback, and whether answers must cite an approved knowledge base. I would state an assumption such as a RAG assistant where correctness and groundedness matter more than stylistic preference. I would organize the design around four pillars: a versioned evaluation corpus, calibrated offline evaluators, production monitoring with segmented slices, and a controlled improvement and rollback loop. The offline set would include golden answers or evidence labels, adversarial prompts, historical failures, and new samples from production, while separating development, validation, and final holdout data. I would compare prompt, model, retriever, and chunking versions using task success, groundedness, citation correctness, refusal quality, latency, and cost. A key tradeoff is using an LLM judge for scale versus human labels for validity: I would calibrate the judge, estimate disagreement, and route uncertain or high-risk examples to humans. Online, I would canary changes and require no regression on safety and latency guardrails even if aggregate quality improves. If I had more time, I would add active learning to prioritize uncertain failures and analyze whether improvements generalize across languages, tenants, and long-tail intents.

A second angle

For Explain Transformers and deploy an LLM safely, the same evaluation mindset is applied earlier, at the model-serving boundary. The candidate should connect causal masking, KV-cache memory, batching, quantization, and decoding choices to measurable p99 latency, throughput, cost, and response quality. The framing shifts from “how do we improve an existing application loop?” to “what architecture and deployment controls make the model reliable under production load?” A strong answer still defines quality and safety gates before selecting serving optimizations. It should explicitly test that compression or speculative decoding does not create regressions in refusal behavior, factuality, or rare but high-severity prompts.

Common pitfalls

Pitfall: Treating lexical overlap such as BLEU or ROUGE as a sufficient hallucination metric is tempting but wrong: a fluent answer can match reference wording while contradicting the evidence. Pair task metrics with claim-level entailment, citation verification, human labels, and retrieval diagnostics.

Pitfall: Saying “we monitor quality” without naming an instrumented metric, slice, threshold, or response action sounds non-operational. State an example such as: “If groundedness drops by 5% for finance queries in a canary, halt rollout, inspect retrieval misses, and automatically route traffic to the previous version.”

Pitfall: Recommending an LLM judge as objective ground truth ignores evaluator bias, prompt sensitivity, and correlated model errors. Explain calibration against humans, judge versioning, disagreement sampling, and why high-risk cases require deterministic checks or manual review.

Connections

The interviewer may pivot to RAG retrieval evaluation, responsible AI and red teaming, **model compression and GPU serving, or A/B testing with sequential monitoring. Be ready to connect each to offline/online parity, slice-level metrics, and rollback criteria.

Further reading

• Attention Is All You Need — the original Transformer architecture and attention formulation.

• HELM: Holistic Evaluation of Language Models — broad, scenario-based evaluation across accuracy, robustness, fairness, and efficiency.

• RAGAS: Automated Evaluation of Retrieval Augmented Generation — metrics and methodology for separating retrieval and generation quality.