Design an experiment assignment service
Company: Visa
Role: Software Engineer
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
Design an online experimentation (A/B testing) assignment service. The API input is (userId, featureName) and it returns a variant identifier (e.g., control or treatment). Clarify and specify: deterministic sticky assignment and bucketing strategy, configuration storage for experiments and features, eligibility targeting and audience filters, traffic allocation and ramps, mutual exclusion and experiment layering, overrides (holdouts, QA, killswitch), exposure logging and idempotency, metric tagging and data pipeline, privacy and compliance (e.g., GDPR/CCPA), latency/availability targets and fallbacks, and the experiment lifecycle (create, start, pause, stop, analyze). Provide a high-level architecture and key components.
Overview: This question evaluates system-design and analytics competencies for online experimentation platforms in the Analytics & Experimentation domain, focusing on deterministic sticky assignment, configuration storage, eligibility targeting, traffic allocation, mutual exclusion, overrides, exposure logging, metric tagging, privacy/compliance, latency and availability targets, experiment lifecycle, and high-level architecture. It is commonly asked because it assesses both conceptual understanding and practical system-design application of distributed-system trade-offs, scalability, data integrity, operational reliability, and privacy constraints, and gauges the ability to define high-level APIs, architecture, and justify key trade-offs.
Community answers
Answer by chengchenghuang77
┌───────────────────────┐
│ Experiment Mgmt UI/API │ create / review / ramp / pause
└──────────┬────────────┘
▼
┌───────────────────────┐ strongly consistent, versioned,
│ Config DB + audit log │ audit trail (Postgres / Spanner)
└──────────┬────────────┘
▼
┌───────────────────────┐ validates conflicts, compiles an
│ Config compiler/publisher│ immutable snapshot vN (protobuf)
└──────────┬────────────┘
push (stream) │ + poll fallback (CDN / blob store)
┌─────────────────┼───────────────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────────────┐ ┌────────────────┐
│Server SDK│ │ Assignment API │ │ Edge/CDN copy │
│(in-proc │ │ stateless, multi-│ │ of snapshot │
│ eval) │ │ region, edge │ └────────────────┘
└────┬─────┘ └────────┬─────────┘
│ ▲ assign()
│ web / iOS / Android SDKs (cache + bootstrap)
│ │
└──────► exposure events ◄──────┘
▼
Collector → Kafka → Stream processor ──► guardrail monitor,
│ (dedupe, SRM) auto-killswitch
▼
Warehouse ──► Analysis engine ──► scorecards
Key components:
Management plane: UI and API, review workflow, audit log.
Config store: the source of truth. Strongly consistent and versioned.
Compiler/publisher: validates configs and produces an immutable snapshot configVersion = N.
Distribution: streaming push plus a polling fallback from blob storage or CDN.
Evaluation: server SDKs evaluate in process. A stateless Assignment API serves thin clients. Both run the same evaluation library so results match.
Exposure pipeline: collector → Kafka → stream processor → war