Safety Policy Enforcement and Audit Systems

What's being tested

Interviewers are probing your ability to design reliable, scalable policy enforcement and audit systems that operate in production: low-latency enforcement in the request path, durable tamper-evident recording for later review, and clear tradeoffs between correctness, performance, and observability. They want to see system decomposition, consistency/availability reasoning, capacity and cost estimates, error/edge-case handling, and test/operational plans a backend engineer would own.

Core knowledge

Enforcement point vs audit trail separation: enforcement must meet strict latency/SLA (often synchronous), while audit storage can be asynchronous and optimized for durability and queryability.
Synchronous vs asynchronous enforcement tradeoff: synchronous provides immediate correctness but increases p99 latency; asynchronous reduces request latency but requires reconciliation and eventual consistency.
Idempotency & deduplication: use unique request identifiers and idempotency keys persisted in a small low-latency store (`Redis`, unique DB constraint) to achieve exactly-once semantics or safe retries under at-least-once delivery.
Append-only audit logs: store events in an immutable append-only stream (`Kafka`, WORM `S3` with versioning); append-only simplifies tamper-evidence, replayability, and schema evolution via version tags.
Storage sizing and retention math: estimate storage: bytes/day = events/sec * avg-event-bytes * 86400; multiply by retention days. For example, 1000 eps * 1KB * 86400 ≈ 86.4GB/day; choose partitioning and compaction accordingly.
Indexing and partitioning for audit queries: design partitions by time + primary keys (e.g., day + account_id) so ad-hoc queries and replays scan minimal partitions; maintain materialized views or OLAP replicas for heavy query loads.
Tamper-evidence mechanisms: use cryptographic hashes/MACs or Merkle trees over batched logs, signed by a key in `KMS` to prove integrity; separate write and read roles via strict RBAC to prevent admin tampering.
Exactly-once vs at-least-once in distributed pipes: prefer idempotent consumers and deduplication over heavyweight distributed transactions; use transactional producers (`Kafka` transactions) when strong ordering/atomicity is required.
Latency budgets and SLAs: set clear p50/p95/p99 targets for the enforcement path (e.g., p99 < 100–300ms). Monitor end-to-end latency including policy evaluation, network, and I/O.
Observability & testing: instrument with histograms, counters, and distributed tracing for enforcement calls; populate rule coverage and false-positive/negative metrics; include synthetic traffic, replay tests, and chaos tests for partial-failure resilience.

Worked example — "Design a low-latency policy enforcement service that blocks prohibited content and provides an audit trail"

Frame first: clarify SLA (p99 latency target), expected throughput (req/sec), types of decisions (binary block/allow or actions), and retention/compliance requirements. Skeleton: (1) a lightweight enforcement service in the request path that performs a cached policy check and short-circuits, (2) a policy evaluation engine that fetches rules from a config store and supports versioning, (3) an append-only audit publisher to `Kafka` (or direct DB write for small scale) to stream events, and (4) an asynchronous worker that persists enriched audit records into long-term storage (`S3` + partitioned Parquet). Flag tradeoffs: synchronous publishing to audit store increases request latency and coupling; prefer fire-and-forget to a fast local buffer with background flush and durable write to `Kafka` to avoid lost events. Close by describing operational controls: rollout via canary, metric dashboards for enforcement rate and false positives, and a replay harness. If more time: add tamper-evidence (signed batches), per-tenant retention, and multi-region replication for compliance.

A second angle — "Design an offline auditing system that replays historical interactions to detect missed violations"

Here the service is oriented to high-throughput batch processing, not low-latency decisioning. Use the append-only event stream as the source of truth and build a scalable replay pipeline with consumer groups reading time-partitioned topics. Important constraints change: ordering within keys matters for correct re-evaluation, so partition by subject-id; compute nodes should be stateless and idempotent, writing findings to an OLAP store for analysts. Emphasize cost/throughput tradeoffs: prefer bulk-compute frameworks (e.g., `Spark`/`Flink`) for large backfills; for continuous detection use stream processors with windowed state and retention. Ensure the pipeline supports schema evolution (versioned event payloads) and reconciliation hooks for recovered events.

Common pitfalls

Pitfall: Assuming synchronous writes to an external audit DB are free.
Synchronous writes amplify p99 latency and create cascading failures. Design a durable local buffer or use `Kafka` with producer retries; treat the audit path as a critical-but-asynchronous dependency unless SLA dictates otherwise.

Pitfall: Treating audit logs as mutable application data.
Allowing ad-hoc edits destroys forensics. Enforce append-only writes, strict RBAC, and cryptographic integrity checks. Provide correction records (new events marking rollbacks) rather than mutating history.

Pitfall: Skipping replayability and schema/versioning upfront.
Without version metadata and backward-compatible event schemas, re-processing historical events becomes brittle. Embed schema version, provide per-event timestamps, and maintain a schema registry or simple `version` field.

Connections

This area ties closely to observability & tracing (distributed traces help trace enforcement decisions), storage & partitioning (OLTP vs OLAP separation), and security/compliance (RBAC, `KMS`, and data-retention policy). Interviewers may pivot to distributed consensus (Raft) or exactly-once semantics in message systems.