System Design: Tamper-Evident Audit Logging for Video Evidence Chain of Custody

Design a scalable, tamper-evident audit logging system for video evidence that preserves an accurate chain of custody. The system must record all actions (upload, view, redact/delete sensitive data, transcribe), never lose an event, and support legal discovery.

Context and assumptions (add or adjust as needed):

• Multi-tenant environment supporting thousands of agencies and millions of assets (videos, photos, transcripts).
• Peak audit event rate: 10–50k events/sec; steady state 1–5B events/month.
• Chain-of-custody must be legally defensible, with immutable, verifiable proof of event history.
• Devices and users may be offline and later reconnect; the system must handle late/out-of-order arrivals without loss.

Specify the following:

1. Functional and non-functional requirements.
2. Architecture and storage choices (e.g., append-only log, replication, WORM/immutable storage).
3. Event schema, ordering, idempotency, and delivery guarantees (exactly-once vs at-least-once).
4. Integrity mechanisms (hash chaining/Merkle trees, digital signatures), encryption and key management, and access controls.
5. Scalability, partitioning, and backpressure handling.
6. APIs and query patterns for auditors and investigators.
7. Retention policies, legal hold, and compliance considerations.
8. Monitoring, audit-of-the-auditor, and disaster recovery.
9. Cost/latency trade-offs and capacity planning.
10. Testing, verification, and rollout plan.