Choose lock granularity for concurrent storage

You are implementing a component inside a storage engine that is accessed concurrently (e.g., an in-memory index, metadata cache, or block map).

Discuss the trade-offs and propose an approach for concurrency control:

1. When would you use a coarse-grained lock (single global mutex)?
2. When do you need fine-grained locking (per-bucket/per-shard/per-node locks)?
3. How do lock choices affect latency (especially p99) and throughput under contention?
4. What pitfalls must be handled (deadlocks, priority inversion, convoying, starvation)?
5. When are lock-free / wait-free approaches feasible, and what are the engineering costs?

Assume you may have many threads, read-heavy workloads, and occasional writes (updates).