Design scalable worker pool for template jobs

You have implemented a function that takes a template string (or many template strings) and replaces placeholders like `{{db_host}}` and `{{db_port}}` using a key–value dictionary. Now consider that you run this in production as a backend service and must process **a very large volume of such replacement jobs** (e.g., millions of template strings per hour). Each job consists of: - An identifier - A template string (or a small set of template strings) - A dictionary of substitution values The system must: - Handle high throughput and scale horizontally. - Avoid being blocked by slow or heavy jobs. - Use a **worker pool** model to process jobs concurrently. ### Design Tasks 1. **High-level architecture** Design a system that can process a large number of template-substitution jobs reliably. Describe the main components (e.g., API layer, queues, workers, data stores) and how they interact. 2. **Worker pool mechanism** Explain in detail how the worker pool works. In particular: - How are jobs produced and put into the system? (Describe the **producer** side.) - How are jobs consumed by workers? (Describe the **consumer** side.) - How does this implement the classic **producer–consumer model**? - How do you control concurrency and avoid overloading the system? 3. **Work distribution among workers** Suppose you want to distribute jobs across multiple workers. Discuss: - Different strategies for assigning jobs to workers (e.g., round-robin, random, hashing by key, multiple queues vs. a single shared queue). - When and why you might choose **hash-based assignment on some key** (for example, to keep all jobs related to a given customer or resource on the same worker). - Trade-offs between fairness, load balancing, and preserving ordering for related jobs. 4. **Scalability and reliability** Explain how your design: - Scales out when job volume increases (e.g., adding more workers, sharding queues). - Handles failures (e.g., worker crashes in the middle of a job, retry logic, idempotency). - Provides backpressure so that producers do not overwhelm the system. 5. **Implementation considerations** Briefly discuss: - What technologies you might use for the queue (e.g., Kafka, RabbitMQ, cloud message queues) and why. - Metrics and monitoring you would put in place (e.g., queue length, worker utilization, job latency). - Any specific optimizations for this string-template-replacement domain (e.g., caching parsed templates, batching jobs). Provide a step-by-step, detailed design explaining your reasoning and the trade-offs you are making.