Design a hybrid-cloud GPU allocation service
Company: Mithril
Role: Software Engineer
Category: System Design
Difficulty: medium
Interview Round: Onsite
## System Design: Hybrid-Cloud GPU Resource Allocation & Job Management
Design a service that allocates and manages **GPU resources** for ML training jobs in a **hybrid cloud** environment (some GPUs on-prem, some in a public cloud).
### Core requirements
1. **Hybrid cloud scheduling**
- Select where to run jobs (on-prem vs cloud) based on GPU availability and policy.
2. **User job submission**
- Users can upload or reference:
- training program (code/container)
- training data
- resource needs (GPU type/count, CPU/RAM), and optional constraints (region, cost, priority).
3. **Long-running job monitoring**
- Track status for long-running training jobs: queued/running/completed/failed/canceled.
- Provide logs/metrics and basic retry semantics.
### Non-functional requirements (discuss and make reasonable assumptions)
- Multi-tenant fairness and quota.
- Reliability and failure handling (node failure, cloud API failure).
- Security for code/data (isolation, IAM).
- Scalability: many concurrent users and jobs.
### Deliverables
- High-level architecture and main components.
- APIs (submit job, get status, list jobs, cancel).
- Scheduling strategy and data model.
- How data/code is stored and made available to compute.
- Observability and operational concerns.
Quick Answer: This question evaluates system-design competency in distributed resource management, hybrid-cloud scheduling, GPU allocation, multi-tenant fairness, reliability, security, API design, and observability within the system design domain.