Design an object-oriented, GPU-aware pod scheduler and cluster manager. Each Node has the shape {name: string, total_gpu: int, running_pods: Pod[]}. Each Pod has the shape {name: string, gpu_required: int}. Implement APIs: add_node(name, total_gpu), remove_node(name), add_pod(name, gpu_required), schedule_pod(pod_name) that assigns the pod to a node with enough free GPUs, remove_pod(pod_name), get_node_utilization(name), and list_nodes()/list_pods(). Specify data structures to support efficient lookups of nodes by available GPUs and pods by name. Describe and justify a placement strategy (e.g., best-fit or first-fit) and how you'd update indexes on every add/remove/schedule/evict operation. Discuss concurrency control (simultaneous adds/schedules), idempotency, and failure handling (e.g., removing a node that still has running pods, pod rescheduling on node removal). Provide time and space complexity for each API and write pseudocode for schedule_pod using your chosen strategy. Include edge cases like gpu_required > total_gpu on any node and fragmentation when multiple small pods occupy a large node.

This interview question evaluates requirements, scale assumptions, API/data design, architecture, trade-offs, failure modes, and rollout in a realistic interview setting. A strong answer for Design a GPU-aware pod scheduler states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

How do I approach System Design interview questions?

System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master system design interviews.

What difficulty level is this interview question?

This is a hard difficulty System Design question, commonly asked during Technical Screen rounds at Together AI.

What role is this question designed for?

This question is commonly asked for Software Engineer candidates at Together AI during technical interviews.

Design a GPU-aware pod scheduler | Together AI Interview Question

Design a GPU-aware pod scheduler

GPU-Aware Pod Scheduler and Cluster Manager (OO Design)

Context

You are designing a simplified, object-oriented cluster manager with a GPU-aware pod scheduler. Nodes provide a fixed number of GPUs. Pods request a fixed number of GPUs and must be placed on a single node with enough free GPUs.

Each Node has the shape: { name: string, total_gpu: int, running_pods: Pod[] } Each Pod has the shape: { name: string, gpu_required: int }

Assume pods cannot be split across nodes and GPUs are fungible (no topology/NUMA awareness).

Requirements

Implement APIs:

add_node(name, total_gpu)
remove_node(name)
add_pod(name, gpu_required)
schedule_pod(pod_name) — assigns the pod to a node with enough free GPUs
remove_pod(pod_name)
get_node_utilization(name)
list_nodes() / list_pods()

Also provide:

Data structures to support efficient lookups of nodes by available GPUs and pods by name.
A placement strategy (e.g., best-fit or first-fit) and justification.
How to update indexes on every add/remove/schedule/evict operation.
Concurrency control for simultaneous adds/schedules, idempotency, and failure handling (e.g., removing a node that still has running pods; pod rescheduling on node removal).
Time and space complexity for each API.
Pseudocode for schedule_pod using your chosen strategy.
Edge cases, including gpu_required > total_gpu on any node and fragmentation when multiple small pods occupy a large node.

Constraints & Assumptions

Preserve the scope, facts, inputs, and requested outputs from the prompt above.
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask Guidance

Clarify users, core use cases, read/write patterns, scale, latency, availability, and data retention.
State explicit assumptions before making sizing or architecture decisions.
Prioritize the functional path first, then address reliability, security, observability, and rollout.

What a Strong Answer Covers Guidance

A scoped requirements summary with concrete non-goals and success metrics.
API, data model, architecture, consistency, capacity, and operations.
Reasoned trade-offs among simple and scalable designs, including bottlenecks and failure modes.
A validation, monitoring, migration, and launch plan appropriate for the risk level.

Follow-up Questions Guidance

What breaks first at 10x traffic or data volume?
How would you degrade gracefully during dependency failures?
What metrics and alerts would prove the design is healthy after launch?

Design a GPU-aware pod scheduler

GPU-Aware Pod Scheduler and Cluster Manager (OO Design)

Context

Each Node has the shape: { name: string, total_gpu: int, running_pods: Pod[] } Each Pod has the shape: { name: string, gpu_required: int }

Assume pods cannot be split across nodes and GPUs are fungible (no topology/NUMA awareness).

Requirements

Implement APIs:

add_node(name, total_gpu)
remove_node(name)
add_pod(name, gpu_required)
schedule_pod(pod_name) — assigns the pod to a node with enough free GPUs
remove_pod(pod_name)
get_node_utilization(name)
list_nodes() / list_pods()

Also provide:

Data structures to support efficient lookups of nodes by available GPUs and pods by name.
A placement strategy (e.g., best-fit or first-fit) and justification.
How to update indexes on every add/remove/schedule/evict operation.
Concurrency control for simultaneous adds/schedules, idempotency, and failure handling (e.g., removing a node that still has running pods; pod rescheduling on node removal).
Time and space complexity for each API.
Pseudocode for schedule_pod using your chosen strategy.
Edge cases, including gpu_required > total_gpu on any node and fragmentation when multiple small pods occupy a large node.

Constraints & Assumptions

Preserve the scope, facts, inputs, and requested outputs from the prompt above.
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask Guidance

Clarify users, core use cases, read/write patterns, scale, latency, availability, and data retention.
State explicit assumptions before making sizing or architecture decisions.
Prioritize the functional path first, then address reliability, security, observability, and rollout.

What a Strong Answer Covers Guidance

A scoped requirements summary with concrete non-goals and success metrics.
API, data model, architecture, consistency, capacity, and operations.
Reasoned trade-offs among simple and scalable designs, including bottlenecks and failure modes.
A validation, monitoring, migration, and launch plan appropriate for the risk level.

Follow-up Questions Guidance

What breaks first at 10x traffic or data volume?
How would you degrade gracefully during dependency failures?
What metrics and alerts would prove the design is healthy after launch?

Design a GPU-aware pod scheduler

Quick Overview

Design a GPU-aware pod scheduler

Design a GPU-aware pod scheduler

GPU-Aware Pod Scheduler and Cluster Manager (OO Design)

Context

Requirements

Constraints & Assumptions

Clarifying Questions to Ask Guidance

What a Strong Answer Covers Guidance

Follow-up Questions Guidance

Submit Your Answer to Earn 20XP

Design a GPU-aware pod scheduler

Quick Overview

Design a GPU-aware pod scheduler

Design a GPU-aware pod scheduler

GPU-Aware Pod Scheduler and Cluster Manager (OO Design)

Context

Requirements

Constraints & Assumptions

Clarifying Questions to Ask Guidance

What a Strong Answer Covers Guidance

Follow-up Questions Guidance

Submit Your Answer to Earn 20XP