### 1) Clarify scope, assumptions, and rough scale I'll design a **Colab-like, multi-tenant cloud IDE** where the primary constraint is **safely running untrusted code**. Everything else (latency, density, cost) is optimized *subject to* that isolation guarantee. **In scope:** per-session isolated compute, secure execution of arbitrary code, near-real-time output streaming, durable workspace files, full session lifecycle. **Optional:** real-time collaboration (called out at the end). **Assumptions:** - Workloads are bursty and interactive — short cells, long idle gaps. Mostly Python/JS; a minority need a GPU (which, as §3 explains, forces a *different* substrate — GPUs can't ride the default microVM). - A session is single-user; multiple sessions belong to a workspace. - Untrusted by default: assume the user is an attacker trying to escape the sandbox, exfiltrate data, mine crypto, or reach internal services. - We control a cloud fleet (bare-metal or nested-virt-capable hosts) and can run a microVM layer; GPU hosts are provisioned separately. **Back-of-envelope (to ground density/lifecycle decisions):** - Target ~100k registered users, ~5k concurrent *active* sessions at peak, plus a long tail of *idle/suspended* sessions (say 10×, ~50k). - A modest session ≈ 1–2 vCPU, 2–4 GB RAM. So peak active ≈ 5–10k vCPU and 10–20 TB RAM. Sizing the fleet from both dimensions: at ~64–128 vCPU/host the vCPU side needs ~80–150 hosts; at ~256–512 GB RAM/host the memory side needs ~25–60 hosts. The vCPU side dominates, so **~80–150 CPU hosts** at peak (plus headroom for fragmentation and warm pools). GPU capacity is sized separately off GPU demand. - The idle/suspended tail must cost near-zero (no running CPU/RAM), which drives the **suspend-to-snapshot** strategy below. - Cold-start target: a fresh session should be interactive in **single-digit seconds** (p95), which rules out booting a full guest OS on the request path and motivates **warm pools + snapshot restore**. --- ### 2) High-level architecture Two planes: a multi-tenant **control plane** and a per-session **data plane**. The browser never talks directly to a runtime. ``` Browser (Monaco editor + notebook UI + xterm) │ HTTPS (control) │ WSS (output stream) ▼ ▼ ┌─────────────── Control Plane ───────────────┐ ┌──── Streaming Gateway ────┐ │ API Gateway / AuthN-AuthZ │ │ sharded WS terminators, │ │ Workspace Service (metadata, files index) │ │ auth, fan-out, replay, │ │ Session Manager (lifecycle, quotas) │ │ backpressure │ │ Scheduler/Capacity (bin-pack, autoscale) │ └────────────┬──────────────┘ │ Policy Engine (images, egress, limits) │ │ gRPC └──────────────┬───────────────────────────────┘ │ │ provision / control (gRPC) │ ▼ ▼ ┌──── CPU Host (bare metal) ────┐ per-session microVM │ Host Agent ── Firecracker/ │ ───► ┌──────────────────────────┐ │ jailer; warm microVM pool │ │ Guest kernel (locked down)│ └───────────────────────────────┘ │ Runtime Agent (exec, fs, │ │ log capture, health) │ ┌──── GPU Host (single-tenant) ─┐ │ user kernel (Jupyter-like)│ │ Host Agent ── full VM w/ PCIe │ └──────────────────────────┘ │ passthrough OR hardened │ │ NVIDIA-stack container │ └───────────────────────────────┘ Object store (workspace files, snapshots, large rich-output payloads) Log store (durable output + audit) ``` - **Control plane** services are stateless (any instance serves any request), backed by a metadata DB (Postgres) and a queue for async provisioning. - **Data plane** is two host fleets: **CPU hosts** running a warm pool of microVMs, and **GPU hosts** running the GPU substrate (see §3). Each host runs a **Host Agent** that manages its runtimes and launches/snapshots them. - **Streaming Gateway** is a *separate, stateful* tier (it holds per-session replay buffers and routing) — see §8 for how it's sharded. - **Runtime Agent** runs *inside* each runtime as the only privileged-ish process the platform controls; it execs user code, captures output, serves file ops, and reports health. The same agent runs in both the CPU and GPU substrates. --- ### 3) Compute substrate: VM vs container vs gVisor vs microVM The core decision. For **untrusted** code, a shared host kernel is the attack surface — a single kernel CVE means a full host compromise across tenants. That pushes toward kernel-level isolation. | Option | Isolation | Cold start | Density | Notes | |---|---|---|---|---| | Plain container (namespaces + cgroups) | Weak — shared host kernel, huge syscall surface | ~ms | Highest | Unacceptable alone for untrusted code; one kernel escape = host-wide breach | | Hardened container (seccomp + AppArmor + dropped caps + read-only rootfs) | Better, still shared kernel | ~ms | High | Reduces surface but doesn't eliminate the shared-kernel risk | | **gVisor** (userspace kernel intercepts syscalls) | Strong-ish — sandboxed kernel, no real VM | tens to hundreds of ms | High | Great middle ground; some syscall-compat gaps and CPU overhead | | **microVM** (Firecracker / Cloud Hypervisor) | Strong — hardware-virtualized, own guest kernel, tiny VMM device surface | ~100 ms boot, faster via snapshot | High (low per-VM overhead) | **Default choice for CPU-only untrusted execution** | | Full VM (QEMU/cloud image) | Strongest, but heavy device model | seconds | Low | Heavy/slow for the common case, **but the only path that supports GPU PCIe passthrough** | **Default (CPU) substrate: microVMs.** Firecracker gives near-VM isolation with a minimal device model (virtio-net, virtio-block, a serial console — no BIOS, no PCI), ~5 MB of memory overhead per VM, and ~125 ms boot. A common production pattern is **Kata Containers using Firecracker (or Cloud Hypervisor) as the VMM** so each "pod" is actually a microVM while keeping a Kubernetes-native control surface — a reasonable way to manage the fleet. gVisor is a good *secondary* tier for cheaper/short-lived "scratch" runs where the perf overhead is acceptable. **GPU substrate (explicit, separate path).** Firecracker's minimal device model is exactly why it *cannot* host a GPU: it has **no PCI bus and no PCIe/VFIO passthrough**, so a GPU device cannot be attached to a Firecracker microVM. GPU sessions therefore run on a **different substrate**, on a separate pool of **single-tenant GPU hosts**: - **Preferred:** a **full VM (QEMU/Cloud Hypervisor) with VFIO PCIe passthrough** of a dedicated GPU (or a MIG slice). This keeps the per-tenant-kernel isolation property while exposing the device; it pays a heavier boot/device-model cost, mitigated with a small warm pool of pre-attached GPU VMs. - **Alternative** where passthrough VMs aren't available: a **hardened container with the NVIDIA container stack** (seccomp + AppArmor + dropped caps + read-only rootfs) on a host that is **never co-tenanted** with another customer for the GPU's lifetime — accepting the weaker (shared-kernel) isolation explicitly, and confining the blast radius by single-tenanting the whole host. - Either way, GPU hosts are **physically separate from the CPU microVM fleet**, scheduled from their own pool (§11), and a GPU session never shares a host with another tenant. So the rule is: **microVM by default; GPU sessions are routed to the GPU pool and run as passthrough full-VMs (or single-tenant hardened containers).** The substrate is chosen per session from the requested `gpu` count. Operationally (CPU fleet): - Run Firecracker under **`jailer`** (chroot, dedicated uid/gid, cgroup, separate network namespace) so even a VMM compromise is contained on the host. - Hosts are bare-metal (or nested-virt enabled) since Firecracker needs KVM. - One runtime per session, never shared across tenants — on both fleets. --- ### 4) Isolation model (filesystem, process, network, credentials) This is the security core; isolation is layered so no single bypass is fatal. The controls below apply to both substrates; the GPU substrate additionally relies on single-tenant host placement to compensate for its weaker (when container-based) kernel isolation. **Process / kernel:** each session = its own guest kernel (microVM, or passthrough full VM on GPU). Inside, the user process runs unprivileged with `seccomp` (deny dangerous syscalls), dropped Linux capabilities, and cgroups for CPU/mem/PID/IO limits. The host never trusts the guest. **Filesystem:** - Guest rootfs is a **read-only base image** (language toolchain + agent) plus a **writable overlay** for the user's workspace and installed packages. - Disk quota on the overlay (per-tier cap) to stop a user filling the host. - No host paths are bind-mounted into the guest beyond the controlled block device. **Network (most-attacked path):** - Each runtime gets its own **network namespace + veth pair + NAT**; no L2 adjacency between tenants. - **Default-deny egress.** Outbound goes through an **egress proxy** that allowlists package repos (PyPI, npm, conda, apt mirrors). Everything else is dropped. - **Block the cloud metadata endpoint** (`169.254.169.254` and the link-local range) at the host firewall — this is the classic SSRF→credential-theft path on cloud providers. As defense-in-depth, **also enforce IMDSv2** (token-required, session-oriented, with a hop limit of 1) on every host so that even if the firewall block regressed, a guest can't trivially reach the metadata service via SSRF. - No inbound from the internet; the runtime is reachable only by the control plane and streaming gateway over an internal network. **Credentials / identity:** - The guest holds **no node-level or cloud credentials**. The Host Agent never injects long-lived secrets. - If a session needs to reach an internal resource (e.g., the user's object-store prefix), it uses a **short-lived, narrowly-scoped token** minted per session, audited, and expiring on idle/terminate. - Snapshots restored from a warm pool must **re-seed entropy and per-session identity** so two sessions don't start from identical RNG state or share a token. **Supply chain:** base images are signed and scanned; package installs flow through the proxy where they can be scanned/cached; abusive runtime behavior (sustained max CPU/GPU, known crypto-miner signatures, mass-egress attempts) is detected and the session throttled or killed. --- ### 5) Session lifecycle States and transitions: ``` Provisioning ──► Running ──► Idle ──► Suspended ──► Terminated ▲ │ │ └───────────┴──────────┘ (resume) ``` - **Provisioning:** Session Manager admits the request (quota check), Scheduler picks a host from the **right pool** (CPU vs GPU, per the requested `gpu` count), Host Agent **claims a warm runtime** and attaches the workspace overlay. Because the runtime is pre-booted (or snapshot-restored), this is sub-second to a few seconds rather than a full boot. (GPU passthrough VMs warm-pool more coarsely since they're heavier and scarcer.) - **Running:** Runtime Agent serves exec/fs; output streams out. - **Idle:** no active client WS *and* low CPU for N minutes → mark Idle (still resident, but a candidate for suspend). - **Suspend (key cost lever, CPU substrate):** snapshot the microVM (memory + disk state) to the host/object store and **free the host's CPU and RAM**. This is what makes the 50k-idle tail cheap. Firecracker's **snapshot/restore** persists full VM memory; resume restores it. Note GPU passthrough VMs **don't snapshot-resume cleanly** (device state isn't captured), so idle GPU sessions are instead **checkpointed to the workspace and torn down** — GPU is too expensive to leave resident anyway. - **Resume (CPU substrate):** restore the snapshot (fast — memory is loaded back, no re-boot, no re-`import`), re-attach networking with fresh credentials/entropy. Resume from a local snapshot is much faster than from object store, so keep recent snapshots warm-local and tier older ones off. - **Terminate:** flush final files/logs, destroy the runtime, release the overlay (after a durable checkpoint), revoke tokens. **Cold-start strategy:** maintain **warm pools** of pre-booted base microVMs per language/image so the request path is "claim + attach", not "boot". Pool size is autoscaled off arrival rate. For popular notebook stacks, keep **post-import snapshots** (kernel already started, heavy libs imported) so the user's first cell is instant. **Quotas / limits:** per-user/org caps on concurrent sessions, vCPU, RAM, GPU count, and wall-clock for free tier; hard idle timeouts to reclaim resources. --- ### 6) Data persistence Three distinct durability classes — conflating them is the common mistake: | Class | Where | Durability | Resume cost | |---|---|---|---| | **Workspace files** (code, notebooks) — source of truth | Versioned **object store** (optionally git-backed) | Durable, survives session death | n/a — fetched on attach | | **Runtime overlay** (installed pkgs, caches) | Local ephemeral disk + periodic snapshot | Best-effort; rebuildable | Restore snapshot (fast) or re-install (slow) | | **Live VM memory** (CPU suspend/resume) | Firecracker snapshot, local then tiered | Best-effort | Restore = near-instant interactivity | **Strategy:** - **Object store is the durable home** of workspace files. The Runtime Agent autosaves notebooks/files and checkpoints to object storage on a timer and on suspend/terminate. So even if a host dies, the user loses at most the last few seconds of unsaved edits — never their project. - The **overlay** (packages, caches) is treated as a cache: persisted via snapshot for fast resume, but reconstructable from the workspace's environment spec if lost. - **Paid tier** can get a **persistent network volume** mounted per workspace for large datasets, accepting the throughput/contention tradeoff vs. cheap object-store + local-disk for free tier. **Tradeoff to state explicitly:** network volumes make resume trivial but can bottleneck and add per-IOPS cost; snapshot-to-object-store is cheaper at steady state but adds resume latency. Default to object-store + local ephemeral + snapshots; offer volumes as an upgrade. --- ### 7) Execution model Inside each runtime, the **Runtime Agent** is the platform's trusted entrypoint. It supervises a language kernel (a **Jupyter-style kernel protocol** internally is a clean fit — execute requests, stream results, interrupt) but the design stays kernel-agnostic. It exposes an internal (control-plane-only) API: - `POST /execute` — run a cell/command; returns an `execution_id`, streams output via the agent→gateway channel. - `POST /interrupt` — SIGINT the running cell (Ctrl-C). - `GET /health` — kernel + resource status (for idle detection and liveness). - `GET /fs/{path}` / `PUT /fs/{path}` / `DELETE /fs/{path}` — file ops (upload/download, list). Output is captured as a typed stream so the UI can render **rich output**: `stdout`, `stderr`, structured events (`cell_started`, `cell_finished`, `exit_code`), and MIME-typed payloads (text, image/png, HTML) for plots and tables. **Rich-output sizing (don't inline large blobs).** Plots/tables can be large, so they're *not* shoved raw onto the same low-latency text stream. The agent applies a per-payload threshold (e.g., ~256 KB): below it, the MIME payload rides the stream inline; above it, the agent **writes the blob to object store and emits a small reference event** (`{mime, object_key, size, seq}`) that the browser fetches out-of-band. This keeps a single large figure from blowing the streaming buffer or stalling stdout, and is also why the truncation logic in §8 is framed around the text/event stream, not multi-megabyte images. --- ### 8) Log / output streaming (near-real-time, resumable) Goal: low-latency stdout/stderr + structured events to the browser, survivable across reconnects, with bounded buffering. **Path:** 1. Runtime Agent captures kernel output into a **local bounded ring buffer**, tagging each chunk with `(execution_id, stream, seq)` where `seq` is a **monotonic per-stream sequence number**. 2. Agent pushes chunks to the **Streaming Gateway** over gRPC (internal network only — runtimes are never exposed to the internet). 3. Gateway authenticates the browser's WSS connection (scoped session token), then **fans out** chunks to the client. It also **asynchronously persists** output to a durable log store (debugging + audit), decoupled from the live path so log-store latency never stalls the user. 4. Browser renders incrementally; ack'd `seq` lets the gateway advance. **Gateway scaling, state, and reconnect routing.** The gateway is the one *stateful* tier (it holds each session's replay buffer and the agent↔client routing), so the "stateless control plane" claim in §2 explicitly does **not** cover it. It scales horizontally with **consistent-hash sharding by `session_id`**: a session's agent stream and its browser WSS both map to the same gateway shard, so fan-out and replay are local to one instance. The browser is given the **shard-stable endpoint** (via the session record / a routing layer that hashes `session_id`), so a reconnecting client lands back on the **same shard that holds its ring buffer**. If that shard has restarted or the buffer was evicted, the gateway **rehydrates from the durable log store** using `last_seq` before resuming live fan-out — so a shard failover degrades to a slightly slower replay, never lost output. At ~5k concurrent WSS this fits comfortably in a handful of shards; the shard ring is the autoscaling unit. **Delivery semantics & robustness:** - **Reconnect/replay:** on reconnect the client sends `last_seq`; the gateway replays missing chunks from its buffer (or the durable log if evicted). Delivery is **at-least-once**, idempotent on the client via `seq` dedup — the user never silently loses output mid-run. - **Backpressure:** if the client is slow or gone, buffer up to a cap (e.g., a few MB / N seconds) of *text+events*, then drop the oldest with an explicit `"... output truncated ..."` marker rather than OOMing the gateway or stalling execution. Execution keeps running. (Large rich-output blobs don't count against this cap — they're sideloaded per §7.) - **Output floods:** the agent rate-limits/caps total output per execution (a `while True: print()` loop must not DOS the pipeline) and emits a truncation event. - **Multiplexing:** one WSS per session with logical channels (`stdout`, `stderr`, `events`) so the client demuxes cleanly. - **Decoupling running from watching:** if the user disconnects, the cell keeps running (policy choice — matches Colab-style behavior); output buffers and the idle timer governs reclamation. A gateway (vs. direct runtime→browser) centralizes auth, rate limiting, TLS termination, protocol translation, and keeps runtimes off the public internet. --- ### 9) Data model (control plane) ```sql workspace(id, owner_id, org_id, name, env_spec, default_image, created_at, last_active_at) session(id, workspace_id, host_id, runtime_id, substrate /*microvm|gpu_vm|gpu_container*/, state, image, cpu, mem, gpu, gateway_shard, token_id, created_at, idle_since, expires_at) snapshot(id, session_id, kind /*disk|mem*/, location, size_bytes, created_at) file_index(workspace_id, path, object_key, version, size, checksum, updated_at) quota(subject_id /*user|org*/, max_concurrent, max_vcpu, max_mem, max_gpu, tier) audit_event(id, actor, action, session_id, image, net_policy, ts) ``` (`substrate` records which compute path served the session; `gateway_shard` lets a reconnecting client be routed back to the gateway instance holding its buffer.) --- ### 10) API sketch (control plane + streaming) ``` # Workspaces & files POST /v1/workspaces -> create workspace GET /v1/workspaces/{id}/files -> list files PUT /v1/workspaces/{id}/files/{path} -> upload GET /v1/workspaces/{id}/files/{path} -> download # Sessions (lifecycle) POST /v1/workspaces/{id}/sessions -> create/attach runtime {cpu,mem,gpu,image} (gpu>0 routes to the GPU substrate/pool) GET /v1/sessions/{sid} -> state + connection info (incl. gateway endpoint) POST /v1/sessions/{sid}:suspend -> snapshot + free host resources (CPU substrate) POST /v1/sessions/{sid}:resume -> restore snapshot POST /v1/sessions/{sid}:terminate -> checkpoint + destroy # Execution (proxied through gateway to runtime agent) POST /v1/sessions/{sid}/execute {code} -> {execution_id} POST /v1/sessions/{sid}/interrupt # Output stream (WebSocket; endpoint is shard-stable for this session) GET wss://gw/v1/sessions/{sid}/stream?token=...&last_seq=... server frames: {type: stdout|stderr|event, execution_id, seq, mime, data} large rich-output frames carry {object_key,size} instead of inline data ``` --- ### 11) Scheduling, autoscaling, and fairness - **Admission control** in Session Manager: reject/queue at quota boundaries before any host work. - **Substrate routing:** the requested `gpu` count selects the pool. `gpu == 0` → CPU microVM fleet; `gpu > 0` → GPU host pool (passthrough VM or single-tenant hardened container). This is decided at admission so a GPU request never lands on a Firecracker host that can't satisfy it. - **Scheduler bin-packs** sessions onto hosts by `(cpu, mem, gpu)` with separate pools — **CPU-only, GPU (single-tenant), high-mem** — so a GPU session doesn't strand CPU capacity and vice versa. Prefer hosts with a warm runtime of the requested image (locality cuts cold start). - **Autoscaling:** scale each host pool off its own active+queued demand and warm-pool depletion. Suspended CPU sessions consume no host CPU/RAM (only snapshot storage), so the CPU fleet scales with *active* load, not registered users. GPU capacity is scaled and budgeted independently since idle GPU sessions are torn down rather than suspended. - **Fairness:** weighted fair sharing per org; for free tier, preempt-to-suspend the longest-idle CPU sessions when capacity is tight (their state is snapshotted, so preemption is non-destructive). GPU is gated by hard quotas rather than oversubscribed. Hard wall-clock and idle caps on free tier. --- ### 12) Observability **Metrics (SLIs):** session-start latency p50/p95 (warm vs. cold, per substrate), warm-pool hit rate, resume latency, streaming lag and dropped-chunk rate, running/idle/suspended counts, CPU vs GPU host utilization, noisy-neighbor / preemption events, snapshot size & age, gateway shard fan-out and rehydration rate. **Tracing:** end-to-end span on `create session → admit → choose substrate → schedule → claim runtime → attach overlay → agent-ready`, so cold-start regressions are attributable to a stage. **Audit logs (security-critical):** who started which session with which image, substrate, and network policy; **denied egress attempts**, metadata-endpoint hits, token mints/uses, suspicious-syscall signals. These feed abuse detection and incident response. --- ### 13) Failure modes & edge cases - **Cold start too slow** → warm pools + post-import snapshots + small base images + image caching on hosts (GPU warm pools are smaller and coarser due to cost). - **GPU requested but no Firecracker support** → handled at admission: `gpu > 0` is routed to the GPU substrate/pool (passthrough VM or single-tenant container), never to a microVM. - **Host dies mid-session** → workspace files are already durable in object store (autosave + checkpoint); user reconnects and a new session re-attaches the workspace. In-flight unsaved output is best-effort from the durable log. - **Gateway shard dies / buffer evicted** → reconnecting client is re-routed to the (restarted or replacement) shard for its `session_id`, which rehydrates the replay buffer from the durable log using `last_seq` before resuming live fan-out. - **User installs huge deps / fills disk** → overlay disk quota + cached package layers; OOM-kill the cell, not the host. - **Infinite output / fork bombs / crypto-mining** → output caps, cgroup CPU/PID limits, egress denial + abuse signals → throttle or terminate. - **Disconnected client** → execution continues; idle timer governs suspend; output replayable on reconnect. - **Snapshot reuse leaking entropy/identity** → re-seed RNG and mint fresh per-session credentials on every restore. - **Secrets leakage** → never mount broad cloud credentials; scoped per-session tokens only; metadata endpoint blocked *and* IMDSv2-enforced. --- ### 14) Optional: collaboration Separate the two concerns: - **Document state** (who's typing what) → a **CRDT** (or OT) sync service over WebSocket, independent of the runtime. This scales to many viewers/editors cheaply. - **Execution** → keep a **single owner kernel** per notebook (one runtime) to avoid contention and permission ambiguity; collaborators see streamed output via the same gateway fan-out but exec requests are owner-gated (or queued). Trying to share one runtime symmetrically across users creates resource-contention and trust problems, so the clean model is "shared editing, single-owner execution," with per-user runtimes if true parallel execution is needed. --- **Summary:** microVMs (Firecracker, run under `jailer`) as the **default CPU** untrusted-execution substrate, with GPU sessions routed to a **separate single-tenant pool** running full VMs with PCIe passthrough (or hardened NVIDIA-stack containers) because Firecracker has no PCI bus; layered isolation (own guest kernel, read-only base + quota'd overlay, default-deny egress with the metadata endpoint blocked *and* IMDSv2-enforced, no node credentials + short-lived scoped tokens); warm pools + snapshot/restore for sub-second-to-few-second start and near-zero-cost idle on the CPU fleet; object-store as the durable home for workspace files (with large rich-output blobs sideloaded there too); and a resumable, backpressure-aware, sequence-numbered streaming pipeline through a **session-sharded** internal gateway so output is real-time, lossless-on-reconnect, and the runtimes never face the internet.