One Line of Code Halved Our Storage. Another Stopped Us Re-Encoding Two-Hour Films.

Your transcode atom is the segment, not the video. How one retry key and one packaging merge cut our storage in half at 260 PB a year.

The farm re-encoded a whole film because six seconds broke

Our transcode farm re-encoded a full-length film, every rendition of it, because one six-second segment came back corrupt. I know exactly why. I wrote the job key, and I keyed the work by video ID because that was the simple thing to do on a Friday.

Look, the pipeline was textbook. Split the video, fan it out to a farm of FFmpeg workers, reassemble. Every system design page draws that same DAG. What none of them tell you is where the money leaks once real traffic lands on it.

Nobody audits the transcode bill because the DAG works

At the scale this assumes, 260 PB a year of stored media, two lines on the cloud bill dwarf everything else: transcode compute and segment storage. Both hide well. The DAG completes, videos play, dashboards stay green. So nobody looks.

A pipeline can produce correct output and still be a money pit.

The naive assumption: a video is the unit of work

Most engineers assume a video is a unit of work. You submit one, you get one back. So you split ingest and delivery, correctly, because they pull in opposite directions. Upload is compute-heavy and bursty. Viewing is bandwidth-heavy and constant. You never let one starve the other, and you keep metadata in SQL sharded by video_id, media in blob, never both in one store.

Ingest is standard: chunked, resumable upload to blob, GOP-aligned so every chunk decodes on its own, app servers off the byte path.

But the assumption that a video is the atom breaks one layer down.

Your transcode atom is the segment, not the video.

The two paths never share a spine. Upload is compute-heavy and bursty, viewing is bandwidth-heavy and constant, so bytes go straight to blob and never touch app servers.

The two bills I signed on the same Friday

The first problem was the job key. Because I'd keyed it by video ID, a failure anywhere inside a job, one worker OOM, one spot instance reclaimed mid-encode, one corrupt segment, retried the whole video. Two hours of footage re-encoded across every rendition because six seconds failed. On a farm running thousands of concurrent encodes, that's not an edge case. That's just the afternoon.

The unit of work in a transcode pipeline is (video, rendition, segment). Everything expensive that goes wrong, goes wrong at that granularity. Key the work there and a failure retries one segment in one rendition. Nothing else moves.

Fixing the retries stopped the farm burning compute on failures. But the storage bill was still double what it needed to be, and that took a different fix entirely.

The same Friday decision that keyed the job by video ID had locked in a second mistake. I just didn't see it for a year.

The second problem was packaging, and it was subtler. I'd wired FFmpeg to emit an HLS package and a separate DASH package, because HLS wanted MPEG-TS and DASH wanted fMP4, and that was the documented way. So every video sat in storage twice. Same frames, same bitrates, two copies, because the two protocols disagreed on their container.

They don't have to. CMAF gives you one set of fMP4 segments that both an HLS playlist and a DASH manifest can point at. You package once.

HLS and DASH are two tables of contents. Store the book once.

The two bills I signed on the same Friday

The first problem was the job key. Because I'd keyed it by video ID, a failure anywhere inside a job, one worker OOM, one spot instance reclaimed mid-encode, one corrupt segment, retried the whole video. Two hours of footage re-encoded across every rendition because six seconds failed. On a farm running thousands of concurrent encodes, that's not an edge case. That's just the afternoon.

The unit of work in a transcode pipeline is (video, rendition, segment). Everything expensive that goes wrong, goes wrong at that granularity. Key the work there and a failure retries one segment in one rendition. Nothing else moves.

Fixing the retries stopped the farm burning compute on failures. But the storage bill was still double what it needed to be, and that took a different fix entirely.

The same Friday decision that keyed the job by video ID had locked in a second mistake. I just didn't see it for a year.

The second problem was packaging, and it was subtler. I'd wired FFmpeg to emit an HLS package and a separate DASH package, because HLS wanted MPEG-TS and DASH wanted fMP4, and that was the documented way. So every video sat in storage twice. Same frames, same bitrates, two copies, because the two protocols disagreed on their container.

They don't have to. CMAF gives you one set of fMP4 segments that both an HLS playlist and a DASH manifest can point at. You package once.

HLS and DASH are two tables of contents. Store the book once.

One encode, one segment set, two manifests. The picture the docs draw, and the one I ignored for a year while paying to store every segment twice.

The line that lied, and a flag I never questioned

The retry key that cost us a farm's worth of wasted compute:

job_key = video_id                              # any failure re-runs the ENTIRE video

job_key = (video_id, rendition, segment_index)  # now only the broken segment retries

The packaging mistake was quieter. No bug, just a config flag I copied from the docs and never questioned for a year:

same frames, written to disk twice

ffmpeg ... -f hls  out/hls/     # MPEG-TS segments

ffmpeg ... -f dash out/dash/    # a second, fMP4 copy of identical video

Package once, point both manifests at the same segments:

Shaka Packager: one CMAF segment set, both manifests reference it

packager in=enc.mp4 --hls_master out.m3u8 --mpd_output out.mpd  # the duplicate just stopped existing

For new uploads, we just stopped creating the duplicate. The old files stayed, but the new ones came out single-copy.

The four checks I now run on any pipeline

Before the checklist, a caveat: none of this matters if your scale is small. Scale the fix to the size of the problem. Serve a few thousand videos and you should store both HLS and DASH and move on, because the duplicate storage is rounding-error money. But if your transcode and storage lines are big enough to see from orbit, read on.

Fixing those two didn't end it. It kicked off an audit of everything else, and four more checks turned up four more leaks.

First, check your retry granularity. Go read the job key, don't assume. We'd fixed the main path but were still retrying whole renditions in a couple of corners, each one quietly paying the film-re-encode tax on a smaller scale.

Second, check whether you're storing the same media twice. If HLS and DASH live in separate folders with separate segments, that's a duplicate you delete with a packaging change, not a re-encode. Cheapest money you'll ever find.

Third, check whether you're transcoding views that will never happen. Views follow a power law: a tiny head takes almost all the traffic, and a very long tail gets close to none. Eagerly transcoding 4K for a video three people will open is compute you burn so nobody can watch it. Transcode the tail on demand, on first request, and let it fall out of cache after. That was the biggest single win we didn't see coming.

Fourth, check what your CDN hit rate actually is. Segments are immutable, so they should cache nearly forever. Set them max-age one year, immutable, and you should sit north of 95%. A CDN under that number, unless you're serving nothing but long-tail cold assets, isn't doing its job, and ours was leaking exactly there.

What the fix cost, honestly

The wins were real. Killing the duplicate package halved segment storage, straight off the bill, no re-encode for anything new. Segment-level retries stopped nuking whole videos, so wasted compute dropped to the actual failure rate. A corrupt six-second segment now costs six seconds of re-encode, not two hours.

But the migration hurt. Migrating the back catalogue to CMAF was a full re-package of everything already stored. That's not a config flag, that's a grind, and it ate real engineering time over the better part of a quarter before it cleared. Just-in-time transcoding the long tail traded storage for latency, so the first viewer of a cold 4K title now waits while it encodes, and on a large asset that wait runs into double-digit seconds. We took complaints, mostly from people unlucky enough to be the first request for something obscure.

We softened it, not solved it. We pre-warm the popular head so the titles that actually get traffic are always hot, and we eat the cold-start on the tail because the storage saving is worth more than a rare wait. And someone owns the cache-hit dashboard now, permanently, because the day it silently drops, the origin bill is the thing that tells you, and by then it has already cost you.

View-count-at-scale is its own beast, a whole problem in itself, and deserves a separate piece. I'm skipping it here.

Where the interviewer takes this next

This comes up regularly in system design interviews. The question is always: fine for VOD, now make it live. Live collapses the comfortable gap between transcode and delivery. Segments shrink to a second or two, the DAG becomes a streaming pipeline you can't fully parallelise because segment N+1 doesn't exist yet, and you reach for low-latency HLS or chunked-transfer delivery to claw the lag back down. Same three nouns, ingest, transcode, deliver, far less slack.

Interviewers always ask three things: why not just serve MP4s, who picks the quality, and where do the retries queue.

Serve plain MP4 and you lose adaptive bitrate. One slow network and the viewer buffers instead of dropping to a lower rendition mid-stream. ABR needs segments and a manifest.

On quality, the client picks, not the server. The server doesn't know the viewer's network conditions, so the client samples throughput as segments download and requests the next segment from the manifest at a bitrate it can sustain without stalling.

And the failed-segment retries queue on the same job queue that feeds the farm, keyed at the segment level, which is the whole point of this article.

The savings are real, but they're not permanent

There's no clean ending here, and I'd distrust anyone who gave you one. 4K, and now whatever comes after it, is quietly eating the storage we clawed back. Fix one leak and another opens.

But the fixes are still in place, and the habit of auditing the job key and the manifest format is what actually lasts. The retry key still runs. The CMAF merge still holds. And every new leak gets spotted sooner, because now I know the four places to look before the bill finds them for me.

If this kind of war story, where the fix is one line but the insight takes a year, helps you think about system design, follow Beyond Localhost for more. These are the questions that show up in interviews, and the docs never give you the honest answers.

If you're preparing for a system design interview, or just trying to survive your next production outage, followPracHubfor more engineering deep dives.