Engineering · the render pipeline
Six GPUs, four cities, one shot at a time.
The architecture behind the episodes.
We build long-form animated documentary on our own hardware. This is how a written line becomes a finished episode — the stages it passes through, the file each one leaves behind, and the reason a render farm needs a scheduler that understands what a video model actually costs. Everything below is measured on the cards we own, and the live farm page shows it running right now.
The spine
Six stages, strictly ordered. The unusual one is first: the voice is synthesised before any picture exists, because the length of the spoken line decides how many frames the clip must be. Get that backwards and every clip is either padded or truncated.
Holding a character still across 600 shots
A video model cannot remember a face. Ask it for the same person in two shots and you get two people. So identity never lives in the video prompt — it lives in the still, and the still's identity is a fixed integer.
One block, one seed, full denoise
Every character carries a verbatim appearance description and a seed that is never re-rolled. The still is drawn from that pair; the clip is then seeded from the still. Change the seed and you have changed who is on screen.
Re-takes move the camera, not the face
A bad take re-seeds the motion and the scenery — deliberately leaving the character seed alone. A defect in a face is fixed by rewriting its description, never by rolling again.
The registry is tested, not trusted
Two characters sharing one seed would render as one person under two names, in every shot either appears in, with no file missing and nothing raised. A test asserts uniqueness across every cast list in the repository, including ones not yet written.
Why a video farm needs a real scheduler
Clips are not interchangeable units of work. A shot's length is set by its line, so the farm holds jobs ranging from 73 to 337 frames — and at 1920×1088 the decode of a long one no longer fits a 32 GB card. What happens then depends on the operating system, not the silicon.
The lesson generalises. A scheduler that routes by hardware specification will keep handing work to a machine that cannot do it, because a card's memory figure does not tell you what its operating system will do when that memory runs out. Capability has to be measured per host and then enforced, and the failures have to be counted — a node that fails silently looks exactly like a node that is busy.
The fleet
Six GPUs across four machines in three cities, plus a separate box that serves the vision model used for quality control. Every node reports its own temperature, power draw and loaded models to the live page.
| Where | Card | Memory | Runs |
|---|---|---|---|
| Dubai | RTX 5090 × 2 | 32 GB each | Stills, clips, long clips |
| Bangalore | RTX PRO 6000 | 96 GB | Stills, clips, long clips |
| Bangalore | RTX A5000 | 24 GB | Stills only — too small for HD video |
| Bhilai | RTX 5090 × 2 | 32 GB each | Stills and clips under 265 frames |
| Bhilai | RTX A5000 × 2 | 24 GB each | Serves the vision model that grades the output |
Work is pulled, not pushed: each node takes the next job it is capable of, so a slow card simply takes fewer jobs instead of holding up the batch. Near the end of a run a node that is measurably slower than the fastest stops taking new work, so the tail is not spent waiting on one machine.
It has to survive the night
A render is measured in days, so the interesting failures are not bad frames — they are a stage that stopped while nobody was watching. Two things fail: the machine, and the process on it. They need different answers.
A reboot brings the farm back
Every host starts its own render service at boot, without anyone logging in. A power cut that once cost most of a working day now costs the time it takes the machine to come up.
A crash is caught in a minute
A watchdog on each host checks whether the port is actually answering — not whether a process exists, which is a question that lies. If the service has died it is restarted; if it is healthy the watchdog does nothing at all.
Nothing is ever killed to fix it
No recovery step stops a running render. A watchdog that restarts a healthy job is worse than no watchdog, so every guard is written to stand down rather than intervene when it is unsure.
Watch it work
The farm page is not a diagram — it is the live state of these six cards, refreshed every fifteen seconds: temperature, watts against the power limit, the models each card currently holds in memory, the exact shot it is rendering, and how far through the season the build has got.
If you want this pipeline pointed at your own footage, the commercial write-up is on the render pipeline page, and video & image covers buying render time by the hour.