Shipping video object swap: the plan was never real, and the fix is better than the plan

We shipped video object swap this week: name an object in a clip, describe what should replace it, get back the same footage with that one thing changed. The architecture we ended up with is not the one we designed. Here is what actually happened.

12-second demo: source clip -> tracked cutout -> generated replacement -> locked composite

The plan that was never real

The workflow (workflows/executor.py, workflows/compositor.py) was written against two self-hosted models: sam-3-track to produce a binary tracking mask, and wan-2.2-inpaint to repaint the masked region under a prompt. Both rows were seeded in a migration back in August. Neither was ever activated — no GPU runner was ever registered for them. Every call this pipeline made failed at the gateway with "unknown model." The pipeline had never worked, not once, since the day it was written.

We don't rent GPUs for this. So the fix wasn't "turn on the fleet" — it was finding hosted vendors that could do the same job, cheaply, through APIs we already route.

Stage A: tracking, the easy half

oxen/sam-3-video (Oxen.ai, $0.02 a call) tracks a named object through a clip. We expected a binary mask back — white where the object is, black everywhere else, the shape sam-3-track was designed to produce. What it actually returns, with apply_mask: true, is a cutout: the object's own pixels, full color, full detail, composited onto solid black everywhere else.

That's strictly more information than a mask, not less. The object's silhouette is exactly "any pixel that isn't the cutout's own black," so a binary mask falls out of it for free — no model, no inference, just a threshold. We added compositor.cutout_to_mask to do that, and it's simpler than what the mask-based design needed in the first place.

Three real integration bugs showed up only once we ran it through our own gateway instead of hitting the vendor directly:

• The catalogue row's wire config pointed at the wrong submit path. It had never been dispatched end-to-end before, so nothing had ever caught it.
• It was wired for the vendor's synchronous endpoint, which doesn't fit our async submit-then-poll dispatcher. Switched to their queue endpoint, which returns the same job-id shape every other vendor on this gateway already uses.
• apply_mask wasn't being forced onto the wire, so every call was silently returning the full, unmasked frame instead of the cutout we needed.

And one thing that isn't a bug: the model is genuinely non-deterministic. The same clip, same prompt, same call resolved the target once and came back with an empty, all-black cutout on three repeat attempts right after. We added a coverage check — if the cutout is suspiciously close to all background, retry the whole call, up to twice, before giving up. A $0.02 retry is cheap insurance against paying for stage B on a target that was never found.

Stage B: there's no hosted mask-conditioned inpainter

We checked every active video row across every vendor we route to. None of them do mask-conditioned video inpainting — "repaint only this region, leave the rest alone" — at any price. That capability only exists in open-weight form, and we don't run GPUs.

What does exist, and is cheap: reference-to-video models that take a source clip as a motion/scene reference, an optional image as the new identity, and a prompt, then generate a whole new clip. We picked wan-3.0-r2v (Alibaba Wan 3.0, via AtlasCloud, $0.05/s) because it's the only one in the catalogue that takes the source as an actual video reference rather than just an image — it gets the real motion and background to imitate, not just a style cue.

We tested it on a clip of a flower and asked for "the flower, replaced by a red rose, same motion and camera framing." The regenerated clip reproduced the original framing, background, and camera motion almost exactly, with only the named subject changed. That's a believable claim from a vendor's marketing copy and a different thing to watch it actually hold on a real request.

Why the compositor doesn't care

This is the part of the design that turned out to matter most. compositor.py's job has never been "trust the model" — it's the opposite. It takes the inpainter's (or in this case, the generator's) pixels only inside the tracked region, and the original footage's own pixels, bit-for-bit, everywhere else. It proves that invariant on every frame before letting the output out, and raises a hard error if it's ever violated.

That discipline is why swapping stage B's entire approach — from "repaint one region" to "regenerate the whole frame" — didn't require touching the compositor's core logic at all. A generator that redraws everything is a *worse* input to trust blindly, which is exactly why the background lock exists in the first place. It was built assuming the model outside the mask can't be trusted; it turns out that's true whether the model is editing one region or inventing the whole frame.

What's not solved

oxen/sam-3-video only resolves a target by text prompt. We tried a points field for click-based targeting and got nothing — the model wants a name, not a coordinate. If you only have a point to describe an object, this pipeline isn't there yet. It fails clearly rather than pretending to support it.

The replacement stage also regenerates the whole frame rather than truly "inpainting" one region, so it inherits every limitation of a generative reference-to-video model: it can drift from the exact framing on a longer or more complex clip in ways a true masked repaint wouldn't. The background lock catches any case where the drift would have shown up as a changed background pixel — it can't fix a replacement that's simply a worse creative match for the scene.

Both routes are live now, priced at $0.02/call and $0.05/s, dispatched through the standard gateway path like everything else we serve.