Benchmarks
What has been measured so far, how, and what the numbers do not show.
Every figure on this page comes from a recorded run. Each section says what was given to the model, what counted as a pass and what it cost. All runs used one fast, low-cost model, the kind you would use for a director that answers in a few seconds. Where a result has a cost, the cost is next to it.
Read this first. A pass means the result compiled with no problems on the first reply. It does not mean the film looks good. No designer has reviewed the output. Each case was run once or twice, so these are counts, not rates.
Summary#
| What was asked | Given | Result | Cost |
|---|---|---|---|
| Write a product film from a brief | The kit files only, about 7,600 tokens | 2 of 2 compiled first try, no notes | 319 and 725 tokens out, 4.4 s and 5.4 s |
| Change an existing film | The film and the edit task file, 640 to 924 tokens | 6 of 6 applied and compiled first try | Operations of 73 to 387 characters against films of 906 and 1,724 |
| Edit a talking video from its transcript | A skill, the stage, the transcript | 15 of 15 first edits and 12 of 12 follow-ups first try | Median 1,604 tokens in, 215 out, 2.9 s |
| Keep pieces from covering each other | The same 15 edits, measured on the built scene | 11.6 s of overlap as written, 0.8 s after the arranger | 17 of 112 pieces left out, 6 ended early |
| Dress every film for every brand | 6 films, 5 design systems, 3 products | 90 of 90 combinations compile | None: no model involved |
Films written from the kit alone#
The model received docs/index.md, docs/tasks/new-film.md and the reference files for film and product pieces. It did not see this site, the source or an example of the film it was asked for. The brief was two or three sentences describing a film shape taken from studying launch films.
| Film | Tokens in | Tokens out | Time | Length | Elements | Film size |
|---|---|---|---|---|---|---|
| Requests in, work out | 7,606 | 725 | 5.4 s | 26.2 s | 137 | 1,172 characters |
| Claim, then proof | 7,626 | 319 | 4.4 s | 26.0 s | 176 | 919 characters |
Both films are in the playground exactly as written, marked as written by a model. Both contain no colour value, no typeface and no product copy, so both can be dressed in any of the five systems for any of the three products.
An earlier version of the task file produced a film pinned to the top of the frame, because the model wrapped it in a full-height box. One line was added to the task file and the run repeated clean. That is one iteration on the instructions, and it is why the count here is two runs and not a rate.
Edits as operations#
The model received a film and docs/tasks/edit.md. It replied with a list of operations. The list was applied with applyPatch and the result compiled.
| Request | What it sent | Operations | Film |
|---|---|---|---|
| Show the Director feature before the Timeline feature. | move scenes/4 to scenes/3 | 85 characters | 1,724 |
| Make the command palette land on the third result instead of the second. | replace scenes/1/node/pick | 73 | 1,724 |
| Remove the bento scene and hold the closing name one second longer. | remove, replace len, replace duration | 169 | 1,724 |
| Open with the number instead of the claim; the claim comes second. | move scenes/2 to scenes/0 | 85 | 906 |
| Use a push between every pair of scenes. | four replace, one add | 387 | 906 |
| After the wheel, add a four second scene with a checklist of the product's checks. | add scenes/2, replace duration | 178 | 906 |
Six of six applied on the first reply. In every case the scenes the request did not mention are byte-for-byte the same after the edit, because the format gives the model no way to touch them. Time per edit ranged from 2.4 s to 15.8 s; the slowest replies spent their tokens reasoning before a one-line answer.
A director on real recordings#
Three real talking videos, with their word timings, were each given to the model with five different skills. A skill is a config, a stage, a short list of allowed pieces and a few rules. The model wrote the edit in a compact line format (id = Piece(args) @ when) which the kit parses, arranges and builds into a scene. Each edit then received a follow-up such as "make the hook bigger" or "remove the chart".
| Measure | Result |
|---|---|
| First edits that built with no problems | 15 of 15 |
| Follow-ups that built with no problems | 12 of 12 |
| Median prompt | 1,604 tokens |
| Median reply | 215 tokens |
| Median time | 2.9 s |
The recordings are private and are not on this site. Captions were not part of what the model wrote: the stage publishes a caption lane and a host draws its own captions in it.
Overlap, measured on the built scene#
The audit mounts a scene in a browser, steps through it and measures every pair of pieces whose boxes intersect, and every piece that leaves the frame. It measures. It does not fix. The arranger is the only part of the kit that enforces layout rules, and it can be turned off with arrange: false.
| As the model wrote it | After the arranger | |
|---|---|---|
| Seconds in which two pieces overlap, across 15 edits | 11.6 s | 0.8 s |
| Pieces left out | 0 of 112 | 17 of 112 |
| Pieces ended early | 0 | 6 |
The arranger buys the drop in overlap by removing or shortening pieces: 15% of what the model asked for does not appear. Each decision is returned with its reason, so a host can show it or send it back to the model. Whether that trade is right depends on the video, which is why enforcement is separate from the library.
What these numbers do not show#
- Quality. Nothing here scores how a film looks or whether its timing feels right.
- Variance. One or two runs per case. A model that passes twice may fail the third time.
- Other models. An earlier run with the full guide in the prompt passed 6 of 6 scenes on three different models. The kit-files runs above used one.
- Long films. The longest film written was 26 seconds.
- Hard edits. The six edit requests each name what to change. Vague requests ("make it punchier") were only tested on the talking videos.
Run them yourself#
The benchmark script and its recorded results are in bench/. It needs a key for a model gateway in the environment and writes its results next to the script.
npx tsx bench/films.ts # write the films, then run the six edits
npx tsx cli.ts check film.json # the same check the benchmark uses