Reelify Motion Kit 0.1 preview
Docs/Project

Benchmarks

What has been measured so far, how, and what the numbers do not show.

Every figure on this page comes from a recorded run. Each section says what was given to the model, what counted as a pass and what it cost. All runs used one fast, low-cost model, the kind you would use for a director that answers in a few seconds. Where a result has a cost, the cost is next to it.

Read this first. A pass means the result compiled with no problems on the first reply. It does not mean the film looks good. No designer has reviewed the output. Each case was run once or twice, so these are counts, not rates.

Summary#

What was askedGivenResultCost
Write a product film from a briefThe kit files only, about 7,600 tokens2 of 2 compiled first try, no notes319 and 725 tokens out, 4.4 s and 5.4 s
Change an existing filmThe film and the edit task file, 640 to 924 tokens6 of 6 applied and compiled first tryOperations of 73 to 387 characters against films of 906 and 1,724
Edit a talking video from its transcriptA skill, the stage, the transcript15 of 15 first edits and 12 of 12 follow-ups first tryMedian 1,604 tokens in, 215 out, 2.9 s
Keep pieces from covering each otherThe same 15 edits, measured on the built scene11.6 s of overlap as written, 0.8 s after the arranger17 of 112 pieces left out, 6 ended early
Dress every film for every brand6 films, 5 design systems, 3 products90 of 90 combinations compileNone: no model involved

Films written from the kit alone#

The model received docs/index.md, docs/tasks/new-film.md and the reference files for film and product pieces. It did not see this site, the source or an example of the film it was asked for. The brief was two or three sentences describing a film shape taken from studying launch films.

FilmTokens inTokens outTimeLengthElementsFilm size
Requests in, work out7,6067255.4 s26.2 s1371,172 characters
Claim, then proof7,6263194.4 s26.0 s176919 characters

Both films are in the playground exactly as written, marked as written by a model. Both contain no colour value, no typeface and no product copy, so both can be dressed in any of the five systems for any of the three products.

An earlier version of the task file produced a film pinned to the top of the frame, because the model wrapped it in a full-height box. One line was added to the task file and the run repeated clean. That is one iteration on the instructions, and it is why the count here is two runs and not a rate.

Edits as operations#

The model received a film and docs/tasks/edit.md. It replied with a list of operations. The list was applied with applyPatch and the result compiled.

RequestWhat it sentOperationsFilm
Show the Director feature before the Timeline feature.move scenes/4 to scenes/385 characters1,724
Make the command palette land on the third result instead of the second.replace scenes/1/node/pick731,724
Remove the bento scene and hold the closing name one second longer.remove, replace len, replace duration1691,724
Open with the number instead of the claim; the claim comes second.move scenes/2 to scenes/085906
Use a push between every pair of scenes.four replace, one add387906
After the wheel, add a four second scene with a checklist of the product's checks.add scenes/2, replace duration178906

Six of six applied on the first reply. In every case the scenes the request did not mention are byte-for-byte the same after the edit, because the format gives the model no way to touch them. Time per edit ranged from 2.4 s to 15.8 s; the slowest replies spent their tokens reasoning before a one-line answer.

A director on real recordings#

Three real talking videos, with their word timings, were each given to the model with five different skills. A skill is a config, a stage, a short list of allowed pieces and a few rules. The model wrote the edit in a compact line format (id = Piece(args) @ when) which the kit parses, arranges and builds into a scene. Each edit then received a follow-up such as "make the hook bigger" or "remove the chart".

MeasureResult
First edits that built with no problems15 of 15
Follow-ups that built with no problems12 of 12
Median prompt1,604 tokens
Median reply215 tokens
Median time2.9 s

The recordings are private and are not on this site. Captions were not part of what the model wrote: the stage publishes a caption lane and a host draws its own captions in it.

Overlap, measured on the built scene#

The audit mounts a scene in a browser, steps through it and measures every pair of pieces whose boxes intersect, and every piece that leaves the frame. It measures. It does not fix. The arranger is the only part of the kit that enforces layout rules, and it can be turned off with arrange: false.

As the model wrote itAfter the arranger
Seconds in which two pieces overlap, across 15 edits11.6 s0.8 s
Pieces left out0 of 11217 of 112
Pieces ended early06

The arranger buys the drop in overlap by removing or shortening pieces: 15% of what the model asked for does not appear. Each decision is returned with its reason, so a host can show it or send it back to the model. Whether that trade is right depends on the video, which is why enforcement is separate from the library.

What these numbers do not show#

  • Quality. Nothing here scores how a film looks or whether its timing feels right.
  • Variance. One or two runs per case. A model that passes twice may fail the third time.
  • Other models. An earlier run with the full guide in the prompt passed 6 of 6 scenes on three different models. The kit-files runs above used one.
  • Long films. The longest film written was 26 seconds.
  • Hard edits. The six edit requests each name what to change. Vague requests ("make it punchier") were only tested on the talking videos.

Run them yourself#

The benchmark script and its recorded results are in bench/. It needs a key for a model gateway in the environment and writes its results next to the script.

text
npx tsx bench/films.ts          # write the films, then run the six edits
npx tsx cli.ts check film.json  # the same check the benchmark uses
Reelify Motion Kit 0.1 preview. Built 2026-11-21.llms.txt · SKILL.md · 109 pages