- Enhanced the `present` function to support background track processing in the `media_segment` tool, allowing for real-time tracking and reporting of mask states. - Introduced a new `presentContext` function to manage the output of background tracks, including their progress and contact sheets. - Updated the `context` tool to report the status of ongoing background tracks, including their state and associated images. - Modified the `media_segment` tool to return tracking states and paths, improving integration with the `<mask>` component in the editor. - Enhanced documentation to clarify the new behavior and usage of background tracks in the context of media segmentation.
7.5 KiB
7.5 KiB
Footage analysis
How to understand source material before editing it. Inspect only the modalities the decision turns on — speech, action, music, graphics, or atmosphere may lead, so there is no fixed priority. Sample the picture against what the audio tells you.
- Always probe first.
media_probereports the container and its tracks, telling you up front whether the file has a video track, an audio track, or both. Everything after branches on that. - Get the lay of the land. Render a
media_waveform(audio) and amedia_filmstrip(video) for a fast, cheap overview of where the loud and quiet stretches fall, and where the visual scene changes are. A filmstrip shows coarse structure and scene state, not crop, framing, readability, or an exact cut frame. - Listen to the audio. Run
media_listenwith a prompt tailored to the context (what you actually need to know), and explicitly ask the model to include timestamps in its answer. See media-listen.md for prompt patterns. - Transcribe speech. For speech,
media_transcribewrites the full transcript with word-level start/end times to a JSON file and returns its path — search the file for the line you need and read its times from there, rather than loading it whole. - Sample the video against the audio. Use
media_grabto pull frames. When the audio has already pointed you at specific moments, feed those timestamps straight in from the transcript or listen output astimes(e.g.00:32,00:45). When you need a visual pass without such cues, reach forauto: it scans the footage and keeps only the frames where the picture settles into a new visual state, dropping near-duplicates.
The editing loop
- Write the brief first. For anything nontrivial, capture the edit as a markdown file: it is the plan every save works toward and the thing to check the result against.
- Lay down the A-roll. Assemble the primary footage as JSX and save. Get the spine of the edit right before anything else.
- Layer the rest on top. Once the A-roll holds, add B-roll and secondary assets (sound effects, captions, overlays) in the same source.
- Symlink media the project uses into its
assets/folder and name it by library path (assets/b-roll/drone.mp4is"b-roll/drone.mp4"); local and remote paths work too but stay outside the library. contextreports which folder the app actually has open, where the playhead sits, and where everygenerate.*declaration stands — poll it to wait for generations without blocking.
The source is the document. A project is a folder of JSX; open a folder once, then write the files and save. Saving recompiles and re-renders the canvas.
Compositing
- Chrome, scaffolding, and ornament all draw from a visual budget whose default balance is
0; prefer not to use them. A cut, hold, or change of size can separate two ideas as clearly as a divider without adding visual clutter. An element earns its place by deepening the story, guiding attention, or expanding imagination, never by filling space. - Video is its own medium, with its own rules; it is not a website, poster, slide, or UI. It is watched, not read.
- Don't darken, blur, or cover the picture to make something on top of it legible
- Let visuals, sound, and voice carry context; let text punctuate rather than explain. Do not add copy, eyebrows, labels, underlines, or brand color highlights unless the brief or explicit video guidance calls for them; examples alone are not instructions.
- Choose easing from the intended weight, energy, and continuity of the action.
- When the brief, project, or user specifies branding, follow it. Only when none is specified, fall back to the house style — its design, voice, video, and library references, and the components and compositions bundled with them.
Verification
How to confirm a change actually produced what you intended. A clean save does not guarantee a correct-looking composition.
- Run
checkon the scene first. It catches black frames, clips that never become visible, zero-duration or fully transparent nodes, and assets that failed to load or generate. - Use
captureon the scene to see what the viewer actually gets. - Reconcile captured frames with the brief, and the brief with these guidelines.
- Verify after every stage, not only at the end — build the composition incrementally so a problem is caught next to the change that caused it.
- Scale verification to the change. A small or incremental tweak the user asked for needs no visual confirmation so the user gets the result back fast and can keep iterating.
- Fix the largest viewer-facing problem before polishing details, and recheck related moments after structural changes, since pacing, continuity, emphasis, and meaning are relational.
- Use
screenshotorlogsto debug issues. - DO NOT export/render the scene for visual confirmation —
captureis equivalent to a render but far more efficient. Rendering to a video should be a user-triggered action unless explicitly requested in the prompt.
Best practices
- Wrap clips in
<sequence>tags wherever the parent tag supports it — A-roll, B-roll, and other clips belong in sequences so the timeline stays structured rather than a flat, messy pile. A sequence does not place its children: give every clip an explicitstart. - Use the built-in tags for the media a composition is made of (audio, video, images, captions).
- Hoist the properties that define the composition's look — title copy, font family and size, accent colors, key padding — into top-level consts annotated with
@inspect, so they become live controls in the app's inspector. - For anything 3D, use Three.js drawn into a
<surface>tag. - To cut an object out of footage, or confine an effect to it, track it with
media_segment(preview the prompt first) and name the mask file in a<mask>under the clip's<effect>. A track into the library runs in the background: keep working, and pollcontextuntil its row inmasksis done before naming it. - For motion graphics, overlays and UI-heavy graphics, the
<html>tag driven by a paused anime.js timeline - Before animating anything, read the easings reference and choose easings deliberately — default or linear easing is what makes motion read as a slideshow.
- Add auto captions last, after everything else is assembled, so they transcribe the finished audio at its final placement.
- For tasks that don't need an editing UI, keep the app in the background: from a shell,
diffusion open -b <dir>launches it that way. Over MCP the app is already running, andopenonly opens the folder. - Only render (export) the result when prompted.
- Start with a fresh project.
Docs
The authoring reference is served by the app for the installed version: read it there and trust it over memory.
- JSX reference — elements, timing, paints, generation, captions
- Tool reference — every tool, its fields, its CLI spelling and its output
- Examples — complete compositions, basics through shaders
- Easings: which cubic-bezier to use and when
- House style — the fallback when no other branding is specified
Examples
Read worked example(s) that match your context.