- Deleted the `fetch` command from the CLI and associated handlers, streamlining the toolset. - Updated documentation to reflect the removal of the `fetch` command, emphasizing the use of `yt-dlp` directly for downloading videos. - Adjusted various components and types to remove references to `fetch`, ensuring consistency across the codebase. - Enhanced error handling to clarify the absence of the `fetch` functionality.
3.8 KiB
3.8 KiB
Watching footage
How to understand source material. Inspect only the modalities the question turns on — speech, action, music, graphics, or atmosphere may lead, so there is no fixed priority. Sample the picture against what the audio tells you.
- Always probe first.
media_probereports the container and its tracks, telling you up front whether the file has a video track, an audio track, or both. Everything after branches on that. - Get the lay of the land. Render a
media_waveform(audio) and amedia_filmstrip(video) for a fast, cheap overview of where the loud and quiet stretches fall, and where the visual scene changes are. A filmstrip shows coarse structure and scene state, not crop, framing, readability, or an exact cut frame. - Listen to the audio. Run
media_listenwith a prompt tailored to the context (what you actually need to know), and explicitly ask the model to include timestamps in its answer. See media-listen.md for prompt patterns. - Transcribe speech. For speech,
media_transcribewrites the full transcript with word-level start/end times to a JSON file and returns its path — search the file for the line you need and read its times from there, rather than loading it whole. - Sample the video against the audio. Use
media_grabto pull frames. When the audio has already pointed you at specific moments, feed those timestamps straight in from the transcript or listen output astimes(e.g.00:32,00:45). When you need a visual pass without such cues, reach forauto: it scans the footage and keeps only the frames where the picture settles into a new visual state, dropping near-duplicates.
The media_* tools take a path or a direct media URL. Footage behind a page URL (YouTube, TikTok, Instagram, …) is downloaded first with yt-dlp from a shell — audio only (yt-dlp -x --audio-format m4a -o talk.m4a <url>) unless the question needs frames; see the tool reference.
Matching depth to the question
Read only as much of the footage as the answer requires — each pass costs time, and media_listen costs credits.
- A duration or format question ends at
media_probe. - "Where is the quiet part" or "how is it paced" is usually answered by the waveform and filmstrip alone.
- A question about what was said resolves fastest through
media_transcribe; search the transcript file for the passage and quote it with its times. - Questions about non-speech audio (music, tone, sound events, speaker identity) are what
media_listenis for. - Only questions about what is seen need frames — and the audio pass usually tells you which moments to grab, so grab those instead of scanning blind.
- For open-ended questions ("summarize this", "what happens here"), combine passes: structure from waveform + filmstrip, content from transcript or listen, then frames at the salient moments to confirm what the picture shows.
Answering
- Ground every claim in something you actually saw or heard — a transcript line, a listen answer, a grabbed frame. If the evidence is ambiguous, say so rather than smoothing over it.
- Anchor answers to the timeline. Give timestamps as
MM:SS(orHH:MM:SSfor long footage) so the user can jump straight to the moment; for a scene or segment, give its start and end. - When asked to find a scene or moment, return the timestamp range plus a one-line description of what identifies it, so the user can confirm it is the right one.
- Summaries follow the footage's own structure: what happens, in order, with the timestamps where each part begins. Length matches what the user asked for, not what the footage contains.
- This skill only reads footage. When the user wants the footage changed — cut, composed, captioned, exported — that is the
editorskill's job.