2026-09-27 12:14:37 +02:00

4.3 KiB

Setup

Everything runs locally. The free path (your brand, a type-only hook, a scratch voice, a scratch music bed, the synthesised sound palette) needs no account. Paid generation runs on your own Higgsfield account.

What you need

Tool Why Required
Node.js 20 or later, npm the brand extractor and HyperFrames (the renderer) yes
ffmpeg every audio and video step yes
Python 3.10 or later and uv the audio pipeline (voice prep, sound, mix, review) yes
whisper.cpp (whisper-cli) and a model checking and timing the spoken words yes for real voice takes
torch and torchaudio forced alignment, the precise word timing recommended, pulled on demand by uv
Higgsfield CLI and account narrator voice, music, sound kits, generated footage optional
Blender 5.x 3D glass objects optional
ACE-Step 1.5 free local music takes optional
Dragonfly Plate Reverb (VST3) the shared reverb for sound effects; a built-in reverb is used without it optional

macOS

brew install node ffmpeg uv whisper-cpp
npm install                                   # installs the brand extractor's headless browser
uv venv .venv && source .venv/bin/activate
uv pip install -r requirements.txt
mkdir -p models
curl -L -o models/ggml-base.en.bin https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin
curl -L -o models/ggml-small.en.bin https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.en.bin   # better on product names

The scratch voice uses macOS say, which is already installed. HyperFrames downloads its own headless Chrome on the first render (npx -y hyperframes@0.8.77 ...).

Linux

sudo apt install ffmpeg espeak-ng build-essential cmake    # espeak-ng is the scratch voice
curl -LsSf https://astral.sh/uv/install.sh | sh
# Node 20+: from nodesource or nvm
git clone https://github.com/ggml-org/whisper.cpp ~/whisper.cpp
cmake -S ~/whisper.cpp -B ~/whisper.cpp/build && cmake --build ~/whisper.cpp/build -j
export PATH="$HOME/whisper.cpp/build/bin:$PATH"            # provides whisper-cli

Then the same npm install, venv and model steps as on macOS. Headless Chrome needs the usual system libraries (libnss3, libatk-bridge2.0-0, libgbm1, libasound2 and friends); if a render fails to launch the browser, install those.

Environment variables

Variable Default Used by
WHISPER_MODEL models/ggml-base.en.bin in the repo voice scripts, reference study
REVERB_VST3 ~/Library/Audio/Plug-Ins/VST3/DragonflyPlateReverb.vst3 sfx_forge.py
ACE_STEP_ROOT ~/ACE-Step-1.5 ace_score.py
BLENDER_GPU METAL (CUDA, OPTIX or NONE elsewhere) blender_glass_object.py
PYTHON python3 study_reference.sh

Optional tools

  • Forced alignment: nothing to install by hand. uv run -q --with torch --with torchaudio --with soundfile python skills/launch-voice/scripts/force_align.py LINE.wav "the words" pulls torch on first use and caches the aligner model (about 1.2 GB) in ~/.cache/torch.
  • Higgsfield: npm install -g @higgsfield/cli, then higgsfield auth login with your own account. Check your balance with higgsfield account status. For deeper Higgsfield know-how, add their skills: npx skills add higgsfield-ai/skills.
  • HyperFrames skills: npx skills add heygen-com/hyperframes gives your agent the full composition contract, motion catalog and CLI docs.
  • Blender: install 5.x from blender.org (macOS: brew install --cask blender).
  • ACE-Step 1.5: follow its README to install it into its own folder with its own venv, then set ACE_STEP_ROOT. Models download on first run (several GB).
  • Dragonfly Reverb: free and open source (GPL); install the VST3 plug-in from its website. Without it the sound palette uses pedalboard's built-in reverb.
  • Local voice clone: skills/launch-voice/scripts/clone_local.py pulls Chatterbox through uv on first run (a few GB).

Check the install

node -v && ffmpeg -version | head -1 && whisper-cli --help | head -1
source .venv/bin/activate && python3 -c "import numpy, scipy, soundfile, pedalboard, pyloudnorm, librosa; print('audio ok')"
python3 skills/launch-sound/scripts/sfx_forge.py palette /tmp/palette --key A    # 13 sounds, free