AudioReactiveVisuals

Audio-reactive video generator. Input: audio → output: video where the camera flies forward forever and punches in on every kick, with the punch strength driven by low-frequency (<150 Hz) energy. Snares whisper, kicks punch.

Reference: reference_video.mp4 (style/behavior only — never overwrite it).

How it works

audio (librosa) → per-beat kick/snare strengths → beat canvases (Z-Image) → frames → ffmpeg

Beat chain (default, --beat-mode chain). One canvas per beat (at 200 BPM that's a fresh image every 0.3 s). Each canvas is img2img'd from the previous beat's zoomed end view (rebase), so the flight zoom never steps back. Inside a beat the zoom is linear plus a 2-frame kick step:

z_beat = min(--zmax, --drift + --kick-gain * kick^1.5 + --snare-gain * snare)

• kick = <150 Hz spectral-flux envelope (max near the beat), snare = 150–500 Hz.
• Seams hidden with a 2-frame dissolve (--beat-blend), per-beat palette locked with a LAB color match, opening blooms from white (--intro-fade).
• Beat 0 comes from --init-image (a proven canvas) or auto-picked from --seed-tries txt2img candidates.
• Framing is texture-seeking: each beat crops the most detailed window, so the flight never gets stranded in flat sky (plus: put texture in every prompt's sky).
• Grade tames neon extremes + highlight rolloff + S-curve (skip: --no-grade).

Legacy --beat-mode cycle (crossfading keyframes) is kept for --no-diffusion pipeline tests only — its blend resets cause visible zoom-back.

Project layout

generate.py            # thin entry shim (all logic in the package below)
audioreactive/         # the actual codebase
  config.py            # models, prompts constants, ALL tuning numbers
  audio.py             # tempo / onsets / kick-snare bands / beat grid
  camera.py            # rebase crops, texture-seeking + clamped pans
  color.py             # LAB match, tame_blue, anchor_pull, grade, upscale
  diffusion.py         # model adapters (Z-Image, Qwen-Image-2.1) + beat chain (GPU only)
  render.py            # procedural mocks, beat-chain + legacy cycle renderers
  cli.py               # argparse, chain/cycle orchestration, run metadata
jobs/                  # SLURM jobs (submit from repo root)
  common.sh            # env, prompt pack A, shared chain/full-model flags
  sample.sh            # Turbo check on the 10 s sample
  smoke.sh             # final-render settings on the 10 s sample
  iter.sh              # full song at 640x368 for drift checks (srun)
  newbeginning.sh      # full-song HD render
  qwen_sample.sh       # Qwen-Image-2.1 chain on the 10 s sample
  ztest.sh             # model timing kick test (scripts/ztest.py)
scripts/               # dev tools: ztest.py (model kick test), check_drift.py
tests/                 # CPU-only pytest suite (camera/color/cli/audio)

python -m audioreactive works too (same flags). Tuning numbers live in audioreactive/config.py — change them there, never as magic literals.

Setup

Conda env nibbler (/home/d.ramos/miniconda/envs/nibbler: torch+CUDA, diffusers 0.40, librosa, opencv). On this cluster, always:

export LD_LIBRARY_PATH=/home/d.ramos/miniconda/envs/nibbler/lib:$LD_LIBRARY_PATH

(without it, the ohpc libstdc++ breaks scipy/diffusers imports). There is no system ffmpeg — generate.py falls back to the imageio-ffmpeg binary. Model Tongyi-MAI/Z-Image-Turbo is cached in ~/.cache/huggingface/hub (reused offline via HF_HUB_OFFLINE=1); the login node has no GPU, so diffusion runs on the SLURM gpu partition (see jobs/).

Usage

Fast pipeline check, no GPU (procedural canvases, exercises the full audio → zoom → mux path):

export LD_LIBRARY_PATH=/home/d.ramos/miniconda/envs/nibbler/lib:$LD_LIBRARY_PATH
/home/d.ramos/miniconda/envs/nibbler/bin/python generate.py \
  --audio audio_sample.wav --output output/test.mp4 \
  --no-diffusion --beat-mode chain --tempo 200 --prompt "abstract thing"

Procedural renders are deterministic: when refactoring, compare md5s of the chain and --beat-mode cycle outputs before/after.

CPU-only test suite (no GPU, runs on the login node):

/home/d.ramos/miniconda/envs/nibbler/bin/python -m pytest tests/ -q

Diffusion renders run on the GPU node; submit from the repo root:

sbatch jobs/sample.sh        # 10 s sample, 34 beats, Turbo
sbatch jobs/smoke.sh         # 10 s sample, full model at 960x544
sbatch jobs/newbeginning.sh  # NewBeginning.wav, 188 s ≈ 628 beats, 1280x720

(or bash jobs/newbeginning.sh directly on a GPU machine with CUDA_VISIBLE_DEVICES set — the login node has no GPU). All jobs seed beat 0 from output/chain_init.png (--init-image).

Key flags: --tempo (else estimated — librosa often halves 200→99, pass it manually), --width/--height (HD = 1280×720), --model-id, --steps and --guidance-scale (default per model, see Models), --gen-w/--gen-h (default per model: 640×368 Z-Image sharp domain, 960×544 full-model HD, 1024×576 Qwen; snapped to the model's multiple), --canvas-scale (1.2 output headroom), --beat-strength (0.4 chain continuity), --beats-per-prompt (4), --kick-gain (0.30), --snare-gain (0.05), --drift (0.03), --zmax (0.30), --kick-step (0.05, 2-frame attack), --beat-blend (2-frame seam dissolve), --intro-fade (0.35 s bloom opening), --init-image (proven beat-0 canvas), --seed-tries (auto-pick contentful txt2img), --max-duration (trim for tests).

Outputs land in output/ (gitignored): video, .json (model rev, seed, beats, kick strengths, zoom params), keys/ canvases, slurm logs.

Prompt packs

Every prompt shares a STYLE prefix; each one must describe sky texture (clouds/birds/branches) — flat unique-color skies read as dead air on quiet beats. Prompts cycle (--beats-per-prompt beats each). The negative prompt must keep the anti-blue tail (config.ANTI_BLUE_NEG, also in the CLI default).

Pack A — Greek temples (used for newbeginning_hd.mp4)

Lives in jobs/common.sh (STYLE, NEG, 12 SCENES) — the single copy all jobs share.

Pack B — abstract shapes (untested — start with --test / short --max-duration)

STYLE = abstract digital painting, flowing three-dimensional shapes, intricate surface detail everywhere, no flat backgrounds, no text, no watermark (negative: blurry, low detail, watermark, text, flat color, empty background, gradient background)

1. molten gold and teal silk ribbons folding in dark space, glowing filaments, drifting sparks
2. crystalline cavern of refractive prisms, light shafts, floating dust motes
3. close flight through giant translucent membranes with bioluminescent veins, drifting motes
4. slow cosmic lava blobs of magenta and cyan, grainy nebula texture, tiny stars
5. endless corridor of glowing arches over a reflective floor, luminous haze
6. macro shot of iridescent oil film, swirling rainbow slick, tiny bubbles
7. layered frozen glass waves, refractions and caustics, suspended droplets
8. vortex tunnel of glowing embers and ice crystals spiraling inward

Validating motion

Measure per-frame zoom % as optical-flow (Farneback) divergence. The meter used during development lived in /tmp/opencode/measure_zoom.py and was never committed. Reference: median ≈ +1.9 %/frame, spikes to +9 %, never negative. Procedural mocks should read median ≳ 1.2 with 0 % zoom-back within beats; on rich diffusion content the meter under-reads ~2× (dark/pastel areas go quiet), so judge cruising speed on mocks and aesthetics on contact sheets.

Models

Per-model defaults (steps, guidance, gen size, dimension multiple) live in config.MODEL_PRESETS; diffusion.load_image_model picks the adapter.

• Default: Tongyi-MAI/Z-Image-Turbo (8 steps, no CFG) — fast checks.
• Full Tongyi-MAI/Z-Image (30 steps + guidance_scale 4): use --model-id Tongyi-MAI/Z-Image --steps 30 --guidance-scale 4.0.
• Qwen/Qwen-Image-2.1 (7B, Qwen Research License, untested here): use --model-id Qwen/Qwen-Image-2.1 (sbatch jobs/qwen_sample.sh). Needs diffusers>=0.41 + transformers>=5.17; not cached yet (first run downloads it). Differences from Z-Image:
  • No img2img strength. The previous beat's end view is passed as a condition image and each beat is denoised from scratch, so --beat-strength is ignored; continuity comes from the reference. If beats wander, tune config.QWEN21_CHAIN_PROMPT (wraps each chained prompt).
  • CFG off by default (the model is meant to run without it), which makes the negative prompt — and its anti-blue tail — inactive. Pass --guidance-scale 4 to enable it (~2× cost).
  • Dims must be multiples of 32; defaults to 40 steps at 1024×576.
• **Kick test verdict (H200 NVL, 640×368, 30 steps): txt2img 3.1 s, img2img ~1.4 s → a 188 s / 200 BPM song (≈628 beats) renders in ~15 min — far under 2 h. Quality is a clear step up (painterly detail, natural skies, prompt adherence). Tested via scripts/ztest.py / jobs/ztest.sh (output/ztest_*.png, verdict UNDER_2H in the job log). Recommended for final renders; generate at 960×544 for HD output.
• **Head-to-head at HD gen res (960×544, same prompt/seed, foreground srun, output/cmp_*.png): Turbo (8 steps) txt2img 1.9 s / img2img 0.8 s → song ≈ 8 min, BUT on this seed it drew an abstract yellow-red gradient (weak prompt adherence); full Z-Image (30 steps + guidance 4) txt2img 4.8 s / img2img 2.0 s → song ≈ 21 min (UNDER_2H) with detailed temples, storm clouds, eagles. Outputs differ hugely (mean abs diff 94.8 → genuinely different weights, not a relabel). Full model wins for finals; Turbo stays for fast checks. Proof that --model-id selects the weights: runs log model= + rev= ([chain] model=... line) and the .json records both — Turbo rev 013496ad, full rev 04cc4abb.