# Sound and music Everything is synthesized with WebAudio: no sample files. A sound is plain data (a `Patch`), a theme is plain data (a `Song`). The runtime owns the audio context and the mixer, so the players' volume settings keep working. Match the game's mood (toy-like and bouncy for a party game, low and sparse for a scary one); someone with their eyes shut should still be able to follow the game. ## Contents 1. What Flow already plays 2. `sounds.ts` and wiring 3. The Patch format 4. Recipes 5. The Song format and the theme 6. Checking your work: `sounds.test.ts` --- ## 1. What Flow already plays Don't duplicate these: the theme (soft under the title card, ducked for 3-2-1, full at GO), the countdown ticks and GO, the **finale** (a "hurry" sting, then the theme 12% faster for the last 10 s, and ticks on the last 5), the FINISH hit or TIME UP whistle, the results fanfare or sad trombone, and a whoosh on every `flow.hud.banner`. Use `flow.musicVolume(0, 0.5)` for a tense silence and `flow.intensify()` to kick into the fast version early (sudden death, a frenzy). ## 2. `sounds.ts` and wiring ```ts // sounds.ts: plain data, from the headless core, so bun can load it in tests import { INSTRUMENTS, SFX, type Patch, type Song, type SoundDef } from '@voxelparty/sdk/core'; export const SOUNDS = { catch: { … } as Patch, rattle: (rnd) => ({ … }), // a factory: a fresh random patch on every play land: SFX.land, // reusing a library sound is fine; a tuned variant is better } satisfies Record; export const MUSIC: Song = { … }; // pass it to defineGame({ music: MUSIC }) ``` - Aim for **8–16 sounds** covering every meaningful event, named for what happens (`sheepJump`, `countWrong`), not how they sound. - `sounds.ts` imports only from `@voxelparty/sdk/core` and your own sound files: no game code, no `@voxelparty/sdk` main entry (it throws under bun, and `sounds.test.ts` imports this file). The data and types are all there: `INSTRUMENTS`, `SFX`, `SONGS`, `Patch`, `Layer`, `Song`, `Track`, `SoundDef`, `PlayOpts`, `midi`, `mtof`, `noteFreq`, `noteName`, `parsePattern`, `checkSong`. Playing them (`sound`, `Sfx`) is `game.ts`'s job, from the main entry. - **Only presentation code plays sound** (`game.ts`, effects), never `rules.ts` or `bot.ts`. Hook sounds where the visuals already react to an event, so every client hears each event once. - **Yours vs theirs:** your own actions full volume; rivals' quieter (0.5–0.7) and panned by where they are. `Sfx` does it: ```ts const sfx = new Sfx({ you: seats.you, halfWidth: 8, rival: 0.6, rivalPitch: -2, live: () => flow.live }); sfx.for(SOUNDS.jump, i, f.x); // player i's sound sfx.at(SOUNDS.land, x); // a world sound, panned sound.play(SOUNDS.win, { vol: 0.8, pan: sfx.pan(x) }); ``` First person and chase cameras hear from the camera instead: `ears` (a three.js camera works as is) and `at3d`, quieter with distance and panned by where the sound is left or right of the view: ```ts const sfx = new Sfx({ you: seats.you, ears: () => this.view.camera, falloff: 0.07, live: () => flow.live }); sfx.at3d(SOUNDS.boom, x, y, z); // half as loud 14 units away; too faint plays nothing sfx.at3d(SOUNDS.shot, x, y, z, { vol: 0.8, pitch: -2 }); sfx.at3d(SOUNDS.creak, x, y, z, { near: 1, reach: 12 }); // full within 1 unit, silent from 12 const { vol, pan } = hear(camera, x, y, z); // or just the numbers ``` `new Sfx({ near, reach })` makes that curve the default for `at3d` and loops. A top-down or chase camera floats far above the hero: `listener: () => this.view.rig.listener()` measures distance from what the camera looks at (pan still goes by the camera). Loops do that by themselves. Footsteps and other constant sounds: local player only, or heavily throttled. - **Sustained sounds** (`sustain: true`: fuses, wind, engines) return a `Voice`. Keep the handle, `voice.bend(semitones, glide)` to retune it, `voice.set({ vol, pan }, glide)` to move its loudness and pan, and `voice.stop(release)` when it should end, at the finish, and in `dispose()`. Nothing may keep playing after the game closes. - **Sounds that keep going at a place** (a campfire, a waterfall, a kart's engine, a generator's hum): `sfx.loop` plays a sustained patch there, heard from the ears like `at3d`, and `sfx.update()` every frame keeps its loudness and pan right as you move and turn. Out of earshot, or while the `live` gate is shut, it goes quiet and costs nothing, and fades back in when it's near. - **Levelled for you.** Each loop patch is measured once (steady loudness, K-weighted like LUFS) and brought to one quiet ambient level, about 8 dB under a typical hit. So `vol: 1` is "a normal ambient bed" whatever the patch's layers add up to: a waterfall, a fire and a hum at `vol: 1` all sit at the same loudness. Use `vol` for intent: 0.5 a faint hum, 1.5 a big bonfire, 2 an engine you're driving. Don't make loop patches loud to be heard; that's what `vol` is for. - **Distance:** full within `near` (default 2), fading, silent at `reach` (default 16: a campfire heard within 10–15 units). A waterfall or a stampede carries further (`reach: 40`), wind everywhere is `falloff: 0`. The older `falloff` still works (the same curve up close, silent from 3 / falloff). - **Stacking:** only the nearest four copies of one patch play (together at most 3 dB over the nearest), and all loops together are held under a ceiling, so twenty torches are a bed, not a wall of noise. Place as many as the map wants. - They play on the **ambience** channel (the player's Ambience slider, inside Effects). ```ts const fire = this.sfx.loop(SOUNDS.crackle, fx, fy, fz, { vol: 1.5, reach: 14 }); // a big fire, heard near it const falls = this.sfx.loop(SOUNDS.falls, wx, wy, wz, { near: 4, reach: 40 }); // a waterfall carries const motor = this.sfx.loop(SOUNDS.engine, 0, 0, 0, { vol: 2 }); motor.follow = kart.position; // or set motor.x / y / z yourself // every frame: motor.bend(kart.speed / 5); // revs this.sfx.update(); // done with it (the fire goes out; dispose() stops them all with sfx.stopLoops()): fire.stop(1.2); ``` - `sound.play` returns `null` while audio is asleep (no user gesture yet, a hidden tab), so use `?.` on the handle. - Don't touch `sound.muted` or `sound.setVolume`: the platform owns the mix. - `vp check` can't hear, but it measures: its sound log prints the game's typical sound and each loop at its loudest as heard, in dB, and warns when a loop comes within 3 dB of the typical sound (lower that loop's `vol` or `reach`). `sound.play(def, opts)` options: `pitch` (semitones), `note` ('E5' or MIDI), `freq`, `vol`, `pan` (-1..1), `dur` (gate seconds), `delay` (seconds from now), `bus`, `force` (skip cooldown and voice limits). Also `sound.panFor(offset, halfWidth)` and `sound.duckMusic(amount, seconds)`. ## 3. The Patch format ```ts interface Patch { layers: Layer[]; // mixed together; most good sounds have 2–4 vol?: number; // default 1; gameplay sounds sit around 0.1–0.5 root?: number; // the pitch layers are written at (for play({ note })) vary?: number; // random ± cents per play: use it on anything that repeats varyVol?: number; // random volume variation 0..1 reverb?: number; // 0..1 send; 0.1–0.3 for gameplay, more for big moments echo?: { time, feedback, mix, tone? }; sustain?: boolean; // hold until voice.stop() maxVoices?: number; // default 8; the oldest copy is stolen cooldown?: number; // ignore replays closer than this (s); default 0.025 bus?: 'sfx' | 'ui' | 'music' | 'ambience'; } interface Layer { wave: 'sine' | 'square' | 'sawtooth' | 'triangle' | 'noise' | 'pink' | 'brown'; freq?: number; to?: number; sweep?: number; curve?: 'exp' | 'lin'; // pitch glide steps?: number[]; stepTime?: number; // arpeggio in semitones vibrato?: { rate, depth /* cents */, delay? }; fm?: { ratio, index, decay? }; // bells, zaps, metal voices?: number; spread?: number; detune?: number; // thick detuned stacks rate?: number; // noise playback rate (lower = darker) env?: { a?, d?, s?, r? }; // defaults a 0.004, d 0.12, s 0, r 0.06 dur?: number; lock?: boolean; at?: number; vol?: number; pan?: number; filter?: { type: BiquadFilterType, freq, to?, time?, q?, track? }; drive?: number; // soft-clip grit, 1..50 crush?: number; // bitcrush levels, 4..64 } ``` Design by layers: a **transient** (a noise click or a sharp FM attack) + a **body** (a pitched tone or a sweep) + optionally a **tail** (reverb, echo). Pitch carries meaning: climb it on streaks (`{ pitch: Math.min(12, streak) }`), bright for rewards, low for danger, a touch lower for rivals. Nothing should be much louder than `SFX.explosion` (vol 0.85, three layers). Library sounds to reuse or compare against, in `SFX`: `coin`, `star`, `itemGet`, `whoosh`, `jump`, `doubleJump`, `land`, `footstep`, `dash`, `fall`, `hurt`, `powerup`, `bonk`, `pop`, `explosion`, `splat`, `crumble`, `bang`, `bellStrike`, `heartbeat`, `boo`, `creak`, `treasure`, `applause`, the loops `fuse`, `wind`, `charge`, and UI sounds (`click`, `confirm`, `deny`, `tick`, …). ## 4. Recipes ```ts /** Pickup: a two-step square "bling" with a glassy FM sparkle. Play with { pitch: streakStep }. */ pickup: { vol: 0.34, vary: 12, maxVoices: 6, cooldown: 0.03, reverb: 0.12, layers: [ { wave: 'noise', env: { a: 0.001, d: 0.012, s: 0 }, vol: 0.25, filter: { type: 'highpass', freq: 6000 } }, { wave: 'square', freq: 1047, steps: [0, 7], stepTime: 0.05, env: { a: 0.001, d: 0.28, s: 0 }, dur: 0.22, vol: 0.7, filter: { type: 'lowpass', freq: 6000 } }, { wave: 'sine', at: 0.05, freq: 2093, fm: { ratio: 2, index: 0.8, decay: 0.08 }, env: { a: 0.001, d: 0.3, s: 0 }, vol: 0.35 }, ] }, /** Hop: a quick rising square chirp over a sine. */ hop: { vol: 0.3, vary: 60, layers: [ { wave: 'square', freq: 250, to: 700, sweep: 0.12, env: { a: 0.002, d: 0.16, s: 0 }, vol: 0.35, filter: { type: 'lowpass', freq: 2600 } }, { wave: 'sine', freq: 500, to: 1100, sweep: 0.1, env: { a: 0.002, d: 0.14, s: 0 } }, ] }, /** Heavy landing: a sub thud, a gravelly crunch and a metal clang. */ slam: { vol: 0.55, vary: 60, reverb: 0.15, maxVoices: 2, layers: [ { wave: 'sine', freq: 110, to: 42, sweep: 0.16, env: { a: 0.001, d: 0.3, s: 0 } }, { wave: 'brown', env: { a: 0.001, d: 0.15, s: 0 }, vol: 0.7, drive: 3, filter: { type: 'lowpass', freq: 700 } }, { wave: 'sine', freq: 520, fm: { ratio: 1.41, index: 3, decay: 0.1 }, env: { a: 0.001, d: 0.35, s: 0 }, vol: 0.22 }, ] }, /** Dash / swoosh: filtered noise sweeping up. */ dash: { vol: 0.3, vary: 40, layers: [ { wave: 'noise', env: { a: 0.02, d: 0.18, s: 0 }, filter: { type: 'bandpass', freq: 600, to: 3200, time: 0.18, q: 1.2 } }, ] }, /** Wrong / fault: a buzzy falling two-note "bwomp". */ wrong: { vol: 0.3, layers: [ { wave: 'sawtooth', freq: 220, steps: [0, -5], stepTime: 0.14, env: { a: 0.005, d: 0.3, s: 0.2, r: 0.1 }, dur: 0.28, filter: { type: 'lowpass', freq: 1200 } }, ] }, /** Rattle: random clicks, new every play (a factory). */ rattle: (rnd) => ({ vol: 0.4, maxVoices: 4, layers: Array.from({ length: 6 }, () => ({ wave: 'noise' as const, at: rnd() * 0.25, env: { a: 0.001, d: 0.02 + rnd() * 0.02, s: 0 }, vol: 0.4 + rnd() * 0.4, filter: { type: 'bandpass' as const, freq: 1500 + rnd() * 3000, q: 5 } })) }), /** A fuse: sustained crackle. const v = sound.play(SOUNDS.fuse); v?.bend(5, 0.4) as it gets frantic; v?.stop() */ fuse: { vol: 0.25, sustain: true, maxVoices: 2, layers: [ { wave: 'noise', env: { a: 0.05, s: 1, r: 0.1 }, filter: { type: 'bandpass', freq: 3500, q: 2 }, vibrato: { rate: 13, depth: 900 } }, ] }, ``` Or reuse and tune: `SFX.jump`, `{ ...SFX.pop, vol: 0.2 }`. ## 5. The Song format and the theme ```ts interface Song { bpm: number; steps?: number /* per beat, default 4 */; loop?: boolean; swing?: number /* 0..0.5 */; vol?: number; tracks: Track[] } interface Track { patch: SoundDef; pattern: string; vol?: number; transpose?: number; pan?: number; gate?: number /* 0..1, default 0.9 */ } ``` The pattern language, one token per sixteenth note: ``` C5 F#4 Bb3 a note C4+E4+G4 a chord x X a drum hit (X accented) - hold the previous note one more step . rest | a bar line (ignored; for reading) (C5 E5 G5 .)*4 repeat a group ``` Each track loops over its own length, so a 1-bar drum pattern runs under an 8-bar melody. **Every track must be a whole number of bars** (16 steps per bar at the default 4 steps per beat) or the parts drift apart. Instruments in `INSTRUMENTS`: `marimba`, `bell`, `lead`, `keys` (electric piano), `flute`, `brass`, `trombone`, `stab`, `pad`, `bass`, `softBass`, and drums `kick`, `snare`, `clap`, `rim`, `hat`, `openHat`, `crash`, `shaker`, `vinyl` (record crackle). Custom instruments are just patches (a steel drum, an organ, a chip lead). Writing an **original** theme that sounds like the game plays: - 8–16 bars, ideally an A and a B section so it doesn't wear thin over a minute; - a real progression for the mood (I–V–vi–IV bright; vi–IV–I–V wistful; i–bVII–bVI–V spooky or tense; a I–IV–V calypso for beaches), a melody built from a motif (repeat it, vary it, answer it), a bass that bounces or walks, drums that fit the genre; - 100–140 bpm; remember Flow speeds it up 12% for the finale; - balance: melody `vol` 0.8–1, chords and bass below, drums punchy but not dominant. A small example (C major, 8 bars as two 4-bar halves): ```ts export const MUSIC: Song = { bpm: 120, swing: 0.06, tracks: [ { patch: INSTRUMENTS.marimba, vol: 0.9, pattern: 'C5 . E5 . G5 . E5 . A5 . G5 . E5 . D5 . | C5 . E5 . G5 . C6 . B5 . G5 . A5 . G5 . | ' + 'F5 . A5 . C6 . A5 . G5 . E5 . C5 . E5 . | D5 . F5 . A5 . G5 - - - . . . . . . | ' + 'C5 . E5 . G5 . E5 . A5 . G5 . E5 . D5 . | C5 . E5 . G5 . C6 . B5 . G5 . A5 . G5 . | ' + 'F5 . A5 . C6 . A5 . G5 . F5 . E5 . D5 . | C5 - - - . . . . . . . . . . . . ' }, { patch: INSTRUMENTS.bass, gate: 0.5, vol: 0.85, pattern: '(C3 . . . C3 . . . A2 . . . A2 . . . | F2 . . . F2 . . . G2 . . . G2 . . .)*4' }, { patch: INSTRUMENTS.kick, vol: 0.6, pattern: 'X . . . x . . . X . . . x . . .' }, { patch: INSTRUMENTS.snare, vol: 0.5, pattern: '. . . . X . . . . . . . X . . .' }, { patch: INSTRUMENTS.shaker, pan: 0.3, vol: 0.7, pattern: '(x x X x)*4' }, ], }; ``` (Melody 8 bars, bass 2 bars × 4 = 8, drums 1 bar each.) The library themes in `SONGS` are there to study or borrow a drum track from; write the game's own theme. (The procedural composer is in `/experimental` since 2.0: don't build on it.) ## 6. Checking your work: `sounds.test.ts` Because `sounds.ts` imports only `/core`, tests can load it. The template ships `sounds.test.ts`; keep it, and run it with `bunx vp test` (and `vp check` runs it too): ```ts import { expect, test } from 'bun:test'; import { checkSong } from '@voxelparty/sdk/core'; import { MUSIC } from './sounds'; // Every track loops over its own length: one that isn't a whole number of bars drifts off the beat. test('the theme loops cleanly', () => { expect(checkSong(MUSIC)).toEqual([]); }); ``` `checkSong(song, beatsPerBar = 4)` returns a `SongIssue[]` (`{ track, steps, problem }`): each track of a looping song whose length isn't a whole number of bars, and each pattern that doesn't parse (a bad note like `H5`, a broken `( … )*n`). Fix the pattern, not the test: pad a short track with `.` rests or finish the phrase. Several songs (a calm theme and a frenzy theme)? Check each one. `parsePattern(pattern)` → `{ events, length }` is there for checks of your own (the melody is 8 bars: `length === 128`). Write one bar per `|` group, 16 tokens each, so the counts are easy to read. `vp check` runs muted, so you can't hear anything: ask the user to listen in `vp dev`. ## Music that has to line up with gameplay (rhythm games) Flow starts `music` under the title card, so it isn't in step with `flow.clock`. For a beat you play to: leave `music` out of `defineGame` (or fade Flow's copy with `flow.musicVolume(0)`), and at GO start your own copy on the chart's clock: ```ts const stepDur = 60 / MUSIC.bpm / (MUSIC.steps ?? 4); // seconds per step this.song = sound.playSong(MUSIC, { offset: flow.clock / stepDur }); // offset counts steps // dispose(): this.song?.stop() ``` Judge hits on the press time against the chart, not on the frame the press was noticed, and judge every press (two can land in one frame): ```ts for (const p of input.pressTimes('action')) { const beat = ((p.at - link.startAt) / 1000) / stepDur; // `at` is on the link clock, like flow.clock judge(beat); } ```