Music Box

A five-second tune, made sample by sample, saved as sound, drawn as a picture and played back as a video - in Refingere alone, with no codec libraries. It writes five files: the tune as WAV and FLAC, its waveform as PNG and JPEG, and an animated GIF with a playhead crossing the waveform in time with the music. Then it reads each one back to check what survived.

A waveform of sixteen plucked notes, each beat its own shade from blue to red, lighting up as a white playhead crosses it
music-box.gif and music-box.flac, as the program writes them. They're separate files, so start both at once to watch the playhead keep time.

Making the sound

A Sound is samples: numbers from -1 to 1, sampleRate of them a second for each channel. pluck makes one note. The note's pitch comes from frequency, using the MIDI numbering where 69 is the A at 440 Hz and every 12 steps doubles the pitch. Its stereo position is picked by note, low on the left and high on the right.

A sine wave is the height of a point going round a circle. Turning the point by the same small angle every sample takes four multiplications, so sin() and cos() run once a note instead of 22,050 times a second, and the octave above comes free from sin 2a = 2 sin a cos a. Multiplying level by the same factor each sample makes the note die away exponentially, like a struck tine. The interleaved samples, left then right, become a Sound through Sound.fromSamples. Sound.generate(rate, channels, seconds, λ(time, channel) ...) is the shorter way to write a sound by formula, but it calls the formula once per sample.

song starts with five seconds of silence and mixes each note in at its beat, the melody at half volume over a bass note per bar. normalize(0.9) then scales the whole tune so its loudest sample sits just under full scale, wherever the notes happened to pile up.

Drawing it

waveform splits the sound into one slice per pixel column with slice, and peak() gives each slice's loudest sample. Image.generate then colors a pixel when it's within that height of the middle, picking each beat's own hue around the color wheel. It's drawn twice, bright for the part that has played and dim for the rest.

playing builds the video a frame at a time with a VideoEncoder, which compresses each frame as it's added instead of keeping every Image. A frame is the dim picture with a crop of the bright one drawn over it, up to the playhead. The GIF is small because the pictures are flat: two shades for each of the 20 beats, fewer than GIF's 256-color limit, so nothing needs dithering. GIF times frames in hundredths of a second, so 10 frames a second is exact where 15 would come out as 0.07 s each.

The tune's waveform, fully lit
waveform.png

Formats

Every encode returns Bytes, for &fs.write or anything else that takes them, and every decode recognizes the format from the file's first bytes. The read-back shows the difference between lossless and lossy formats. FLAC stores 16-bit samples in about a quarter of the WAV's size and decodes to exactly the same samples. PNG keeps every pixel. JPEG trades exactness for size: the sharp edges between the bars and the background are its worst case, and it comes back a few levels off on average.

  • Sound: WAV, AIFF, AU and FLAC, read and written - encode(AudioFormat.Flac) or encodeFlac().
  • Image: PNG, JPEG, GIF, BMP, QOI, TGA and PNM - see Image.
  • Video: animated GIF, APNG, Y4M and Motion-JPEG AVI - Video to read and edit one whole, VideoEncoder to write one a frame at a time.

The frames can come from the GPU too: Gpu.offscreen renders without a page, and readImage turns what it drew into an Image to add to a video.

Concepts shown

  • Building a Sound from samples, then mixing, normalizing and slicing it.
  • A sine oscillator that needs no sin() per sample.
  • Drawing with Image.generate, crop and draw.
  • Writing a video a frame at a time with VideoEncoder.
  • Encoding to and decoding from WAV, FLAC, PNG, JPEG and GIF.
  • Passing errors up with Error ! return types and !!.