Benchmarks in Refingere are ordinary functions with an attribute, like tests, written next to the code they measure. --bench compiles them, times each one and measures the memory a call allocates, and then - when the project has an @main - runs the program once under a profiler to show where its time goes. This page covers writing benchmarks, reading their results, picking which ones run, and profiling.
@bench makes a function a benchmark. It takes no arguments and isn't @async: the runner calls it over and over, with nothing, and can't wait on a promise. Whatever it returns is thrown away. Run the project's benchmarks with --bench:
Like a test, a benchmark can go in any file, any number of them. They're compiled - to WebAssembly, or to JavaScript with --target=js - and timed as the program would run, optimizations and all. --bench can't be combined with --interpret.
A string first in @bench - or displayName := "..." - is the name the report shows. Without one, a benchmark is named by its function - with the folder it's in, from the project root, in front, for one in a subfolder - the way the test report names tests.
Each benchmark is timed on its own, in the order they're written:
const it reads, aren't counted. Time is read from the monotonic clock, which only goes forward, so changes to the system time don't distort it. A slow benchmark still runs its warm-up and five batches of one call each: six calls, however long they take.
Each benchmark gets one line:
| Part | What it is |
|---|---|
8.41 ms a call | The median batch's time for one call - the number to compare |
(8.12 ms - 9.03 ms, ...) | The fastest and slowest batch, a call each. A wide range means something else was competing for the machine, or the work varies from call to call |
128 calls | How many calls ran in all, the warm-up and sizing calls included |
2.29 MiB in 100k allocations a call | The memory one call allocates, and in how many blocks - or nothing allocated. See What a call allocates |
Times are shown to three significant digits, in whichever of s, ms, µs and ns fits. The median is used rather than the mean, so one batch slowed by garbage collection or another process doesn't move the result much.
Every value a program makes - an array, a text, a struct, a number too large to keep inline - takes memory, and the garbage collector spends time getting it back. So each line ends with the memory a call allocates: its bytes, and the number of separate blocks they're in. Two versions of the same work can be compared by it:
filter makes a new array of the items, only for size to count it; the loop counts them as they go by, in a quarter of the memory. Fewer allocations aren't always less time, though - here the loop is the slower one. The time is the result; the memory says why it is what it is, and what the collector will have to do.
The memory is counted exactly, not sampled: after the timing, the benchmark is called again, a batch of the same size, with the runtime counting every block allocated, and that's divided by the calls. What the runner's own loop allocates is measured the same way around a function that does nothing, and taken off - so an empty benchmark shows nothing allocated. Memory that's allocated and given back within the call counts too: this is what a call allocates, not what it keeps. To see what a program keeps - and which function allocates what - use --profile.
Only the WebAssembly runtime counts what it allocates - a JavaScript engine doesn't let a program see it. So with --target=js, the benchmarks are first compiled to WebAssembly and called there to measure their memory, and then timed as JavaScript, each line showing both:
The numbers are WebAssembly's, then: a JavaScript engine lays values out its own way, so the bytes a call takes there differ. They still compare one version of the code with another. When the WebAssembly build can't be made or run, the benchmarks are timed without memory.
A benchmark fails the way a test does: when a call returns an error, which !! passes up as usual. It's reported as FAIL with the error, instead of times, and the rest are still timed. When any fails, the command exits with status 1 - so a CI pipeline that runs benchmarks notices when one breaks.
Everything in a benchmark's body is timed, on every call. Input that's built inside the body is measured along with the work you meant to measure. Build it in a const instead: a const is made the first time it's read - during the warm-up - and is the same value every time after.
The other side of this: the const is shared by every call. A benchmark that changes it - sorting it in place, pushing to it - leaves the next call different input. Work on a copy, as sorted() does, or build the input in the body and measure building it separately, so you can subtract it.
A benchmark can inject context values like a test does, and every injection lasts until the end of the call. Replace what would make the time depend on the machine, or flood the terminal: a NullOutputStream for &stdout, a MemoryFileSystem for &fs, a FixedClock for &clock, whose sleep returns at once. See Testing for the full list. Like @main and @test functions, benchmarks have the builtin context values provided and declare no @require. They can use @testHelper functions, too.
Some benchmarks take too long to run on every change, or need a service running. Stage names after the display name - or on their own, @bench(e2e) - work as they do for tests: a benchmark in no stage is always timed, and one in stages only when asked for:
--bench-whole times the project's tests as well as its benchmarks, each the way an @bench function is - for a project that has tests but no benchmarks yet, or to find which tests make the suite slow. The benchmarks come first, then the tests, named as the test report names them:
The tests are picked by their own stages, by the same flags as the benchmarks:
Async tests are left out, since a benchmark can't be waited on. A test that fails is reported as FAIL, as a benchmark is. Keep in mind that each test is called many times over: one that prints, writes files or reaches a service does that thousands of times - give it replacements, or leave it in a stage --bench-whole isn't asked for.
After the benchmarks, when the project has an @main, --bench compiles the program again and runs it once, in a process of its own, under Bun's sampling profiler. The profiler records which function is running many times a second. The report lists the 20 functions it found most often:
| Column | What it is |
|---|---|
self | The share of the time spent in the function's own code |
total | The share of the time it was running - its own code, and the functions it called |
function | Its name, and the file and line it's defined on |
A high self means the function's own code is the cost; a high total but low self, that what it calls is. Work done by the runtime rather than your code is listed under (runtime), with all of garbage collection as a single line - a large share there means the program makes a lot of short-lived values.
What follows -- is @main's arguments, as with --run, so the profile can run on realistic input. A project with no @main - a library - is only timed, not profiled. The profiled run's exit status counts: when @main fails, so does the command.
For more than the top 20 - every function, who called whom and how many times, as a call graph and a flame graph you can explore - use --profile, which writes the profile as a website. See Profiling.
The refingere command takes the same flags. There, --bench only chooses what's compiled - the timing module instead of @main - so it times and profiles when --run is given too, and without it writes the module to --out. With --js, the WebAssembly module that measures the memory is bench-memory.wasm, beside it. It also keeps the profiler's own full report in a profile folder of the compiled output.