A macro is code that runs while the program compiles, and puts what it gives where it was used (see Macros). For the compiler that means a detour: it has to find each use, run the macro, and then take what it gave through the same steps as the rest of the program - lexed, parsed, analyzed, checked, lowered - since it's code too. That detour is five steps.
It's done in rounds, since what a macro gives back can use macros - or #comptime - of its own: each round finds the uses still in the program, runs them, and puts what they gave in their place, and the next round looks again, until one leaves nothing new to run (64 rounds at most - a macro giving back a use of itself would never stop). Running them is the comptime engine's, which runs string and body macros today. Every step is written: a round expands, runs, lexes and parses, puts what each macro gave in its use's place in the syntax tree, and analyzes, checks and lowers the whole program again with it there.
Given the program in the intermediate language. Gives every use of the program's own macros, in each file - which macro, of which kind, what it's given, and what it's on. In bootstrapping/6-expand-macros; --expand-macros prints what it found.
It works on the intermediate language rather than the syntax tree, because that's what the macros will run from, and by the time it gets there most of the work is done. Lowering has already:
#format a text, #SQL and the other embedded languages embedded documents, their placeholders lowered with the code around them - and reported one written the wrong way, #format {{{ ... }}} or #SQL"...", or a macro name that's nobody's;macro-use, with its input as written, whether it has a body, and where it is; A macro-use can be anywhere an expression can - in a function, a const's value, a default, another macro's own body - so rather than a walk written for each kind of node, expansion walks each file's part of the program as plain values, every field of every node (#runtime.fieldsOf), collecting each macro-use. No kind of node can be missed, even one added later. The derive and attribute macros put on definitions are listed on each file already, and each program macro is a use of its own, where it's defined: it's run once, over the whole program.
Each use is checked against its macro's kind: a @stringMacro against a string, a @bodyMacro with a body, and a derive, attribute or program macro never with #. A use that doesn't fit isn't passed on to be run.
Each line is where the macro is used, its name, its kind and the macro it is, and what it's given - a string or a body as written, placeholders and all, since what they mean is up to the macro - or the definition it's on.
Given what each of a file's macros gave back, from the comptime engine. Gives it as tokens, and what couldn't be lexed. In bootstrapping/7-lex-macros; --macro-tokens prints it.
A macro gives back one of three things. Refingere source - a String - is read as if it were written where the macro was used, so it's lexed here, by the same lexer as every file, with the same rules. Tokens - an Array<Token> - are taken as they are, their text joined for anything that has to be shown. Any other value - an array, a number, a map - is written as the source of a constant - [(1 as S64), (2 as S64)], "text" - and lexed as that. A value that can't be written so - a function, one of the program's own structs - is reported at the use.
What a macro gave has no place in any file, so each result gets a region of its own, past the end of the file: its tokens' positions are within the text it gave, counted from where its region starts, one region after another, round after round, so no two meet. Each lexed result keeps the macro use it came from. A problem found in it - by this step or any after, since the positions stay in the region all the way through lowering - is reported where the macro was used, with the text it gave shown under that, the problem marked in it. When the use was itself in what another macro gave, that's shown under it in turn.
What's been lexed then goes through the same steps as the program, each in a folder of its own:
8-parse-macros, --macro-ast): the tokens parsed into the expression they stand for, then put in the use's place in the file's syntax tree (place.rfn). Like expansion it walks the tree as plain values, rebuilding each node on the way to a use (#runtime.construct) and keeping every node whose span holds none as it is.9-analyze-macros, --analyze-macros): with the results in place the whole program is analyzed again, so a result's names mean what they'd mean written there - it can use the locals around the use. This step's links are the ones in the results' regions.10-typecheck-macros, --typecheck-macros): the program checked again, the same way; this step's types are the ones in the regions. Then it's lowered again, and the next round starts from that. A result that uses a macro of its own - #twice"4" + 1 - is found by the next round's expansion like any other use, at its place in the region, and replaced in turn. The program the backends see has no macros left in it.
A problem in what the inner macro gave is then shown inside what the outer one gave:
Derive, attribute and program macros don't go in an expression's place: they add definitions - a derive macro the implement blocks or methods for the struct it's on, a program macro whatever it makes of the whole program - or, for an attribute macro, give the definition to use instead of the one it's on. And what they add is what the rest of the program uses: point.describe() names a method only @derive(Describe) writes. So they run first, in rounds of their own right after analysis, with what analysis found put aside - see Compile-time Code. The same steps take what they gave in: it's lexed into a region of its own, parsed as definitions rather than an expression, and added to the file's tree, an attribute macro's in place of the definition it's on. A program macro's is a file of its own, in the folder the macro's defined in - commands.macro.rfn for the macro commands - so what it adds is visible like anything written in that folder, and a problem in it is shown in that file. Each runs once. Then the program's analyzed again with it all there, and a problem in what a derive or attribute macro gave is reported at the macro's name, @derive(Describe), with the text it gave shown under it.
A macro written @phase(CompilePhase.Typecheck) runs once the program's been type checked instead, and sees every expression's type through typeOf() - a check like @noFloats needs them. The program's type checked again after it.
src/5-interpret-not-typechecked/expandMacros.ts runs the program's @stringMacro and @bodyMacro macros with the tree-walking interpreter, right after analysis, and puts each result where the macro was used: a String is parsed as source, any other value becomes a constant. Derive, attribute and program macros aren't run yet.
Next: Compile-time Code.