What source looks like before it's parsed: a flat list of tokens, each a stretch of text and the lexer rule it matched. A macro gets them when its input is typed Array<Token> - the body of a body macro, or the string of a string macro, lexed as Refingere - and can return them, to be read in its place. It runs at the Lex phase by default, before there's a Syntax Tree.
Tokens don't overlap and leave no gaps, so joining every token's text() gives the source back exactly - whitespace and comments are tokens too. Most kinds are one lexer rule; EmptyBrackets and Context to StringMacro are merged from the tokens they start with, a bracket pair or a prefix and what follows it. Keywords aren't a kind of their own: let, if and true are Words, which the parser recognizes by their text.
#tokenOne lexed token: its kind, its text as written, and where it is. Token.lex splits source into tokens - what a macro returning Array can build its result with. The Span is the same as a syntax tree node's.
#tokenkindThe lexer rule a token came from - one card each below.
#token-spaceWhitespace and comments, run together: a /* x */ b has one Space token between its two Words. Kept in the token list, so joining every token's text gives back the source - skip them to see only the code. A tab is a lex error.
Rule (?:\s+|\/\*(?:[^*]|\*(?!\/))*\*\/|\/\/\/[^\n]*)+
SyntaxComment
#token-wordA name - and every keyword, and true and false, which lex as Words the parser then recognizes. The superscript and subscript letters are left out, so xᵢ is a Word and a Subscript.
Rule [_\p{ID_Start}][\p{ID_Continue}]*
SyntaxWord Reference, Boolean Literal
#token-numberA number in any base, underscores included. Its text is as written - -1 is a Symbol and a Number.
Rule (0x[0-9a-fA-F_]+(\.[0-9a-fA-F_]+)?)|(0d[0-9_]+(\.[0-9_]+)?)|(0o[0-7_]+(\.[0-7_]+)?)|(0b[0-1_]+(\.[0-1_]+)?)|([0-9][0-9_]*(\.[0-9_]+)?([eE][+-]?[0-9][0-9_]*)?)
SyntaxNumber Literal
#token-stringA string in any of the four quote styles, quotes and escapes included in its text.
Rule '''(?:\\[\s\S]|(?!''')[\s\S])*''' | """(?:\\[\s\S]|(?!""")[\s\S])*""" | '(?:\\[\s\S]|[^'\\])*' | "(?:\\[\s\S]|[^"\\])*"
SyntaxString Literal
#token-pathA backtick-quoted name, backticks included.
Rule \`(?:\\[\s\S]|[^\`\\])*\`
SyntaxPath Reference
#token-symbolAn operator, :=, . - one of the spellings in the Symbols and punctuation index, or the compound <op>= of an infix operator. The longest one wins, and nothing else is merged: a+-b has the Symbols + and -, and f()!!.x has !! and .. A symbol character that starts no known spelling is a Symbol of its own. A lone < or > is a Bracket. ?> and !> are Symbols only after a space, so Array<String?> still closes its generics with a Bracket; ?|> and !|> always are.
Rule the longest known symbol or <op>=, else one symbol character
SyntaxSymbol Literal, Operator Expression
#token-bracketOne bracket - each is a token of its own, even when two are next to each other, so Map<K, Array<V>> ends in two. < and > are brackets for generics, and also less-than and greater-than; <=, >= and ..< are Symbols.
Rule \{|\[|\(|\)|\]|\}|<|>
SyntaxBlock, Call, Index Access, Generic Instantiation
#token-emptybracketsAn opening bracket right before its closing one, as one token - the empty tuple, list or block, or a call with no arguments.
Rule () | [] | {}
SyntaxTuple Literal, List Literal, Block
#token-superscriptA power - a² is a ^ 2. Digits, a sign, or superscript letters, which stand for a name.
Rule [⁺⁻]?[⁰¹²³⁴⁵⁶⁷⁸⁹ᵃᵇᶜᵈᵉᶠᵍʰⁱʲᵏˡᵐⁿᵒᵖʳˢᵗᵘᵛʷˣʸᶻ]+
SyntaxBinary Operation
#token-subscriptAn index - xs₀ is xs[0] - or a root's degree, √₃x.
Rule [₊₋]?[₀₁₂₃₄₅₆₇₈₉ₐₑₕᵢⱼₖₗₘₙₒₚᵣₛₜᵤᵥₓ]+
SyntaxIndex Access, Root
#token-contextAn injected value read by name. & followed by a space is a Symbol instead.
Rule & Word, with no space between
SyntaxContext Reference
#token-attributeAn attribute's name. Its arguments, if any, are the tokens after it.
Rule @ Word, with no space between
SyntaxAttributes
#token-macroA macro used by name, with no body.
Rule # Word, with no space between
SyntaxMacro Invocation
#token-macrobodyA macro with its body, all one token: the body is text for the macro, not tokens of its own. It ends at the first unescaped }}}.
Rule Macro, one space, \{\{\{(?:\\[\s\S]|[^\\])*?\}\}\}
SyntaxMacro With Body
#token-stringmacroA macro applied to a string, all one token - the string is text for the macro.
Rule Macro, then a String, with no space between
SyntaxString Macro
#token-placeholderA #{ expression } in a body or string macro's text, when the macro takes it as Array<Token>: one token, its text the whole placeholder. Ast.splitPlaceholders parses them instead, for a macro taking a String.
Rule #{ … } - only in a macro's input
SyntaxString Macro, Macro With Body