Lexing splits a file's text into tokens: the words, numbers, strings, symbols and brackets it's written in, each with where it is. The parser then works with tokens instead of characters, which makes it much simpler - it never has to think about spaces inside a name, or where a string ends.
Given a file's text. Gives its tokens, in order. In bootstrapping/1-lex; --tokens prints them. The kinds of token are listed, each with its rule and examples, on Tokens.
Every character of the file belongs to exactly one token. Whitespace and comments are tokens too - Space tokens - so joining every token's text gives back the file exactly, character for character. That's what lets a macro that takes tokens give them back unchanged, a formatter rewrite a file without losing its comments, and an error point at an exact stretch of the source.
Each kind of token has a rule - a regular expression - and the rules are tried in a fixed order at each position. The first rule that matches there makes the token, and lexing goes on from where it ends. Where two rules could match, the order decides: the rules for merged tokens come before the rules for what they start with, so #format"..." is one StringMacro token rather than a Macro followed by a String.
A Symbol is the longest operator known at that position - ..<, !?, :=, and any operator followed by = for a compound set - so a..<b doesn't lex as ., ., <. A lone < or > is a Bracket, since the parser decides whether it's comparing or opening a type's arguments. Keywords aren't a kind of their own: let, if and true are Words - so a new keyword never changes how anything lexes.
Each token also has its spelling: for a keyword, a bracket, a comma or an operator, the case of the Spelling enum it is - Spelling.Let, Spelling.OpenBrace, Spelling.Assign - and Spelling.None for anything else, a name or a number or a string. Every way of writing an operator has the same one: ≔ and := are both Spelling.Assign, λ and lambda both Spelling.Lambda. The lexer looks each one up once, and the parser tells tokens apart by their spelling - comparing two cases of an enum is quicker than comparing two texts.
Trying every rule at every position would be slow, so the lexer only tries the rules that can start with the character there - a letter is only ever a Word's start, a digit only a Number's - and remembers, for each character it meets, which rules those are. It tries them in the same order, so the first to match is the same one it would be.
The body of a body macro - #SQL {{{ ... }}} - is taken whole, up to its first unescaped closing braces, since it's written in another language and only its macro reads it. A string macro's string likewise. Their #{ } placeholders are found later, when they're parsed.
Each token records the line it starts on and how far that line is indented - the spaces it starts with. A token with a line break inside it - a Space spanning lines, a string or comment over several lines - moves the count on, and the spaces after its last break are the next line's indentation. The parser uses the lines to tell where a statement ends: a statement ends with its line, unless it's inside brackets.
Each token has its kind, its start and end as positions in the file (counted in characters), its line, its line's indentation, its text as written, whether it goes on with the statement on the lines above, and its spelling.
A character that no rule matches - No token starts with '§', at that character. A tab is one of them: it isn't allowed anywhere in Refingere source, since indentation is spaces, so that it looks the same in every editor. A character that's a symbol, like →, lexes - it's the parser that finds it doesn't belong where it is. The first such character stops lexing that file, since nothing after it can be trusted to lex the way it was meant to.
src/1-lexer works the same way, with the same kinds and the same rules; a test checks the rules on Tokens against it.
Next: Parsing.