grove

Hand-written lexers and parsers for source code analysis, in pure Go (CGO_ENABLED=0).

The first language is Python. The first consumer is pyscn, where grove replaces the tree-sitter (cgo) parsing path.

Status: early development. The Python lexer and parser are complete: parser.Parse reproduces CPython 3.13's ast.parse exactly across the whole CPython standard library.

Layout

The package layout mirrors the Go standard library's go/token, go/ast, go/parser:

python/token    token kinds, positions, spans
python/lexer    tokenizer (INDENT/DEDENT, PEP 701 f-strings, t-strings)
python/ast      typed AST modeled on CPython's ast module
python/parser   recursive descent parser (Parse, ParseExpr)

Python lexer

• Logical NEWLINE tokens; newlines inside brackets and replacement fields are suppressed.
• INDENT / DEDENT from an indentation stack, with tab/space consistency checks.
• PEP 701 f-strings: FSTRING_START / FSTRING_MIDDLE / FSTRING_END with regular tokens inside replacement fields, quote reuse, nesting, and format specs. Token boundaries match CPython's tokenizer exactly.
• t-strings (PEP 750) via the same machinery.
• Comments are collected on a side channel, not emitted as tokens.
• Supported grammar: Python 3.9 through 3.14.

Position contract

Positions follow CPython's ast module:

• Line is 1-based.
• Col is a 0-based UTF-8 byte offset within the line.
• Offset is a 0-based byte offset within the file.
• Spans are half-open: [Start, End).

Python parser

parser.Parse(src) parses a module like CPython's ast.parse(src, mode="exec"), and parser.ParseExpr(src) parses a single expression like ast.parse(src, mode="eval"). Both return a typed AST (python/ast) whose node names, field names, and positions match CPython's ast module exactly:

• The full expression grammar of Grammar/python.gram: precedence climbing over the operator tower, comparison chains as a single Compare, conditional expressions, lambdas (positional-only /, keyword-only *, defaults), starred and double-starred elements, walrus :=, await, parenthesized yield / yield from, all four comprehension forms (including async for), and full attribute/call/subscript/slice trailers.
• String literals are fully decoded: escape sequences (including \N{...} via a generated Unicode name table in python/internal/unicodename), bytes, raw strings, implicit concatenation, and f-strings (JoinedStr/FormattedValue with conversions, nested format specs, and {x=} debug fields). Constant nodes also keep the raw source text.
• The full statement grammar: assignments (chained, augmented, annotated), if/while/for/with/try (including except*), functions and classes with decorators and PEP 695/696 type parameters, type aliases, imports, async variants, and match statements with the complete pattern grammar. Soft keywords (match, case, type, _) and parenthesized with-items are disambiguated by bounded backtracking, mirroring the PEG grammar's ordered choice; non-ASCII identifiers are NFKC-normalized like CPython's parser.
• Syntax errors carry positions; the parser stops at the first error.

Testing

Both layers are verified against CPython oracles with golden tests:

• Lexer: tools/golden/gen.py records tokenize token streams (kind, text, positions) and the Go tests compare them exactly. Corpus in python/lexer/testdata/; point GROVE_CORPUS at X.py/X.json pairs for more (the full CPython 3.13 standard library passes).
• Parser: tools/golden/gen_ast.py records ast.parse trees as JSON (node types, all fields, positions, bit-exact constant values) for source chunks separated by #--- lines, in eval mode (expression files) or exec mode (stmts_* files). Corpus in python/parser/testdata/; point GROVE_AST_CORPUS at X.py/X.json pairs for more (157k expressions extracted from the CPython 3.13 standard library pass).
• Full files: tools/golden/gen_corpus.py converts a whole source tree into a full-file exec-mode corpus checked via GROVE_AST_CORPUS_EXEC. Every parseable file of the CPython 3.13 standard library (1747 files) matches ast.parse exactly: make corpus corpus-test.

make test        # unit + golden tests
make golden      # regenerate lexer goldens (needs python3 >= 3.12)
make golden-ast  # regenerate parser goldens (needs python3 == 3.13)
make corpus corpus-test  # verify against the full CPython 3.13 stdlib

License

MIT