feat(lexer): emit one jsx_text token per JSX text child in .jsx (#61) - #69
Merged
Conversation
JSX child text was tokenized as ordinary JS tokens (an identifier per word) and the surrounding whitespace was dropped as trivia, so the token stream diverged from espree/@typescript-eslint (which emit one JSXText token spanning the whole text node). The `jsx_text` tag and the `jsx_text_node` AST node already existed with the correct range — only the token stream was wrong. The lexer is context-free, so it now tracks JSX structure itself: a stack of frames, each an element BODY (text context) or an expression container EXPR. `<tag>` pushes a BODY, `</tag>` pops it; `{` in a body pushes EXPR and the matching `}` pops it (the frame's low bits count nested object/block braces so the right `}` closes the container); self-closing `<br/>` and fragments `<>` are handled. In body text context the scanner emits everything up to the next `< { > }` as one jsx_text token, whitespace included; a bare `>`/`}` ends the run and is lexed as its own token, which the parser rejects (matching the reference). Scoped to plain JSX (.jsx): in TSX a `<T>` is ambiguous (JSX element vs generic type args vs generic arrow) and only the parser's speculative parse can disambiguate — committing in the lexer regressed 14 tsx/ts conformance cases, so element-body tracking is gated off for TS. TSX keeps its prior token stream (tracked as a follow-up). `jsx_text_mode` is threaded as a COMPTIME parameter so the whole text-tracking path compiles away for non-JSX input — the hot lexer for the vast majority of files is byte-for-byte identical to before (measured: no perf delta on a 274 KB non-JSX file). Validated: token streams match the reference exactly (the issue repro yields JSXText[5,18] + [37,38]); full suite green incl. test262 3966/3966 · 1389/1389; babel 1928/1928 · 1548/1548; TS conformance 17910/17913 · 1210/1223 — all at baseline.
…ace paths Per review: add the three untested code paths the change introduced — the heap frame-stack spill (>64-deep nesting, the only allocating path), the has_newline_before carry after a multi-line jsx_text token, and the expression-container brace counting (object literal whose inner `}` must not close the container early). Also cover parent-body trailing text and whitespace-only text, and bump the helper's range buffer 16 → 64 so a many-text-child input can't index out of bounds. (Reviews otherwise clean: correctness verified under a 2M-iteration fuzz; conformance byte-identical to main; comptime gating proven by codegen size with no measurable non-JSX perf delta.)
…est gaps Adversarial fan-out review (8 dimensions, each finding independently reproduced) surfaced one latent crash, one real perf regression, and test gaps. Fixes: - perf: the prior gating left the [64]u32 frame array + the structural-switch code in the `false` instantiation (shared by non-JSX and TSX), costing a measured ~2% on the non-JSX hot lexer (interleaved median 3.68 vs 3.60s). Move the JSX-text state into a comptime-conditional `JsxState(enabled)` struct — empty (zero-size) when off — and comptime-split the structural switch into the full tracker (plain JSX) vs the original tag-header/attr-brace tracker (TSX). The non-JSX worker is now byte-identical to main: re-benchmarked at parity (median 3.49 vs 3.53s), frame array gone. - crash: Parser.tokenText null-deref'd on the new jsx_text token (ordinal 130 fell outside its variable-lexeme fast-path range, and lexeme(.jsx_text) is null). Latent today (no caller hits it — parseJsxChildren uses tok span fields) but a landmine; add an explicit jsx_text case and fix the stale comment that wrongly claimed jsx_text sat in the trailing range. - tests: the self-closing assertion didn't actually guard `prev != .slash` (bug-injection still passed) — replace with `<br/>tail` -> none. Add the two untested paths: the EXPR-container heap-grow site and bare `>`/`}` text-splitting. Accepted (noted, not fixed): a bare `}` after only whitespace (`<div> } </div>`, invalid input) loses one diagnostic main happened to emit — no valid program or conformance baseline is affected, and it merely makes the parser's already-lenient bare-`}` handling uniform. Validated: full suite green incl. test262 3966/3966 · 1389/1389; babel 1928/1928 · 1548/1548; TS 17910/17913 · 1210/1223; jsx_text token streams unchanged.
This was referenced Jun 26, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #61 for plain JSX (
.jsx). TSX coverage is a documented follow-up (see below).Problem
JSX child text was tokenized as ordinary JS tokens and the surrounding whitespace was dropped:
produced
identifier "unrelated"with the leading"\n "missing — diverging from espree/@typescript-eslint, which emit oneJSXTexttoken per text node. Thejsx_texttag andjsx_text_nodeAST node already existed with the right range; only the token stream was wrong.Fix
The lexer is context-free, so it now tracks JSX structure itself: a stack of frames, each an element BODY (text context) or an expression container EXPR.
<tag>pushes a BODY,</tag>pops it;{in a body pushes EXPR and the matching}pops it (the frame's low bits count nested object/block braces so the right}closes the container); self-closing<br/>and fragments<>are handled. In body context the scanner emits everything up to the next< { > }as onejsx_texttoken, whitespace included. A bare>/}ends the run and is lexed as its own token, which the parser rejects (matching the reference).The issue repro now yields exactly
JSXText[5,18]("\n unrelated") +JSXText[37,38].Scope:
.jsxonly (not.tsx)In TSX a
<T>is ambiguous — JSX element vs generic type-args vs generic arrow — and only the parser's speculative parse can disambiguate. Committing to "this is JSX" in the lexer regressed 14 tsx/ts conformance cases (<T,>() =>,<Foo<T>>, …). So element-body tracking is gated off for TS (jsx_text_mode = is_jsx and !is_ts); TSX keeps its prior token stream. Full.tsxcoverage needs the parser-driven / post-lex approach — filed as a follow-up.Performance
jsx_text_modeis a comptime parameter, so the entire text-tracking path compiles away for non-JSX input — the hot lexer used by the overwhelming majority of files is byte-for-byte identical to before. Measured on a 274 KB non-JSX file: no perf delta (interleaved runs overlap, best-of 3.65s vs 3.63s).Validation
jsx_text.