Skip to content

feat(lexer): emit one jsx_text token per JSX text child in .jsx (#61) - #69

Merged
ericsssan merged 3 commits into
mainfrom
fix/61-jsx-text-tokens
Jun 26, 2026
Merged

feat(lexer): emit one jsx_text token per JSX text child in .jsx (#61)#69
ericsssan merged 3 commits into
mainfrom
fix/61-jsx-text-tokens

Conversation

@ericsssan

Copy link
Copy Markdown
Owner

Addresses #61 for plain JSX (.jsx). TSX coverage is a documented follow-up (see below).

Problem

JSX child text was tokenized as ordinary JS tokens and the surrounding whitespace was dropped:

<div>
   unrelated{ foo }
</div>

produced identifier "unrelated" with the leading "\n " missing — diverging from espree/@typescript-eslint, which emit one JSXText token per text node. The jsx_text tag and jsx_text_node AST node already existed with the right range; only the token stream was wrong.

Fix

The lexer is context-free, so it now tracks JSX structure itself: a stack of frames, each an element BODY (text context) or an expression container EXPR. <tag> pushes a BODY, </tag> pops it; { in a body pushes EXPR and the matching } pops it (the frame's low bits count nested object/block braces so the right } closes the container); self-closing <br/> and fragments <> are handled. In body context the scanner emits everything up to the next < { > } as one jsx_text token, whitespace included. A bare >/} ends the run and is lexed as its own token, which the parser rejects (matching the reference).

The issue repro now yields exactly JSXText[5,18] ("\n unrelated") + JSXText[37,38].

Scope: .jsx only (not .tsx)

In TSX a <T> is ambiguous — JSX element vs generic type-args vs generic arrow — and only the parser's speculative parse can disambiguate. Committing to "this is JSX" in the lexer regressed 14 tsx/ts conformance cases (<T,>() =>, <Foo<T>>, …). So element-body tracking is gated off for TS (jsx_text_mode = is_jsx and !is_ts); TSX keeps its prior token stream. Full .tsx coverage needs the parser-driven / post-lex approach — filed as a follow-up.

Performance

jsx_text_mode is a comptime parameter, so the entire text-tracking path compiles away for non-JSX input — the hot lexer used by the overwhelming majority of files is byte-for-byte identical to before. Measured on a 274 KB non-JSX file: no perf delta (interleaved runs overlap, best-of 3.65s vs 3.63s).

Validation

  • New token-level tests: the repro, multi-word text (one token), nested elements, expression containers, fragments, self-closing; plus gating tests that TSX and plain JS emit no jsx_text.
  • Full suite green incl. test262 3966/3966 · 1389/1389; babel 1928/1928 · 1548/1548; TS 17910/17913 · 1210/1223 — all at baseline.

JSX child text was tokenized as ordinary JS tokens (an identifier per word) and
the surrounding whitespace was dropped as trivia, so the token stream diverged
from espree/@typescript-eslint (which emit one JSXText token spanning the whole
text node). The `jsx_text` tag and the `jsx_text_node` AST node already existed
with the correct range — only the token stream was wrong.

The lexer is context-free, so it now tracks JSX structure itself: a stack of
frames, each an element BODY (text context) or an expression container EXPR.
`<tag>` pushes a BODY, `</tag>` pops it; `{` in a body pushes EXPR and the
matching `}` pops it (the frame's low bits count nested object/block braces so
the right `}` closes the container); self-closing `<br/>` and fragments `<>` are
handled. In body text context the scanner emits everything up to the next
`< { > }` as one jsx_text token, whitespace included; a bare `>`/`}` ends the run
and is lexed as its own token, which the parser rejects (matching the reference).

Scoped to plain JSX (.jsx): in TSX a `<T>` is ambiguous (JSX element vs generic
type args vs generic arrow) and only the parser's speculative parse can
disambiguate — committing in the lexer regressed 14 tsx/ts conformance cases, so
element-body tracking is gated off for TS. TSX keeps its prior token stream
(tracked as a follow-up). `jsx_text_mode` is threaded as a COMPTIME parameter so
the whole text-tracking path compiles away for non-JSX input — the hot lexer for
the vast majority of files is byte-for-byte identical to before (measured: no
perf delta on a 274 KB non-JSX file).

Validated: token streams match the reference exactly (the issue repro yields
JSXText[5,18] + [37,38]); full suite green incl. test262 3966/3966 · 1389/1389;
babel 1928/1928 · 1548/1548; TS conformance 17910/17913 · 1210/1223 — all at
baseline.
…ace paths

Per review: add the three untested code paths the change introduced — the heap
frame-stack spill (>64-deep nesting, the only allocating path), the
has_newline_before carry after a multi-line jsx_text token, and the
expression-container brace counting (object literal whose inner `}` must not
close the container early). Also cover parent-body trailing text and
whitespace-only text, and bump the helper's range buffer 16 → 64 so a
many-text-child input can't index out of bounds.

(Reviews otherwise clean: correctness verified under a 2M-iteration fuzz;
conformance byte-identical to main; comptime gating proven by codegen size with
no measurable non-JSX perf delta.)
…est gaps

Adversarial fan-out review (8 dimensions, each finding independently reproduced)
surfaced one latent crash, one real perf regression, and test gaps. Fixes:

- perf: the prior gating left the [64]u32 frame array + the structural-switch code
  in the `false` instantiation (shared by non-JSX and TSX), costing a measured
  ~2% on the non-JSX hot lexer (interleaved median 3.68 vs 3.60s). Move the
  JSX-text state into a comptime-conditional `JsxState(enabled)` struct — empty
  (zero-size) when off — and comptime-split the structural switch into the full
  tracker (plain JSX) vs the original tag-header/attr-brace tracker (TSX). The
  non-JSX worker is now byte-identical to main: re-benchmarked at parity
  (median 3.49 vs 3.53s), frame array gone.

- crash: Parser.tokenText null-deref'd on the new jsx_text token (ordinal 130
  fell outside its variable-lexeme fast-path range, and lexeme(.jsx_text) is
  null). Latent today (no caller hits it — parseJsxChildren uses tok span fields)
  but a landmine; add an explicit jsx_text case and fix the stale comment that
  wrongly claimed jsx_text sat in the trailing range.

- tests: the self-closing assertion didn't actually guard `prev != .slash`
  (bug-injection still passed) — replace with `<br/>tail` -> none. Add the two
  untested paths: the EXPR-container heap-grow site and bare `>`/`}` text-splitting.

Accepted (noted, not fixed): a bare `}` after only whitespace (`<div> } </div>`,
invalid input) loses one diagnostic main happened to emit — no valid program or
conformance baseline is affected, and it merely makes the parser's already-lenient
bare-`}` handling uniform.

Validated: full suite green incl. test262 3966/3966 · 1389/1389; babel
1928/1928 · 1548/1548; TS 17910/17913 · 1210/1223; jsx_text token streams
unchanged.
@ericsssan
ericsssan merged commit 7297a40 into main Jun 26, 2026
2 checks passed
@ericsssan
ericsssan deleted the fix/61-jsx-text-tokens branch June 26, 2026 01:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant