Skip to content

feat(parser): emit jsx_text tokens for JSX text in .tsx (#70) - #71

Merged
ericsssan merged 3 commits into
mainfrom
fix/70-tsx-jsx-text
Jun 26, 2026
Merged

feat(parser): emit jsx_text tokens for JSX text in .tsx (#70)#71
ericsssan merged 3 commits into
mainfrom
fix/70-tsx-jsx-text

Conversation

@ericsssan

@ericsssan ericsssan commented Jun 26, 2026

Copy link
Copy Markdown
Owner

Fixes #70 (follow-up to #61/#69, which fixed .jsx only).

Problem

The context-free lexer can't resolve the TSX <T> JSX-vs-generic ambiguity, so #69 gated JSX-text tokenization off for TS. In .tsx, JSX child text stayed lexed as ordinary JS tokens — diverging from espree/@typescript-eslint and from .jsx.

Fix — a single O(n) post-parse pass

The parser already builds correct jsx_text_node/jsx_gap_nodes for .tsx (the AST/ESTree is already right). After parsing, rewriteTsxJsxTextTokens rewrites the token stream to match .jsx:

  • Derive edits from the final nodes — each jsx_text_node → collapse its run to one jsx_text token; each jsx_gap_node → insert a jsx_text token for the whitespace gap. Only committed nodes exist (speculative ones were rolled back via nodes.len), so there are no stale edits.
  • Two in-place cursor passes — forward to collapse runs (shrink), backward to insert gap tokens (grow, within the lexer's spare capacity; otherwise the gap keeps its token-less, still-ESTree-correct node) — building an old→new token-index map.
  • Remap every token-index reference. Audited to be exactly: main_token (all nodes), node_end_toks (all), jsx_text_node.lhs, and the class implements entries in extra_data. (break/continue labels and import/export specifiers store node indices despite misleading ast.zig comments — not remapped.)

In place because the token array is owned by the caller (the Ast only borrows it). Gated to .tsx with JSX text/gaps, so .js/.ts/non-JSX and .jsx carry zero overhead.

Why O(n) (an earlier in-place-during-parse cut was O(n²))

A previous revision collapsed/inserted in place during parsing — one memmove per run → O(n²) (a 4000-element list took ~1.5s). The post-parse pass does it in two linear cursor passes. A 4000-element .tsx list is now single-digit ms.

Result: byte-identical to .jsx

The issue repro <div>\n unrelated{ foo }\n</div> yields JSXText for the content and the trailing gap, matching .jsx exactly. Parity tests assert .tsx == .jsx jsx_text ranges across content, nesting, inter-element + trailing gaps, and generic type-args on a JSX element; TSX generic arrows/calls stay token-free; a remap test extracts import names / labels / implements text after JSX edits (a wrong token index yields wrong text).

Validation

  • Full suite green incl. test262 3966/3966 · 1389/1389; TS 17910/17913 · 1210/1223 (the 14 TSX cases feat(lexer): emit one jsx_text token per JSX text child in .jsx (#61) #69 gated off still pass and now carry jsx_text tokens); babel 1928/1928 · 1548/1548; semantic sweep byte-identical to main, 0 crashes.
  • Perf: 4000-element list ~single-digit ms (was 1.5s); non-JSX unaffected (the pass is gated to .tsx-with-JSX).
  • 3,000,000-iteration ReleaseSafe fuzz of random TSX (3.8M jsx_text tokens): zero crashes, zero non-monotonic token starts, zero out-of-bounds ranges.

Follow-up to #61/#69, which fixed JSX text tokens for .jsx only — the
context-free lexer can't resolve the TSX `<T>` JSX-vs-generic ambiguity, so it
was gated off for TS. The parser CAN disambiguate (it has committed to JSX by the
time it parses children), so the fix is parser-driven: as `parseJsxChildren`
consumes each text run, collapse it into one `jsx_text` token, and materialize
whitespace-only gaps (`emitJsxGap`) as jsx_text tokens too. The .tsx token stream
is now byte-identical to .jsx.

Done WITHOUT remapping any token-index reference. JSX is never parsed inside a
backtracking speculative context (the `(...)` cover grammar parses its contents
once and reinterprets; every `tok_i` rewind is around TS types, which never
contain JSX), so the in-place token-array edits can't be rolled back, and because
the AST is built left-to-right, nodes before a run point at unshifted tokens while
nodes after (incl. the enclosing element's end-token) are built post-shift —
nothing to remap. A `record_tok_muts` guard makes both helpers no-op if we ever
WERE speculating (degrading to the pre-#70 JS-token stream rather than corrupt).

- collapseJsxText: re-tag the first token, memmove the 5 SoA columns down, shrink
  parsed_len/tokens.len. Shrink only → never reallocates → cached pointers stay valid.
- insertJsxGapToken: shift the tail up by one within the lexer's spare capacity
  (collapses free slots faster than gaps consume them); falls back to the existing
  token-less gap node if there's no room. Never reallocates.

Gated to .tsx (in .jsx the lexer already emits the token, zero-cost) and to JSX
parsing, so .js/.ts/non-JSX carry no overhead.

Validated: .tsx token streams byte-identical to .jsx (the issue repro, multi-word
content, nested elements, inter-element + trailing whitespace gaps, generic type
args on a JSX element); TSX generic arrows/calls stay token-free; full suite green
incl. test262 3966/3966 · 1389/1389; TS 17910/17913 · 1210/1223; babel
1928/1928 · 1548/1548; sweep 0 crashes; 3,000,000-iteration ReleaseSafe fuzz of
random TSX (3.8M jsx_text tokens) with zero crashes/non-monotonic/OOB.
…place)

The first cut collapsed/inserted tokens in place during parsing, one memmove per
run — O(n²) on JSX-heavy .tsx (a 4000-element list took ~1.5s; review flagged it
a blocker). Replace it with a single post-parse pass, `rewriteTsxJsxTextTokens`:

- Derive the edits from the FINAL jsx_text_node / jsx_gap_node nodes (speculative
  ones were rolled back via nodes.len, so there are no stale edits — this also
  drops the fragile "JSX is never speculative" reasoning the review found wrong).
- Two in-place cursor passes — forward to collapse text runs (shrink), backward to
  insert gap tokens (grow, within the lexer's spare capacity, else keep the
  token-less gap node) — building an old→new token-index map.
- Remap every token-index reference in the AST. Audit: the ONLY ones are
  `main_token` (all nodes), `node_end_toks` (all), `jsx_text_node.lhs`, and the
  class `implements` entries in extra_data. break/continue labels and import/export
  specifiers store NODE indices despite the ast.zig "…token" comments — remapping
  them was a bug (corrupted node indices), now removed.

In place because the token array is owned by the caller (the Ast only borrows it),
so it can't be swapped for a freshly built one. Gated to .tsx with JSX text/gaps,
so .js/.ts/non-JSX and .jsx carry zero overhead.

Validated: 4000-element list now single-digit ms (was 1.5s); .tsx jsx_text token
streams byte-identical to .jsx; new test extracts import names / labels / impls
text after JSX edits to prove the remap (a wrong index yields wrong text); full
suite green incl. test262 3966/3966 · 1389/1389; TS 17910/17913 · 1210/1223;
semantic sweep byte-identical to main, 0 crashes; 3,000,000-iteration ReleaseSafe
fuzz of random TSX (3.8M jsx_text tokens) with zero crashes/non-monotonic/OOB.
…ssed

Three review agents independently caught that the post-parse token remap was
INCOMPLETE: it covered main_token, node_end_toks, jsx_text_node.lhs and class
implements, but missed four more token-index references, so a construct after a
multi-token JSX text run in .tsx kept a stale index:

- EnumData.name / InterfaceData.name / TypeAliasData.name (token index at
  extra_data offset 0) — ast.nodeName reads these, so e.g. an `enum Color` after
  JSX text was named "{" (its symbol registered under the wrong name/hash,
  breaking resolution). Proven end-to-end through the semantic analyzer.
- jsx_identifier.lhs — the end token of a hyphenated name (`aria-foo-bar`) the
  AST-buffer consumer uses to slice the full text.

The semantic sweep masked this because it prints structural counts, not symbol
names. Add the four to the remap; the targeted #70 test now also extracts
enum/interface/alias names and the hyphenated JSX name after JSX edits (a wrong
index yields wrong text), so this class of omission is caught.

The audit also reconfirmed: break/continue labels and import/export specifiers
store NODE indices (correctly left un-remapped); the impls offset and the
compaction algorithm are correct; perf is clean O(n) (4000-element list ~0.56ms),
non-JSX zero overhead, no leaks. Enum name now resolves to "Color"; full suite
green; TS 17910/17913 · 1210/1223; sweep byte-identical, 0 crashes.
@ericsssan
ericsssan merged commit 67bf983 into main Jun 26, 2026
2 checks passed
@ericsssan
ericsssan deleted the fix/70-tsx-jsx-text branch June 26, 2026 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lexer/parser: JSX text tokens not emitted in .tsx (JSX-vs-generic ambiguity)

1 participant