feat(parser): emit jsx_text tokens for JSX text in .tsx (#70) - #71
Merged
Conversation
Follow-up to #61/#69, which fixed JSX text tokens for .jsx only — the context-free lexer can't resolve the TSX `<T>` JSX-vs-generic ambiguity, so it was gated off for TS. The parser CAN disambiguate (it has committed to JSX by the time it parses children), so the fix is parser-driven: as `parseJsxChildren` consumes each text run, collapse it into one `jsx_text` token, and materialize whitespace-only gaps (`emitJsxGap`) as jsx_text tokens too. The .tsx token stream is now byte-identical to .jsx. Done WITHOUT remapping any token-index reference. JSX is never parsed inside a backtracking speculative context (the `(...)` cover grammar parses its contents once and reinterprets; every `tok_i` rewind is around TS types, which never contain JSX), so the in-place token-array edits can't be rolled back, and because the AST is built left-to-right, nodes before a run point at unshifted tokens while nodes after (incl. the enclosing element's end-token) are built post-shift — nothing to remap. A `record_tok_muts` guard makes both helpers no-op if we ever WERE speculating (degrading to the pre-#70 JS-token stream rather than corrupt). - collapseJsxText: re-tag the first token, memmove the 5 SoA columns down, shrink parsed_len/tokens.len. Shrink only → never reallocates → cached pointers stay valid. - insertJsxGapToken: shift the tail up by one within the lexer's spare capacity (collapses free slots faster than gaps consume them); falls back to the existing token-less gap node if there's no room. Never reallocates. Gated to .tsx (in .jsx the lexer already emits the token, zero-cost) and to JSX parsing, so .js/.ts/non-JSX carry no overhead. Validated: .tsx token streams byte-identical to .jsx (the issue repro, multi-word content, nested elements, inter-element + trailing whitespace gaps, generic type args on a JSX element); TSX generic arrows/calls stay token-free; full suite green incl. test262 3966/3966 · 1389/1389; TS 17910/17913 · 1210/1223; babel 1928/1928 · 1548/1548; sweep 0 crashes; 3,000,000-iteration ReleaseSafe fuzz of random TSX (3.8M jsx_text tokens) with zero crashes/non-monotonic/OOB.
…place) The first cut collapsed/inserted tokens in place during parsing, one memmove per run — O(n²) on JSX-heavy .tsx (a 4000-element list took ~1.5s; review flagged it a blocker). Replace it with a single post-parse pass, `rewriteTsxJsxTextTokens`: - Derive the edits from the FINAL jsx_text_node / jsx_gap_node nodes (speculative ones were rolled back via nodes.len, so there are no stale edits — this also drops the fragile "JSX is never speculative" reasoning the review found wrong). - Two in-place cursor passes — forward to collapse text runs (shrink), backward to insert gap tokens (grow, within the lexer's spare capacity, else keep the token-less gap node) — building an old→new token-index map. - Remap every token-index reference in the AST. Audit: the ONLY ones are `main_token` (all nodes), `node_end_toks` (all), `jsx_text_node.lhs`, and the class `implements` entries in extra_data. break/continue labels and import/export specifiers store NODE indices despite the ast.zig "…token" comments — remapping them was a bug (corrupted node indices), now removed. In place because the token array is owned by the caller (the Ast only borrows it), so it can't be swapped for a freshly built one. Gated to .tsx with JSX text/gaps, so .js/.ts/non-JSX and .jsx carry zero overhead. Validated: 4000-element list now single-digit ms (was 1.5s); .tsx jsx_text token streams byte-identical to .jsx; new test extracts import names / labels / impls text after JSX edits to prove the remap (a wrong index yields wrong text); full suite green incl. test262 3966/3966 · 1389/1389; TS 17910/17913 · 1210/1223; semantic sweep byte-identical to main, 0 crashes; 3,000,000-iteration ReleaseSafe fuzz of random TSX (3.8M jsx_text tokens) with zero crashes/non-monotonic/OOB.
…ssed
Three review agents independently caught that the post-parse token remap was
INCOMPLETE: it covered main_token, node_end_toks, jsx_text_node.lhs and class
implements, but missed four more token-index references, so a construct after a
multi-token JSX text run in .tsx kept a stale index:
- EnumData.name / InterfaceData.name / TypeAliasData.name (token index at
extra_data offset 0) — ast.nodeName reads these, so e.g. an `enum Color` after
JSX text was named "{" (its symbol registered under the wrong name/hash,
breaking resolution). Proven end-to-end through the semantic analyzer.
- jsx_identifier.lhs — the end token of a hyphenated name (`aria-foo-bar`) the
AST-buffer consumer uses to slice the full text.
The semantic sweep masked this because it prints structural counts, not symbol
names. Add the four to the remap; the targeted #70 test now also extracts
enum/interface/alias names and the hyphenated JSX name after JSX edits (a wrong
index yields wrong text), so this class of omission is caught.
The audit also reconfirmed: break/continue labels and import/export specifiers
store NODE indices (correctly left un-remapped); the impls offset and the
compaction algorithm are correct; perf is clean O(n) (4000-element list ~0.56ms),
non-JSX zero overhead, no leaks. Enum name now resolves to "Color"; full suite
green; TS 17910/17913 · 1210/1223; sweep byte-identical, 0 crashes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #70 (follow-up to #61/#69, which fixed
.jsxonly).Problem
The context-free lexer can't resolve the TSX
<T>JSX-vs-generic ambiguity, so #69 gated JSX-text tokenization off for TS. In.tsx, JSX child text stayed lexed as ordinary JS tokens — diverging from espree/@typescript-eslintand from.jsx.Fix — a single O(n) post-parse pass
The parser already builds correct
jsx_text_node/jsx_gap_nodes for.tsx(the AST/ESTree is already right). After parsing,rewriteTsxJsxTextTokensrewrites the token stream to match.jsx:jsx_text_node→ collapse its run to onejsx_texttoken; eachjsx_gap_node→ insert ajsx_texttoken for the whitespace gap. Only committed nodes exist (speculative ones were rolled back vianodes.len), so there are no stale edits.main_token(all nodes),node_end_toks(all),jsx_text_node.lhs, and the classimplementsentries inextra_data. (break/continue labels and import/export specifiers store node indices despite misleading ast.zig comments — not remapped.)In place because the token array is owned by the caller (the Ast only borrows it). Gated to
.tsxwith JSX text/gaps, so.js/.ts/non-JSX and.jsxcarry zero overhead.Why O(n) (an earlier in-place-during-parse cut was O(n²))
A previous revision collapsed/inserted in place during parsing — one memmove per run → O(n²) (a 4000-element list took ~1.5s). The post-parse pass does it in two linear cursor passes. A 4000-element
.tsxlist is now single-digit ms.Result: byte-identical to
.jsxThe issue repro
<div>\n unrelated{ foo }\n</div>yieldsJSXTextfor the content and the trailing gap, matching.jsxexactly. Parity tests assert.tsx==.jsxjsx_text ranges across content, nesting, inter-element + trailing gaps, and generic type-args on a JSX element; TSX generic arrows/calls stay token-free; a remap test extracts import names / labels /implementstext after JSX edits (a wrong token index yields wrong text).Validation
.tsx-with-JSX).