diff --git a/.changeset/java-notices.md b/.changeset/java-notices.md new file mode 100644 index 0000000..04f0263 --- /dev/null +++ b/.changeset/java-notices.md @@ -0,0 +1,5 @@ +--- +'@scalar/java-fmt': patch +--- + +Complete the third-party notices: list commonmark-java 0.28.0 (BSD-2-Clause), which google-java-format bundles and which is compiled into the artifact, with its license text. Also ship TeaVM's `NOTICE` file, as Apache-2.0 asks. No change to the artifact or its output. diff --git a/.changeset/kotlin-ktrs.md b/.changeset/kotlin-ktrs.md new file mode 100644 index 0000000..e62ae58 --- /dev/null +++ b/.changeset/kotlin-ktrs.md @@ -0,0 +1,15 @@ +--- +'@scalar/kotlin-fmt': minor +--- + +Replace the TeaVM-compiled ktfmt 0.64 with [ktrs](https://github.com/Hexay/ktrs) 0.5.3, a Rust reimplementation of ktfmt, compiled to wasm. Output is held to ktfmt **0.65** by a conformance test that runs the real ktfmt jar. That test asserts byte-identical output and matching diagnostics; over okhttp and kotlinx.coroutines (1,699 files) in every style, every file matched. + +The artifact goes from 0.82 MB to 0.20 MB and imports nothing, so `formatSync` now works under Node without `init()`, and the Node floor drops from 24.15.0 to 18. + +Breaking changes: + +- The `./runtime` export (TeaVM's generated runtime) is gone. There is no runtime file any more. +- The `BootModule`, `FormatFunction` and `ModuleLoader` types are removed. `ArtifactSource`, `KotlinFormatter` and `KotlinFormatterBoot` take their place. +- `ktfmtVersion` is now `'0.65'`, and `ktrsVersion` is new. +- Parse errors read as ktfmt's CLI prints them, without the file name (`1:7: error: Expecting ')'`), instead of carrying a Java class name prefix (`...ParseError: 1:7: ...`). +- An unknown `style` or `trailingCommas`, or a negative or fractional width or indent, now throws instead of silently falling back to a default. diff --git a/.claude/architecture.md b/.claude/architecture.md index e0456cd..889132a 100644 --- a/.claude/architecture.md +++ b/.claude/architecture.md @@ -2,7 +2,7 @@ ## Overview -`formatters` provides code formatters — Ruby, Java, C#, Swift, PHP and Rust — +`formatters` provides code formatters — Ruby, Java, Kotlin, C#, Swift, PHP and Rust — under one hard constraint: **formatting must work with nothing installed but Node.** No Ruby install, no gems, no JVM, no .NET SDK, no Swift toolchain, no PHP or Composer, no Rust toolchain, no native binaries, no postinstall downloads. The approach follows the @@ -48,8 +48,8 @@ guarded by a `peek()` that answers "is it booted" without one. One limit shapes it, measured rather than assumed: a browser main thread refuses both `WebAssembly.Module` and `WebAssembly.Instance` above 8MB, so booting in a browser always uses async `WebAssembly.instantiate` and only trap recovery tries -the synchronous form. Node has no such limit, and Ruby uses that: its Node -source reads, decompresses and compiles synchronously, so its `formatSync` +the synchronous form. Node has no such limit, and Ruby and Kotlin use that: their +Node sources read, decompress and compile synchronously, so their `formatSync` needs no `init()` at all. `format.ts` and `boot-module.ts` are factories over an artifact source rather @@ -125,7 +125,7 @@ each pipeline downloads a toolchain into a gitignored subdirectory and a glob would pull whatever TypeScript those contain into the project. Three JavaScript files are not ours to convert and must stay as they are: -`packages/csharp/runtime/*.js` and `packages/{java,kotlin}/*_fmt.runtime.mjs` +`packages/csharp/runtime/*.js` and `packages/java/java_fmt.runtime.mjs` are generated by the .NET and TeaVM toolchains and shipped verbatim, and `build/csharp_fmt/main.mjs` is the csproj's `WasmMainJSPath`, which the .NET SDK requires to be JavaScript. @@ -306,51 +306,53 @@ resolved at run time, the analysis cannot fold it, and the generic ### `@scalar/kotlin-fmt` (`packages/kotlin`) -Reference: **ktfmt**. Status: exact. - -Runs actual ktfmt 0.64 with the Kotlin compiler's own PSI parser, compiled to -WasmGC by the same TeaVM fork the Java package uses. Built by -`build/java_fmt_teavm/kotlin-probe/build.sh`; the directory is named for the -feasibility probe it started as. - -**The exactness claim is two measurements, not one,** because output could change -in two places. `gate2.sh` asks whether the one patched ktfmt file changes its -output on a JVM; `conformance.sh` asks whether compiling that to wasm changes it. -Same 589 files from kotlin-stdlib, kotlinx-coroutines and ktfmt itself, three -styles, **1767/1767 byte-identical each**, failures encoded the same on both -sides so a reworded diagnostic would count. Together they chain from stock ktfmt -to this module. - -**ktfmt as shipped does not compile, and the reason is narrower than it looks.** -Pointing TeaVM at `Formatter.format` reports 94 missing members — StAX, -`ZipFile`, `Runtime.exec`, `ForkJoinPool`, AWT and Swing. That is -`KotlinCoreEnvironment`, the compiler's *CLI environment*, which resolves a -classpath and instantiates extension points named by string in -`META-INF/extensions/compiler.xml`. Parsing needs none of it: a bare -`CoreApplicationEnvironment` with `KotlinParserDefinition` registered drops the -frontier to 40 and takes all of that with it. `patches/ktfmt.patch` is that swap, -20 lines of one file, and `gate2.sh` is the evidence it changes nothing. - -**TeaVM resolves reflection statically,** so `Class.forName`, -`getDeclaredField`, `getConstructor` and `Proxy` all need telling at build time. -`kotlin-probe/plugin/` is that: a build-time TeaVM extension, never compiled to -wasm. Scope its rules by the mechanism that needs them, not by package — a -reflectable member is a dependency root, so a package-wide rule made every class -in `com.intellij` reachable and failed to compile at all. - -**One `Thread.yield()` cost a third of the module.** `AsyncMethodFinder` -propagates async-ness from every `@Async` method to its callers over a call graph -that keeps exception-construction and virtual `toString` edges, so one reachable -`Thread.yield()` in kotlinx-coroutines made 7,055 of 20,513 methods coroutines — -including the `stringToJs` the module initializer calls, whose coroutine prologue -dereferences a fiber that does not exist yet, so the module trapped at -instantiation. Cutting one root does nothing: removing `yield` moved all 7,055 to -`Thread.sleep`. The fork's single-threaded stand-ins leave none reachable. - -**Licensing is unrestricted,** unlike the Java package. ktfmt reads Kotlin with -the Kotlin compiler's PSI and uses google-java-format only for its layout engine, -so javac is not in this module and there is no GPL component; everything embedded -is Apache-2.0. +Reference: **ktfmt** (0.65). Status: **exact by test**, because it is a +reimplementation. + +One wasm artifact, `kotlin_fmt.wasm.br`: [ktrs](https://github.com/Hexay/ktrs), +ktfmt reimplemented in Rust with its own port of the Kotlin compiler's lexer and +parser, built by `build/kotlin_fmt/build.sh` from upstream at a pinned release +commit. `build/kotlin_fmt/entry` is a small crate over `ktrs-fmt` that exposes +plain exports; nothing in ktrs is patched. + +**It is `wasm32-unknown-unknown` and imports nothing.** Formatting is a pure +function of a string, so there is no WASI and no shim: `src/kotlin-formatter.ts` +instantiates it with an empty import object. Options cross as integers, a +negative one meaning "the style's value", so ktrs's presets stay the only copy of +ktfmt's defaults. The module is 864 KB - under the 8 MB a browser main thread +allows synchronously - so instantiation is synchronous everywhere, and after a +trap the next call re-instantiates from the cached module. + +**Traps are the engine's stack, mostly.** Wasm calls run on the engine's native +stack, so how deeply ktrs's recursive parser can nest is V8's or JSC's limit, +not the module's: a larger linker stack measured no deeper. It is still deeper +than ktfmt, whose JVM overflows at 2,000 nested parentheses. A panic aborts on +this target and traps the same way. + +**Source is parsed as a script**, which is how ktfmt reads stdin and how the +TeaVM build parsed everything (ktfmt 0.64 always parsed `temp.kts`). A `.kt` +file would be held to a stricter grammar that rejects top-level statements. +The conformance test writes `.kts` files so the oracle parses the same way. + +**The exactness rule applies in the form it takes for a reimplementation.** +`test/ktfmt-conformance.test.ts` is the evidence: it runs the real ktfmt +release jar, version `KTFMT_VERSION` in `build.sh` and checksum-pinned by +`build/kotlin_fmt/ktfmt.sh`, which CI runs, and asserts byte-identity and +identical diagnostics in every style, over its samples and any corpus named by +`KOTLIN_FMT_CONFORMANCE_CORPUS`. Over okhttp and kotlinx.coroutines (1,699 +files) it matched every file in every configuration. Bump `KTRS_COMMIT` only +with a corpus run, and move `KTFMT_VERSION` to the release that ktrs version +tracks. + +**It replaced ktfmt compiled by TeaVM**, which was exact by construction. That +build cost a Node 24.15 floor, an async-only boot, a 0.82 MB artifact, a patched +copy of ktfmt's `Parser` and patches to the TeaVM fork. It is in git history +as `build/java_fmt_teavm/kotlin-probe`. The fork patches it alone needed are +still in `build/teavm`, because removing one changes the Java module. + +**Licensing is unrestricted.** ktrs is MIT OR Apache-2.0; the Kotlin compiler, +IntelliJ, ktfmt, google-java-format and ec4j code it ports is Apache-2.0, and +`licenses/NOTICE.md` carries those notices. No JDK code is in the module. ### `@scalar/csharp-fmt` (`packages/csharp`) diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index b5abbda..6a43d6c 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -38,7 +38,7 @@ body: id: node-version attributes: label: Node version - description: Output of `node -v`. Java and Kotlin need 24.15 or newer. + description: Output of `node -v`. Java needs 24.15 or newer. placeholder: v24.15.0 validations: required: true diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a3fbb23..effb233 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -86,9 +86,9 @@ jobs: # packages/ruby's conformance test is what backs the "exact by test" claim in the README, and # it skips itself when there is no native RuboCop to compare against - so without these - # steps it never actually runs here. RuboCop installs in seconds, which makes Ruby the one - # reference toolchain cheap enough for CI to carry; the others need a .NET SDK, a Rust - # toolchain or a JDK and still skip. + # steps it never actually runs here. RuboCop installs in seconds, which makes Ruby one of the + # two reference tools cheap enough for CI to carry - ktfmt, below, is the other; the rest + # need a .NET SDK, a Rust toolchain or a locally built toolchain and still skip. # # The Ruby itself does not decide the output: RuboCop parses through the pinned prism gem, # because the package targets Ruby 3.4. It is pinned anyway, so a run is reproducible. @@ -106,6 +106,21 @@ jobs: gem install "$gem" --no-document -v "$(sed -n "s/^gem \"$gem\", \"\(.*\)\"$/\1/p" build/rbcop_fmt/Gemfile)" done + # packages/kotlin is ktrs, a reimplementation of ktfmt, so it is held to the real ktfmt the same + # way: test/ktfmt-conformance.test.ts skips without the jar. The jar is the pinned release, + # checked against a pinned checksum, and runs on the JDK the runner image already carries. + # Cached, because the jar is 71 MB and never changes for a given pin: the key is the script, + # which holds the checksum, and build.sh, which holds the version. The script still verifies + # the checksum of a restored jar. + - name: Cache the ktfmt jar + uses: actions/cache@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0 + with: + path: build/kotlin_fmt/work + key: ktfmt-${{ hashFiles('build/kotlin_fmt/ktfmt.sh', 'build/kotlin_fmt/build.sh') }} + + - name: Download the ktfmt that ktrs is held to + run: build/kotlin_fmt/ktfmt.sh + - name: Run unit tests run: bun run test @@ -118,7 +133,7 @@ jobs: timeout-minutes: 20 # Both ends of the supported range, because each one catches what the other cannot. 24.15.0 is - # the floor the Java and Kotlin packages declare in `engines`, and running it here is the only + # the floor the Java package declares in `engines`, and running it here is the only # thing that keeps that number honest. The newest release catches the reverse: a Node that # stops running an artifact these packages promise it runs. # diff --git a/.gitignore b/.gitignore index 2aafc7f..1604e73 100644 --- a/.gitignore +++ b/.gitignore @@ -42,18 +42,9 @@ build/teavm/toolchain/ build/java_fmt_teavm/toolchain/ build/java_fmt_teavm/target/ build/java_fmt_teavm/*.wasm -build/java_fmt_teavm/kotlin-probe/work/ -build/java_fmt_teavm/kotlin-probe/target/ -# Scratch build directories: run.sh and any hand-run comparison gives each TeaVM -# version its own, because a build that ends "with errors" leaves the previous -# classes.wasm in place. None of them is shipped - the artifact the package uses -# is packages/kotlin/kotlin_fmt.wasm.br, written by build.sh from target-ktfmt. -build/java_fmt_teavm/kotlin-probe/target-*/ -build/java_fmt_teavm/kotlin-probe/async-dump*.txt # Emitted next to every module for diagnosis; not evidence of anything, and the # committed artifact is the module itself. *.wasm-deobfuscator.wasm -build/java_fmt_teavm/kotlin-probe/build/ __pycache__/ # dotnet wasm build scratch (the SDK, the wasm-tools workload, the publish tree) build/csharp_fmt/toolchain/ @@ -84,3 +75,11 @@ build/rust_fmt/*.wasm build/rbcop_fmt/rbcop build/rbcop_fmt/toolchain/ build/rbcop_fmt/*.wasm + +# ktrs build scratch (the ktrs checkout, the uncompressed module, the ktfmt jar +# the conformance test compares against) +# +# `ktrs` has no trailing slash on purpose, matching rust_fmt above. +build/kotlin_fmt/ktrs +build/kotlin_fmt/work/ +build/kotlin_fmt/*.wasm diff --git a/AGENTS.md b/AGENTS.md index 229c39c..c476bb9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -58,7 +58,7 @@ Every package exports the same three things, from both builds: | | | |:---|:---| | `format(source, options?)` | async; boots on demand. The one to reach for. | -| `formatSync(source, options?)` | synchronous; needs `await init()` first, and throws until it has - except Ruby under Node, whose boot is synchronous too. | +| `formatSync(source, options?)` | synchronous; needs `await init()` first, and throws until it has - except Ruby and Kotlin under Node, whose boot is synchronous too. | | `init(options?)` | boots the module. Takes `{ url, bytes, encoding }` in the browser build only. | Booting is the only asynchronous step there has ever been — fetching or reading diff --git a/CLAUDE.md b/CLAUDE.md index 4c3f990..5f4d324 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,7 +23,7 @@ entry stays the default, and `src/index.browser.ts` is the only file that may no touch `node:` built-ins — while `src/index.ts` is the only one that may. Each package exports `format` (async), `formatSync` (needs `await init()` first, -except Ruby under Node, which boots synchronously) and `init`. Booting is the only asynchronous step, so the two formatters run the +except Ruby and Kotlin under Node, which boot synchronously) and `init`. Booting is the only asynchronous step, so the two formatters run the same code and produce the same bytes. ## Exactness is a claim about a named tool diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b3f353d..099bb0c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -16,8 +16,8 @@ output against the real tool. ## Getting set up -You need [Bun](https://bun.sh) and Node 24.15 or newer — `bun run test:node` loads the Java and -Kotlin artifacts, and Node 24.0 through 24.14 will not compile them. The repo pins 26.7.0 in +You need [Bun](https://bun.sh) and Node 24.15 or newer — `bun run test:node` loads the Java +artifact, and Node 24.0 through 24.14 will not compile it. The repo pins 26.7.0 in `.nvmrc` and installs that in CI, so `nvm use` gets you the Node everything is developed against; 24.15 is only the floor, and CI runs both ends. Nothing else — the wasm artifacts are committed, so a fresh clone builds and tests without any language toolchain. diff --git a/README.md b/README.md index d8ac486..eb8ffa3 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ Composer, no native binaries, no postinstall downloads. |:---|:---|---:|:---|:---| | [`@scalar/ruby-fmt`](packages/ruby) | rbcop | 2.2 MB | ✅ exact by test | ✅ | | [`@scalar/java-fmt`](packages/java) | google-java-format | 0.76 MB | ✅ exact | ✅ | -| [`@scalar/kotlin-fmt`](packages/kotlin) | ktfmt | 0.81 MB | ✅ exact | ✅ | +| [`@scalar/kotlin-fmt`](packages/kotlin) | ktrs | 0.20 MB | ✅ exact by test | ✅ | | [`@scalar/csharp-fmt`](packages/csharp) | CSharpier | 4.2 MB | ✅ exact | ✅ | | [`@scalar/swift-fmt`](packages/swift) | swift-format | 12.5 MB | ✅ exact | ✅ | | [`@scalar/php-fmt`](packages/php) | PHP CS Fixer | 0.45 MB | ✅ exact | — | @@ -36,8 +36,8 @@ Composer, no native binaries, no postinstall downloads. Artifact is the brotli-compressed module as committed and published — the whole tool, its parser and its language runtime in one file. The JavaScript that loads -it is small by comparison: nothing for Ruby, Swift, PHP and Rust, a 17 KB -generated runtime for Java and Kotlin, and 0.46 MB of .NET loader scripts for C#. +it is small by comparison: nothing for Ruby, Kotlin, Swift, PHP and Rust, a 17 KB +generated runtime for Java, and 0.46 MB of .NET loader scripts for C#. Ruby runs rbcop, and the reference it is held to is `rubocop --autocorrect --only Layout`: the package corrects @@ -51,13 +51,19 @@ not a reimplementation of it. A reimplementation drifts, because a formatter's line-breaking heuristics are emergent from its implementation rather than specified anywhere, and the drift stays invisible until a consumer's CI fails. -Ruby is the one package that is a reimplementation: it runs -[rbcop](https://github.com/costajohnt/rbcop), RuboCop 1.92.2 reimplemented in -Rust, so it is not exact by the definition above. It is **exact by test**: a -conformance test holds it byte-identical to the real `rubocop` 1.92.2, over its -samples and over any corpus it is given, at two line widths. Corpus runs have -found three rbcop bugs, each fixed and pinned by a sample — [details -below](#ruby). +Two packages are reimplementations, so they are not exact by the definition +above. They are **exact by test** instead: each has a conformance test that runs +the real tool and asserts byte-identical output, over its samples and over any +corpus it is given. + +- Ruby runs [rbcop](https://github.com/costajohnt/rbcop), RuboCop 1.92.2 + reimplemented in Rust, held to the real `rubocop` 1.92.2 at two line widths. + Corpus runs have found three rbcop bugs, each fixed and pinned by a sample — + [details below](#ruby). +- Kotlin runs [ktrs](https://github.com/Hexay/ktrs), ktfmt reimplemented in + Rust, held to the real ktfmt 0.65 jar in every style. Over okhttp and + kotlinx.coroutines (1,699 files) every file matched in every style — + [details](packages/kotlin#the-output-is-ktfmts-held-there-by-test). Browser is the same claim held to the same standard: ✅ means the package has a `browser` export condition and `bun run test:browser` loads that build in real @@ -107,9 +113,10 @@ until it has. Everything after that already was synchronous; `format` was only ever awaiting the boot. Prefer `format` where you can await: it needs no setup call and cannot throw that -error. The one exception to the `init` rule is Ruby under Node, whose module -boots synchronously, so there `formatSync` works straight away — see -[its README](packages/ruby#formatting-without-awaiting). +error. The exceptions to the `init` rule are Ruby and Kotlin under Node, whose +modules boot synchronously, so there `formatSync` works straight away — see +[Ruby's](packages/ruby#formatting-without-awaiting) and +[Kotlin's](packages/kotlin#formatting-without-awaiting) READMEs. ## Browsers @@ -280,28 +287,28 @@ one - enough for `format()`, but it rules out shipping the CLI itself. ## Kotlin -See [`packages/kotlin`](packages/kotlin). It runs the real ktfmt 0.64 - the -Kotlin compiler's own PSI parser included - compiled to WasmGC by -[TeaVM](https://teavm.org), with the same patched TeaVM as the Java package. -The claim is checked in two halves over 589 real Kotlin files (kotlin-stdlib, -kotlinx-coroutines and ktfmt itself) in all three styles: the one patched file -leaves ktfmt's output unchanged on a JVM, and compiling that to wasm leaves it -unchanged again - 1767 of 1767 identical at each step, diagnostics included. A -conformance test asserts byte-identical output against the real ktfmt jar. - -The patch is 20 lines of ktfmt's `Parser`. ktfmt parses through the compiler's -`KotlinCoreEnvironment`, which resolves a JVM classpath, opens jars and registers -extension points named in XML - none of which an ahead-of-time compiler can -follow, and none of which parsing needs - so the patch builds the smaller -container that environment builds underneath, and everything below it is -ktfmt's own. - -Like Java it exports the release it carries, as `ktfmtVersion`, and a test reads -the build script so that export cannot go stale across a bump. It ships as a -0.81 MB `kotlin_fmt.wasm.br` plus the same TeaVM runtime as Java, both committed; -`bun run kotlin:build` rebuilds them from a JDK 21, Maven, git and Node. It shares -Java's caveat too: **Node 24.15 or newer**, for the same exception-handling -reason. +See [`packages/kotlin`](packages/kotlin). It runs +[ktrs](https://github.com/Hexay/ktrs) 0.5.3, ktfmt reimplemented in Rust with its +own port of the Kotlin compiler's lexer and parser, compiled to a plain wasm +module that imports nothing. It is not ktfmt compiled, so it is **exact by +test**, not exact: a conformance test runs the real ktfmt 0.65 jar and asserts +byte-identical output, and the same diagnostic where ktfmt refuses the input, in +all three styles and with unused imports kept. Over okhttp and +kotlinx.coroutines, 1,699 files, every file matched in every configuration. CI +downloads the jar, checked against a pinned checksum, so the test runs there +rather than skipping. + +It replaced ktfmt 0.64 compiled to WasmGC by TeaVM, which was exact by +construction but cost a Node 24.15 floor, an async-only boot, a 0.81 MB artifact +and a patched ktfmt `Parser`. ktrs is 0.20 MB, runs on Node 18 and any browser +from Chrome 96, Firefox 79 or Safari 15, and boots synchronously under Node in +about 15 ms. + +It exports the ktfmt release it is held to as `ktfmtVersion`, and the ktrs +release it is built from as `ktrsVersion`; a test reads the build script so +neither can go stale across a bump. `bun run kotlin:build` rebuilds the artifact +from rustup, git and Node in about 20 seconds, and `bun run kotlin:ktfmt` fetches +the jar the conformance test compares against. ## C# @@ -452,7 +459,7 @@ trap on deeply nested expressions native handles fine. bun install bun run build # compile each package's TypeScript to dist/ bun run test # bun test suite -bun run test:node # load the built packages under plain Node (24.15+ for Java and Kotlin) +bun run test:node # load the built packages under plain Node (24.15+ for Java) bun run test:browser # load the built browser packages in real Chromium bun run check # biome lint + format check bun run types:check diff --git a/SECURITY.md b/SECURITY.md index 04cd7fd..39aec79 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -24,8 +24,8 @@ we want to hear about: - Input to `format()` that escapes its sandbox: touching the filesystem, the network, the environment, or the host process, from a module that is supposed to be able to do none of it. - Input that hangs, or that grows memory until the process dies, in a way an ordinary large - file does not. The known and documented limits — the Ruby VM's recycle threshold, Java and - Kotlin needing Node 24.15 — are not this. + file does not. The known and documented limits — the Ruby VM's recycle threshold, Java + needing Node 24.15 — are not this. - A published artifact that does not match what its build pipeline in `build/` produces from the pinned sources, or a build pipeline that fetches something it does not verify. - A dependency or vendored component shipping a known vulnerability inside one of the artifacts. diff --git a/biome.json b/biome.json index 4bf3a0b..bc736f8 100644 --- a/biome.json +++ b/biome.json @@ -20,13 +20,7 @@ }, "files": { "ignoreUnknown": false, - "includes": [ - "**", - "!packages/ruby/vendor", - "!packages/csharp/runtime", - "!packages/java/java_fmt.runtime.mjs", - "!packages/kotlin/kotlin_fmt.runtime.mjs" - ] + "includes": ["**", "!packages/ruby/vendor", "!packages/csharp/runtime", "!packages/java/java_fmt.runtime.mjs"] }, "formatter": { "enabled": true, diff --git a/build/java_fmt_teavm/binaryen.sh b/build/java_fmt_teavm/binaryen.sh index 870aea8..1ed654d 100755 --- a/build/java_fmt_teavm/binaryen.sh +++ b/build/java_fmt_teavm/binaryen.sh @@ -1,9 +1,9 @@ #!/usr/bin/env bash # Runs wasm-opt over a TeaVM WasmGC module, in place of nothing. # -# Shared by both builds - build.sh for google-java-format, kotlin-probe/build.sh -# for ktfmt - because they emit the same shape of module from the same compiler -# and so want the same pass. Binaryen is fetched into ../toolchain alongside +# Called by build.sh for google-java-format. It was shared with a ktfmt build +# from the same compiler until @scalar/kotlin-fmt moved to ktrs, which is why +# it is a script of its own. Binaryen is fetched into ../toolchain alongside # TeaVM and the JDK, so nothing has to be installed first. # # This used to be skipped. The comment that replaced it said Binaryen rewrote @@ -18,7 +18,7 @@ # pin like any other. # # Optimises in place, deliberately. Every other consumer of the build tree - -# conformance.sh, kotlin-probe/format-all.ts, kotlin-probe/run.ts - resolves +# conformance.sh, corpus-check.ts - resolves # `classes.wasm` by name, so writing beside it would leave all of them checking # bytes the package no longer ships. There is one module in the tree and it is # the shipped one. diff --git a/build/java_fmt_teavm/build.sh b/build/java_fmt_teavm/build.sh index 7968935..b1775b9 100755 --- a/build/java_fmt_teavm/build.sh +++ b/build/java_fmt_teavm/build.sh @@ -22,8 +22,8 @@ cd "$(dirname "$0")" GJF_VERSION="${GJF_VERSION:-1.36.1}" -# The same TeaVM the Kotlin package uses: upstream at a pinned commit plus the -# patches and shims in ../teavm. See ../teavm/README.md. +# Upstream TeaVM at a pinned commit plus the patches and shims in ../teavm. +# See ../teavm/README.md. TEAVM_VERSION="${TEAVM_VERSION:-$(cat ../teavm/TEAVM_VERSION)}" TOOLCHAIN="$PWD/toolchain" diff --git a/build/java_fmt_teavm/kotlin-probe/Errors.java b/build/java_fmt_teavm/kotlin-probe/Errors.java deleted file mode 100644 index 09437f1..0000000 --- a/build/java_fmt_teavm/kotlin-probe/Errors.java +++ /dev/null @@ -1,26 +0,0 @@ -import com.facebook.ktfmt.format.Formatter; - -/** Parse-error diagnostics are part of the contract, and the corpus never hits them. */ -public class Errors { - public static void main(String[] a) { - String[] broken = { - "fun main() {", - "class A { val x: = 1 }", - "fun f(: Int) {}", - "val x = \"unterminated", - "@@@", - "class", - "fun main() { val x = 1 +++ }", - "object O { fun f() { if (true }", - }; - for (String source : broken) { - String result; - try { - result = "OK:" + Formatter.format(source); - } catch (Throwable t) { - result = t.getClass().getName() + ": " + t.getMessage(); - } - System.out.println(result.replace("\n", "\\n")); - } - } -} diff --git a/build/java_fmt_teavm/kotlin-probe/FormatAll.java b/build/java_fmt_teavm/kotlin-probe/FormatAll.java deleted file mode 100644 index 9d1dedd..0000000 --- a/build/java_fmt_teavm/kotlin-probe/FormatAll.java +++ /dev/null @@ -1,49 +0,0 @@ -import com.facebook.ktfmt.format.Formatter; -import com.facebook.ktfmt.format.FormattingOptions; -import java.io.IOException; -import java.nio.charset.StandardCharsets; -import java.nio.file.*; -import java.util.*; -import java.util.stream.Stream; - -/** - * Formats every .kt file under a corpus directory and writes the result - or the - * error text - into an output directory, so two ktfmt jars can be compared file - * by file. One JVM for the whole corpus: ktfmt's Parser holds a single - * environment, and starting a JVM per file would dwarf the formatting. - */ -public class FormatAll { - public static void main(String[] args) throws IOException { - Path corpus = Paths.get(args[0]); - Path out = Paths.get(args[1]); - FormattingOptions options = switch (args[2]) { - case "google" -> Formatter.GOOGLE_FORMAT; - case "kotlinlang" -> Formatter.KOTLINLANG_FORMAT; - default -> Formatter.META_FORMAT; - }; - - List files; - try (Stream walk = Files.walk(corpus)) { - files = walk.filter(p -> p.toString().endsWith(".kt")).sorted().toList(); - } - - int ok = 0, failed = 0; - for (Path file : files) { - String source = Files.readString(file, StandardCharsets.UTF_8); - String result; - try { - result = "O" + Formatter.format(options, source); - ok++; - } catch (Throwable t) { - // The message is part of the comparison: a parser swap that changed a - // diagnostic would be a divergence too. - result = "E" + t.getClass().getName() + ": " + t.getMessage(); - failed++; - } - Path target = out.resolve(corpus.relativize(file)); - Files.createDirectories(target.getParent()); - Files.writeString(target, result, StandardCharsets.UTF_8); - } - System.out.println(files.size() + " files, " + ok + " formatted, " + failed + " errored"); - } -} diff --git a/build/java_fmt_teavm/kotlin-probe/README.md b/build/java_fmt_teavm/kotlin-probe/README.md deleted file mode 100644 index c72d36f..0000000 --- a/build/java_fmt_teavm/kotlin-probe/README.md +++ /dev/null @@ -1,430 +0,0 @@ -# Kotlin feasibility - -Whether ktfmt can become a package the way google-java-format did. **It did** — -this directory now builds [`@scalar/kotlin-fmt`](../../../packages/kotlin), whose -output is byte-identical to ktfmt on a JVM over 589 real Kotlin files in three -styles. The reason it could not moved six times before that, so what follows is -measurements rather than opinions, kept runnable so they can be re-measured when -a version moves. - -The name is historical: this was a feasibility probe and is now also the package -build (`./build.sh`). It sits under `java_fmt_teavm/` because it started on top -of the Java build; the only thing it still uses from there is -`../patches/ktfmt.patch`, so moving it is now just a rename. - -```bash -./stubs.sh # stubs and the build-time TeaVM policies -./ktfmt.sh # ktfmt, patched, into the local Maven repo -./gate2.sh # gate 2: does the parser swap change ktfmt's output? -mvn -f pom-ktfmt.xml package # ktfmt, to wasm -./conformance.sh # gate 3: does compiling to wasm change it? - -mvn -f pom.xml package # the minimal parse, kept as the smaller probe -node teavm-bug/check.ts target-forkthree/wasm/classes.wasm # does it validate? -node run.ts target-forkthree/wasm # does it run? -``` - -`./build.sh` does all of the above and writes the package artifact; the steps are -listed separately because each is a measurement worth being able to run on its -own. It builds TeaVM through `../../teavm/apply.sh`: upstream at a pinned commit, -a patch series, and a jar of `java.*` shims, the same TeaVM the Java package -uses. `../../teavm/README.md` explains which part is which, and which are on -their way upstream. - -Two flags are worth knowing when something goes wrong: `-Dteavm.minifying=false` -puts a name section in the module, without which a trap reports bare function -indices, and `-Dteavm.debugInformationGenerated=true` writes the `.teadbg` that -`run.ts` feeds to the deobfuscator so frames come back as Java method names. -Neither is on by default and diagnosing without them is slow. - -## Where the blocker moved - -**First read — wrong.** ktfmt cannot compile because the IntelliJ platform -resolves extension points reflectively from an XML descriptor. True of ktfmt as -shipped; not true of parsing. - -**Second read — right, and Gate 2 proved it.** The extension machinery belongs to -`KotlinCoreEnvironment`, the compiler's CLI environment. `patches/ktfmt.patch` -swaps `Parser` onto the bare container the PSI actually needs, and `gate2.sh` -shows that changes no output: 589 files from kotlin-stdlib, kotlinx-coroutines -and ktfmt itself, three styles, **1767/1767 byte-identical**, diagnostics -included, with 542 of the files actually reformatted so the comparison means -something. - -**Third read — fixed.** The class-library gap is closable, the size is fine, and -TeaVM's WasmGC backend emitted a module no engine would load. Three backend -bugs, now fixed on a fork. - -**Fourth read — fixed.** The module validated but trapped at load, before any -export was reachable. One `Thread.yield()` in kotlinx-coroutines was making a -third of the program a coroutine. See Gate 1b. - -**Fifth read — cleared.** The module loads, runs, and parses. Nine more gaps -closed on the way, one of them a fifth TeaVM bug. - -**Sixth read — cleared, and it was a miscompile.** ktfmt itself compiled with no -missing members at all, then failed on its first import list because an -optimisation pass deleted a class initializer it should have kept. See Gate 3. - -**Where it now sits.** Gate 3 is green: 1767/1767 byte-identical, and the package -is published from here by `./build.sh`. - -## Gate 1 — the numbers - -The frontier converged, and it never grew: - -| Round | Distinct missing members | -|---|---| -| ktfmt as shipped | 94 | -| minimal parse, before any work | 40 | -| after the concurrency and stub round | 18 | -| after `MethodHandles`, `Thread.State`, `Locale.forLanguageTag` | 16 | -| after the polymorphic-signature descriptors | 2 | -| after stubbing `PathManager` | 0 | -| after the four interfaces the vtable builder needs | **0, module emitted** | - -And it is small — this was the other way the idea could have died: - -| | Classes | Methods | Raw | Brotli | -|---|---|---|---|---| -| Kotlin minimal parse | 3,553 | 20,513 | 3.33 MB | **0.79 MB** | -| before the Gate 1b fix | 2,846 | 20,513 | 3.66 MB | 0.87 MB | -| `@scalar/java-fmt`, for scale | 3,313 | 24,503 | 3.32 MB | 0.84 MB | - -A third of that module was the coroutine transformation, which is why the Gate 1b -fix took 37% off it. - -Fewer classes than the Java formatter, and 45× under the 10 MB brotli threshold -this gate was supposed to kill it at. The 130 MB figure from JetBrains' -native-image `kotlinc` does not apply: that builds the whole compiler, this -builds a parser. - -## Gate 1 — cleared - -`WebAssembly.compile` used to reject the module. It no longer does: - -```bash -mvn -f pom.xml package -Dteavm.version=0.16.0-forkthree \ - -DbuildDirectory=$PWD/target-forkthree -node teavm-bug/check.ts target-forkthree/wasm/classes.wasm -RESULT: module validates — not reproduced -``` - -That was three bugs in TeaVM's WasmGC backend, all the same shape — a place -recording a type from a value where wasm takes it from a declaration, surfacing -only through the coroutine transformation, which is the one consumer that turns -a recorded stack into a block signature. `teavm-bug/` is the reproducer that -found them and the evidence for each. - -**They are master-only.** 0.14.1 has no `processOptimizedConditional`, and its -`WasmTypeInference.visit(WasmTeeLocal)` pushes no type at all, so the fixes do -not port back. That forced a master-based TeaVM here while `@scalar/java-fmt` -was still on 0.14.1 — two toolchains for a while. Java has since moved to the -same TeaVM and re-cleared `../conformance.sh` at 658/658, so there is one. All -three fixes have since landed upstream, in reworked form, so they are not in -`../../teavm/patches` at all. - -## Gate 1b — cleared - -Loading the module used to throw before `parse()` was ever called: - -``` -RuntimeError: dereferencing a null pointer - at wasm-function[47] <- wasm-function[183] <- wasm-function[14760] -``` - -Rebuilt with `minifying=false` and `debugInformationGenerated=true`, those -indices are: - -``` -at org.teavm.runtime.Fiber::isResuming -at org.teavm.jso.impl.wasmgc.WasmGCJSRuntime::stringToJs -at teavm@initializer -``` - -`teavm@initializer` is the wasm *start* function — it runs at instantiation, and -part of what it does is convert the `@JSExport` names to JS strings. It was -calling a `stringToJs` that TeaVM had coroutine-transformed, and the prologue the -transformation emits is `Fiber.current()` then `isResuming()` on the result. -`Fiber.current` is null until a fiber starts, so the start function trapped on -its own first call. - -So the question was why `stringToJs`, of all methods, was a coroutine. TeaVM's -`AsyncMethodFinder` marks every `@Async` method async and then propagates to -callers transitively. Dumping the propagation gave a 139-hop chain and one root: - -``` -asyncMethods=7060 -root 7055 java.lang.Thread.switchContext -root 3 java.lang.Object.monitorEnterWait -root 2 java.lang.Thread.sleep -``` - -7,055 of 20,513 methods, from a single `Thread.yield()` inside -`kotlinx.coroutines.scheduling.WorkQueue.addLast` — code this program never runs. -The chain gets from there to `stringToJs` through -`String.charAt` → `StringIndexOutOfBoundsException.` → -`Throwable.fillInStackTrace` → `Class.getName` → `StringBuilder.append(Object)` → -every `toString()` in the program. Exception construction and virtual `toString` -are edges the points-to analysis has to keep, and they make the call graph dense -enough that *any* reachable `@Async` method reaches most of it. - -That last point is the one worth keeping. Making `yield` a no-op moved the whole -7,055 to `Thread.sleep` — same count, next root. Cutting roots one at a time does -nothing; the fix has to leave no reachable `@Async` method at all. - -This used to be done in TeaVM itself, by making `yield`, `sleep` and `wait` -non-async for every program. That broke upstream's own thread tests, so it is -gone. Upstream has since made async opt-in (`3d8ed09`): with the `asyncEnabled` -option off, which is the default and how this builds, nothing is -coroutine-transformed at all, and a reachable `@Async` method is a compile error -rather than a third of the program. - -Three calls reach one, and none is on a path formatting takes: `Thread.yield` in -kotlinx-coroutines' `WorkQueue.addLast` and in IntelliJ's -`ConcurrentIntObjectHashMap.initTable`, both spinning on a race another thread -would have to be in, and `Thread.sleep` in IntelliJ's `TimeoutUtil`, which only -`PsiUtilCore.ensureValid` calls once an element is already invalid. -`plugin/kprobe/SingleThreadedCalls.java`, a TeaVM `ClassHolderTransformer` -named in `pom-ktfmt.xml`, deletes exactly those three calls and no others, so a -new one after a ktfmt or kotlinx-coroutines bump fails the build instead of -disappearing. Nothing is coroutine-transformed, the start function cannot trap, -and the module stays the size it was. - -### There is a TeaVM bug underneath this, still unfixed - -Cutting the roots is right for this program, but it is a workaround for -something real: **the wasm start function can call a coroutine-transformed -function, whose prologue dereferences a fiber that does not exist yet.** Any -TeaVM WasmGC program that legitimately needs threads and exports anything via -`@JSExport` will trap at load the same way. The same hazard applies to the -exports themselves — a `@JSExport` method that lands in the async closure is -called from JS with no fiber either. - -It has no fix here, only an avoidance, and it has not been reported upstream. -Upstream's `asyncEnabled` default makes it unreachable for any program that -leaves async off, which narrows it to programs that turn async on and export -something. - -## Also found: a crash that should be a diagnostic - -`WasmGCVirtualTableBuilder.addImplementorToInterface` dereferences the table for -every interface a reachable class declares, and a class can declare one the -analysis never reached. The result was -`NullPointerException: Cannot read field "commonImplementorFilled" because "itf" is null` -with no indication of which interface. The Kotlin class-library commit on the -fork turned that into a named report. - -**Already fixed upstream** — TeaVM master has a null check there now, so this -hunk should be dropped rather than sent. The four interfaces this program needed -(`RunnableScheduledFuture`, `ScheduledExecutorService`, -`javax.management.NotificationListener`, `javax.swing.Icon`) are supplied -properly anyway, which is the better answer either way: master's fix skips the -interface silently, and a table with a missing implementor is exactly the kind of -thing that shows up later as a module that will not validate. - -## Gate 3 — cleared: ktfmt in wasm matches ktfmt on a JVM - -```bash -./stubs.sh && ./ktfmt.sh -mvn -f pom-ktfmt.xml package -./conformance.sh -``` - -``` -corpus: 589 Kotlin files - -meta: 589/589 byte-identical -google: 589/589 byte-identical -kotlinlang: 589/589 byte-identical -``` - -**1767/1767.** Same corpus and the same three styles as `gate2.sh`, which asked -the other half of the question - whether the parser swap changes ktfmt's output -on a JVM - and also answered no. Both sides encode the result the same way, `O` -and the formatted source or `E` and the exception, so a difference in how a -failure is reported would count as a difference. Every file formatted; none -errored on either side. - -That is the measurement an **exact** row in the root README would rest on, and -this is the first version of this probe where the number exists at all. - -| | Classes | Raw | Brotli | -|---|---|---|---| -| ktfmt | 4,039 | 3.82 MB | **0.91 MB** | -| the minimal parse, for scale | 3,554 | 3.33 MB | 0.79 MB | -| `@scalar/java-fmt` | 3,313 | 3.32 MB | 0.84 MB | - -### A sixth TeaVM bug, and it was a miscompile - -Getting here needed one more fix, and it is the most serious of the six because -nothing about it is a missing feature - it is an optimisation that deletes -something it should not. - -`ClassInitElimination` walks the dominator tree deleting `InitClassInstruction`s -that a dominating one already covers, and treats **any** invocation as -initializing the class it names: - -```java -if (insn instanceof InvokeInstruction) { - InvokeInstruction invoke = (InvokeInstruction) insn; - step.initializedClasses.add(invoke.getMethod().getClassName()); -} -``` - -Only a static call does that. `ClassInitializerInsertionTransformer` puts the -initializer call at the top of static methods and constructors, not instance -methods, and an interface call initializes the receiver's class rather than the -interface. `ClassInitInsertion`, which puts these instructions in, guards on -`getInstance() == null`; the elimination pass did not. - -What it cost: `SharedImplUtil.getChildrenOfType` iterates the tree through -`ASTNode` instance calls and then returns `ASTNode.EMPTY_ARRAY` when the count is -zero. The instance calls marked `ASTNode` initialized, the initializer before the -field read was deleted, and the constant was read as null - so a `@NotNull` -method returned null and ktfmt failed on its first import list. - -It is worth being precise about how visible this was. The symptom was -`@NotNull method SharedImplUtil.getChildrenOfType must not return null`, five -frames from anything to do with class initialization, and reading -`ASTNode.EMPTY_ARRAY` from an exported method made the *next* call succeed - -which is what turned it from a guess into a diagnosis: - -``` -format first: @NotNull method ... getChildrenOfType must not return null -probe first: "fun main() {\n println(\"hi\")\n}\n" -``` - -A minimal reproducer does not reproduce it: the small case gets an eagerly -initialized class and never needs the instruction that was deleted. It needs a -class whose initializer TeaVM classifies as dynamic, an instance call on that -class, and a read of its static field afterwards, all in one method after -inlining. That is why the fix is checked against the corpus rather than a unit -test - and why it is the one of the six that most deserves upstream attention. - -## Gate 1c — cleared: it parses - -`node run.ts target-forkthree/wasm` now returns -`org.jetbrains.kotlin.psi.KtFile:31`. Getting there was nine failures, each one -step further into `CoreApplicationEnvironment` and then into the PSI, and every -one of them a different kind of thing: - -| Failed at | Was | Fixed by | -|---|---|---| -| `SystemInfoRt.` | `toLowerCase` on an undefined `os.version` | `os.version`, `os.arch` defined | -| `ReflectionUtil.` | `Class.forName("sun.misc.Unsafe")`, `theUnsafe` read reflectively | `stubs/sun/misc/Unsafe.java` + policy | -| `Proxy.newProxyInstance` | listener interfaces not registered proxyable | `proxyable()` in the policy | -| `Proxy.` | `NoSuchMethodException` on the interface's `` | a TeaVM fix — see below | -| `ConcurrentLongObjectHashMap.` | `objectFieldOffset` on its own fields | the container replaced | -| `KotlinElementTypeProvider` | impl looked up by name, `INSTANCE` read reflectively | policy | -| `KtStubElementType.` | PSI classes' `(ASTNode)` constructors not reflectable | policy | -| `MessageBusImpl.syncPublisher` | topic listener interfaces not proxyable | policy | -| `CharsetToolkit.` | `forName` on UTF-32BE, UTF-32LE, windows-1251 | three charsets implemented | -| `LanguageSubstitutors` | DI container found no constructor | policy | -| `Registry` | `misc/registry.properties` not in the module | a resources policy | - -Four of those are the same underlying thing: **TeaVM resolves reflection -statically**, so `Class.forName`, `getDeclaredField`, `getConstructor` and -`Proxy` all need telling at build time. That is what `plugin/` is — a build-time -extension, never compiled to wasm. Each rule is scoped by the mechanism that -needs it rather than by a list of what is reached today: - -- `KeyedExtensionCollector` subclasses, because those are what the DI container - constructs; -- interfaces extending `EventListener` **or** named `*Listener`, because - `EventDispatcher` goes by the first and `MessageBus`'s listener is a `Topic` - type parameter that no build-time rule can see, and IntelliJ names them all - the second way; -- `KtElement` subclasses, because that is the bound both element-type - declarations use. - -Scoping matters more than it sounds: a reflectable member is a dependency root, -so the first attempt at the DI rule — the whole `com.intellij` package — made -every class in it reachable, dragged in the XML stream readers and -kotlinx-serialization, and failed to compile at all. - -### A fifth TeaVM bug, and this one is fixed - -`ProxyDependencySupport.generateWorkersForInterface` walks `itf.getMethods()` -skipping only bridges, so an interface that declares a constant — and therefore -has a `` — gets a proxy worker generated for it. The generated proxy's -initializer then calls `getDeclaredMethod("", ...)`, which can only -throw: `` is not a declared method. Any interface with a constant field -is unproxyable, which in IntelliJ is most listener interfaces, since they carry a -`Topic`. Skipping static methods fixes it, and matches what `Proxy` dispatches -anyway. `upstream/proxy-and-reflection` on the fork. - -Finding it needed two more changes, kept in the same patch: `getDeclaredMethod` -and `getDeclaredField` threw unnamed exceptions, which out of a generated -initializer say nothing at all. They name the member now, and that is what turned -each of the reflection failures above from a bare stack trace into a one-line -diagnosis. - -### The Unsafe containers were on the parse path - -`stubs/…/containers/Unsafe.java` throws on every operation, deliberately, and -said the open question was whether the parse path reaches the containers built on -it. It does — `CoreProgressManager` makes a `ConcurrentLongObjectHashMap`, which -takes `objectFieldOffset` of four of its own fields in its static initializer and -does its updates by CAS on those offsets. - -That has no managed-heap equivalent, so the container is replaced rather than the -member added: a `HashMap` behind the same five-method interface, on the same -reasoning as the `java.util.concurrent` stand-ins. It was the only one the parse -path needed, which is the answer to how wide this went — narrower than it looked. - -### Two things that are not reflection - -**Three charsets.** `CharsetToolkit`'s initializer asks `Charset.forName` for -UTF-32BE, UTF-32LE and windows-1251, and TeaVM had none of them. -`upstream/charsets` on the fork implements them properly rather than -registering empty objects, since a charset that exists but decodes wrongly is the -worse failure. It also fixes the lookup: `forName` uppercases the name it is -given but the registry was keyed by canonical name, which agreed by accident -until `windows-1251` — the first charset whose canonical name is not uppercase. -No longer carried: formatting as it is built now never initializes -`CharsetToolkit`, so the charsets were dropped from `../../teavm/patches` -(`../../teavm/README.md`, "Which patch each formatter needs"). - -**One resource.** `Registry` reads `misc/registry.properties` off the classpath -and throws `MissingResourceException` for a key it cannot find, with no default, -and `PsiManager` asks for `psi.sleep.in.validity.check` on first PSI access. -Resources are not on a classpath at run time, so a `ResourcesPolicy` names it and -TeaVM embeds it. Worth recording why it is a resource policy and not the -`META-INF/services/java.util.ResourceBundle` route the Java build uses for -javac's messages: `Registry` reads the file as a stream rather than as a bundle, -and TeaVM's `.properties`-to-bundle path rebuilds the bundle one entry at a time -through the metaprogramming API, which for 1,882 keys does not finish. - -## What is left, in order - -1. **Decide whether it is worth carrying.** A patched ktfmt and a patched TeaVM - for a second language. A judgement call rather than a measurement, and Gate - 3's number is now in hand to make it with. Of the six TeaVM fixes, three are - ordinary upstreamable bugs, one is a miscompile anyone using WasmGC could hit, - and only the single-threaded stand-ins are deliberate divergences that would - have to be carried indefinitely. - -### One loose end - -Four `UnsupportedOperationException: LockSupport.park would block the only -thread` are printed to stderr while the module starts. Four for the whole -589-file run, not four per file, and the output is byte-identical either way — so -it is start-up noise from a stand-in rather than a failure. A published package -would want it silenced; the probe leaves it visible. - -## What the patches contain - -- `../patches/ktfmt.patch` — the parser swap. Proven neutral by `gate2.sh`. -- **The TeaVM changes** are a patch series and a shims jar in `../../teavm`, - applied to a pinned upstream commit. `../../teavm/HUNKS.md` maps where each - change from the old fork went. -- `plugin/kprobe/SingleThreadedCalls.java` — the three deleted calls above. - -`MethodHandle.invoke` and `invokeExact` are worth calling out. They are -polymorphic-signature methods: the caller writes whatever descriptor it likes. -The descriptors this program emits are spelled out one by one in -`TMethodHandle`, which worked here but is whack-a-mole — two of them differ only -in return type, which Java cannot express, and that is what forced the -`PathManager` stub. A general fix means teaching the compiler to treat those two -names specially. diff --git a/build/java_fmt_teavm/kotlin-probe/build.sh b/build/java_fmt_teavm/kotlin-probe/build.sh deleted file mode 100755 index 00a6ace..0000000 --- a/build/java_fmt_teavm/kotlin-probe/build.sh +++ /dev/null @@ -1,98 +0,0 @@ -#!/usr/bin/env bash -# Compiles ktfmt to the wasm artifact shipped by @scalar/kotlin-fmt. -# -# Requires nothing preinstalled but a JDK 21, Maven, git and Node: TeaVM, ktfmt -# and the repo's pinned Node are fetched into ../toolchain, which is gitignored. -# Expect ~15 minutes on a cold run, most of it compiling TeaVM. -# -# The artifact is committed, so this only needs rerunning when a pin below or -# ../patches/ktfmt.patch changes. Commit the result: the bytes in git are the -# bytes the tests run against. -# -# On the directory this lives in: it is called kotlin-probe and sits under -# java_fmt_teavm for historical reasons - it started as a feasibility probe on -# top of the Java build. It no longer depends on anything in that directory -# except ../patches/ktfmt.patch, so moving it is now only a rename. -set -euo pipefail -cd "$(dirname "$0")" - -# Upstream TeaVM at a pinned commit, plus the patches and shims in ../../teavm. -# See ../../teavm/README.md for what they are and which are upstreamable. -TEAVM_VERSION="${TEAVM_VERSION:-$(cat ../../teavm/TEAVM_VERSION)}" -NODE_VERSION="${NODE_VERSION:-26.7.0}" - -TOOLCHAIN="$PWD/../toolchain" -OUT_DIR="../../../packages/kotlin" -mkdir -p "$TOOLCHAIN" - -JAVA_HOME="${JAVA_HOME:-$(dirname "$(dirname "$(readlink -f "$(command -v java)")")")}" -export JAVA_HOME -# Grep for the banner line rather than taking the first: a JAVA_TOOL_OPTIONS in -# the environment prints ahead of it. -java_version="$("$JAVA_HOME/bin/java" -version 2>&1 | grep -m1 ' version "' | sed 's/.*version "\([0-9]*\).*/\1/')" -if [ "$java_version" != "21" ]; then - echo "need JDK 21 (found ${java_version:-unknown} at $JAVA_HOME); set JAVA_HOME" >&2 - exit 1 -fi - -# ---------------------------------------------------------------------------- -# 1. Node, to run the module. -# -# WasmGC with the js-string builtins needs 24.15 at the least, which is the -# floor the published package enforces - see packages/kotlin/src/boot-module.ts. -# What is pinned here is the repo's toolchain version, which sits above that -# floor; the floor itself is what CI's Node smoke test runs against, not this. -# ---------------------------------------------------------------------------- -NODE="$TOOLCHAIN/node-v$NODE_VERSION-linux-x64/bin/node" -if [ ! -x "$NODE" ]; then - echo "fetching Node $NODE_VERSION" - curl -fsSL -o "$TOOLCHAIN/node.tar.xz" \ - "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.xz" - tar xf "$TOOLCHAIN/node.tar.xz" -C "$TOOLCHAIN" - rm -f "$TOOLCHAIN/node.tar.xz" -fi - -# ---------------------------------------------------------------------------- -# 2. TeaVM: upstream konsoletyper/teavm, patched, plus the java.* shims. -# -# The same TeaVM the Java package builds with. apply.sh skips the build when -# neither the pin nor the patches nor the shims have changed. -# ---------------------------------------------------------------------------- -TEAVM_VERSION="$TEAVM_VERSION" ../../teavm/apply.sh - -# ---------------------------------------------------------------------------- -# 3. The classpath stubs, the build-time policies, and ktfmt itself. -# ---------------------------------------------------------------------------- -TEAVM_VERSION="$TEAVM_VERSION" ./stubs.sh -./ktfmt.sh - -# ---------------------------------------------------------------------------- -# 4. ktfmt to wasm, then compressed into the package. -# -# Brotli because 3.8MB of WasmGC packs to under 1MB, and node:zlib decompresses -# it once per process - so the saving costs an install nothing and adds no -# dependency. -# ---------------------------------------------------------------------------- -rm -rf target-ktfmt -mvn -B -q -f pom-ktfmt.xml -Dteavm.version="$TEAVM_VERSION" package - -mkdir -p "$OUT_DIR" -cp target-ktfmt/wasm/classes.wasm-runtime.js "$OUT_DIR/kotlin_fmt.runtime.mjs" - -# wasm-opt, over TeaVM's output. See ../binaryen.sh for why this was skipped -# until now: the module Binaryen emits was always spec-valid and V8 has stopped -# rejecting it. Worth about 10% off the shipped artifact here, and -# ./conformance.sh is what proves it changes no formatting. -TOOLCHAIN="$TOOLCHAIN" ../binaryen.sh target-ktfmt/wasm/classes.wasm - -"$NODE" -e " -const fs = require('node:fs'), zlib = require('node:zlib') -const raw = fs.readFileSync('target-ktfmt/wasm/classes.wasm') -const packed = zlib.brotliCompressSync(raw, { - params: { [zlib.constants.BROTLI_PARAM_QUALITY]: 11, [zlib.constants.BROTLI_PARAM_SIZE_HINT]: raw.length }, -}) -fs.writeFileSync('$OUT_DIR/kotlin_fmt.wasm.br', packed) -console.log((raw.length / 1048576).toFixed(2) + ' MB raw -> ' + (packed.length / 1048576).toFixed(2) + ' MB brotli') -" - -echo "wrote $OUT_DIR/kotlin_fmt.wasm.br and kotlin_fmt.runtime.mjs" diff --git a/build/java_fmt_teavm/kotlin-probe/conformance.sh b/build/java_fmt_teavm/kotlin-probe/conformance.sh deleted file mode 100755 index 2c2d2d1..0000000 --- a/build/java_fmt_teavm/kotlin-probe/conformance.sh +++ /dev/null @@ -1,57 +0,0 @@ -#!/usr/bin/env bash -# Gate 3: is the wasm module's output the same as ktfmt's own, byte for byte? -# -# gate2.sh asked whether the parser swap changes ktfmt's output, on a JVM, and -# answered no. This asks the other half: whether compiling that same patched -# ktfmt to wasm changes it. Same corpus, same three styles, same encoding on both -# sides - "O" and the formatted source, or "E" and the exception - so a -# divergence in a diagnostic counts as a divergence. -# -# This is the measurement the exactness claim in the root README would rest on. -# It needs ./stubs.sh, ./ktfmt.sh and a built module; run gate2.sh first if the -# corpus is not assembled yet. -set -euo pipefail -cd "$(dirname "$0")" - -WASM_DIR="${WASM_DIR:-$PWD/target-ktfmt/wasm}" -WORK="$PWD/work" -CORPUS="${CORPUS:-$WORK/corpus}" -NODE="${NODE:-$PWD/../toolchain/node-v26.7.0-linux-x64/bin/node}" -[ -x "$NODE" ] || NODE=node - -if [ ! -d "$CORPUS" ]; then - echo "no corpus at $CORPUS - run ./gate2.sh first" >&2 - exit 1 -fi -if [ ! -f "$WASM_DIR/classes.wasm" ]; then - echo "no module at $WASM_DIR - run mvn -f pom-ktfmt.xml package" >&2 - exit 1 -fi - -KOTLIN_VERSION="${KOTLIN_VERSION:-2.3.20}" -RUNTIME_CP="$WORK/gjf.jar:$WORK/guava.jar:$WORK/kotlin-stdlib.jar:$WORK/kotlin-compiler-embeddable.jar" -RUNTIME_CP="$RUNTIME_CP:$WORK/coroutines.jar:$WORK/annotations.jar:$WORK/kotlin-reflect.jar" -RUNTIME_CP="$RUNTIME_CP:$WORK/kotlin-script-runtime.jar:$WORK/kotlin-daemon-embeddable.jar" - -javac -nowarn -cp "$WORK/ktfmt-patched.jar:$WORK/gjf.jar" -d "$WORK" FormatAll.java Errors.java - -total=$(find "$CORPUS" -name '*.kt' | wc -l | tr -d ' ') -echo "corpus: $total Kotlin files" -echo - -failures=0 -for style in meta google kotlinlang; do - rm -rf "$WORK/conf-jvm-$style" "$WORK/conf-wasm-$style" - java -cp "$WORK:$WORK/ktfmt-patched.jar:$RUNTIME_CP" \ - FormatAll "$CORPUS" "$WORK/conf-jvm-$style" "$style" > /dev/null - "$NODE" format-all.ts "$WASM_DIR" "$CORPUS" "$WORK/conf-wasm-$style" "$style" > /dev/null - - if diff -rq "$WORK/conf-jvm-$style" "$WORK/conf-wasm-$style" > "$WORK/conf-diff-$style.txt"; then - echo "$style: $total/$total byte-identical" - else - echo "$style: $(grep -c . "$WORK/conf-diff-$style.txt") differ, see $WORK/conf-diff-$style.txt" >&2 - failures=1 - fi -done - -exit $failures diff --git a/build/java_fmt_teavm/kotlin-probe/format-all.ts b/build/java_fmt_teavm/kotlin-probe/format-all.ts deleted file mode 100644 index 9bf525f..0000000 --- a/build/java_fmt_teavm/kotlin-probe/format-all.ts +++ /dev/null @@ -1,55 +0,0 @@ -// node format-all.ts