Implement full native tokens plugin, use common code

[marcins]

Motivation

The bulk of the @atlaspack/transformer-tokens plugin is native code, with a thin-ish JS wrapper to make it work as a JS plugin. For native asset graph we don't need this JS wrapper, it just adds unnecessary overhead, so we should instead just make it a fully native "plugin" like some of the other code plugins already in Atlaspack.

Changes

• extract shared logic to atlaspack_atlaskit_tokens crate
• update plugin_tokens in Node bindings to use the shared crate
• create atlaspack_plugin_transformer_tokens crate to use shared crate with a native implementation

Checklist

• Existing or new tests cover this change
• There is a changeset for this change, or one is not required
• Added documentation for any new features to the docs/ folder

[changeset-bot]

🦋 Changeset detected

Latest commit: d2275be

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 110 packages

Name	Type
@atlaspack/rust	Minor
@atlaspack/bundler-experimental	Patch
@atlaspack/bundler-default	Patch
atlaspack	Patch
@atlaspack/cache	Patch
@atlaspack/core	Patch
@atlaspack/fs	Patch
@atlaspack/logger	Patch
@atlaspack/source-map	Patch
@atlaspack/utils	Patch
@atlaspack/link	Patch
@atlaspack/optimizer-image	Patch
@atlaspack/optimizer-inline-requires	Patch
@atlaspack/packager-js	Patch
@atlaspack/transformer-compiled-css-in-js	Patch
@atlaspack/transformer-html	Patch
@atlaspack/transformer-js	Patch
@atlaspack/transformer-postcss	Patch
@atlaspack/transformer-svg	Patch
@atlaspack/transformer-tokens	Patch
@atlaspack/node-resolver-core	Patch
@atlaspack/config-default	Patch
@atlaspack/test-utils	Patch
@atlaspack/inspector	Patch
@atlaspack/query	Patch
@atlaspack/config-webextension	Patch
@atlaspack/cli	Patch
@atlaspack/register	Patch
@atlaspack/bundle-stats	Patch
@atlaspack/reporter-bundle-stats	Patch
@atlaspack/transformer-image	Patch
@atlaspack/inspector-frontend	Patch
@atlaspack/package-manager	Patch
@atlaspack/profiler	Patch
@atlaspack/workers	Patch
@atlaspack/watcher-watchman-js	Patch
@atlaspack/types-internal	Patch
@atlaspack/optimizer-css	Patch
@atlaspack/optimizer-cssnano	Patch
@atlaspack/optimizer-swc	Patch
@atlaspack/optimizer-terser	Patch
@atlaspack/packager-css	Patch
@atlaspack/transformer-babel	Patch
@atlaspack/transformer-css	Patch
@atlaspack/transformer-less	Patch
@atlaspack/transformer-sass	Patch
@atlaspack/transformer-typescript-tsc	Patch
@atlaspack/transformer-typescript-types	Patch
@atlaspack/optimizer-blob-url	Patch
@atlaspack/optimizer-data-url	Patch
@atlaspack/optimizer-svgo	Patch
@atlaspack/packager-html	Patch
@atlaspack/packager-raw-url	Patch
@atlaspack/packager-svg	Patch
@atlaspack/packager-webextension	Patch
@atlaspack/packager-xml	Patch
@atlaspack/reporter-build-metrics	Patch
@atlaspack/reporter-bundle-analyzer	Patch
@atlaspack/reporter-cli	Patch
@atlaspack/reporter-dev-server-sw	Patch
@atlaspack/reporter-dev-server	Patch
@atlaspack/reporter-json	Patch
@atlaspack/reporter-lsp	Patch
@atlaspack/reporter-sourcemap-visualiser	Patch
@atlaspack/reporter-tracer	Patch
@atlaspack/resolver-glob	Patch
@atlaspack/runtime-browser-hmr	Patch
@atlaspack/runtime-js	Patch
@atlaspack/runtime-react-refresh	Patch
@atlaspack/runtime-service-worker	Patch
@atlaspack/runtime-webextension	Patch
@atlaspack/transformer-posthtml	Patch
@atlaspack/transformer-react-refresh-wrap	Patch
@atlaspack/transformer-webextension	Patch
@atlaspack/transformer-webmanifest	Patch
@atlaspack/validator-eslint	Patch
@atlaspack/validator-typescript	Patch
@atlaspack/resolver-default	Patch
@atlaspack/resolver-tesseract	Patch
@atlaspack/types	Patch
@atlaspack/codeframe	Patch
@atlaspack/graph	Patch
@atlaspack/plugin	Patch
@atlaspack/reporter-bundle-buddy	Patch
@atlaspack/transformer-xml	Patch
@atlaspack/ts-utils	Patch
@atlaspack/packager-ts	Patch
@atlaspack/transformer-jsonld	Patch
@atlaspack/bundler-library	Patch
@atlaspack/compressor-brotli	Patch
@atlaspack/compressor-gzip	Patch
@atlaspack/compressor-raw	Patch
@atlaspack/namer-default	Patch
@atlaspack/optimizer-htmlnano	Patch
@atlaspack/packager-raw	Patch
@atlaspack/packager-wasm	Patch
@atlaspack/reporter-compiled-css-in-js-migration-map	Patch
@atlaspack/reporter-conditional-manifest	Patch
@atlaspack/transformer-glsl	Patch
@atlaspack/transformer-graphql	Patch
@atlaspack/transformer-inline-string	Patch
@atlaspack/transformer-inline	Patch
@atlaspack/transformer-json	Patch
@atlaspack/transformer-mdx	Patch
@atlaspack/transformer-pug	Patch
@atlaspack/transformer-raw	Patch
@atlaspack/transformer-svg-react	Patch
@atlaspack/transformer-toml	Patch
@atlaspack/transformer-worklet	Patch
@atlaspack/transformer-yaml	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

📊 Type Coverage Report

Coverage Comparison

Metric	Baseline	Current	Change
Coverage Percentage	92.09%	92.09%	➡️ 0.00%
Correctly Typed	203,748	203,748	+0
Total Expressions	221,242	221,242	+0
Untyped Expressions	17,494	17,494	+0

Files with Most Type Issues (Top 15)

File	Issues	Affected Lines
packages/core/integration-tests/test/javascript.ts	999	658
packages/core/integration-tests/test/cache.ts	885	626
packages/core/integration-tests/test/scope-hoisting.ts	623	490
packages/utils/node-resolver-core/test/resolver.ts	476	177
packages/core/integration-tests/test/html.ts	468	294
packages/core/integration-tests/test/sourcemaps.ts	356	176
packages/core/test-utils/src/utils.ts	330	205
packages/core/integration-tests/test/incremental-bundling.ts	298	206
packages/core/core/src/dumpGraphToGraphViz.ts	251	108
packages/core/integration-tests/test/output-formats.ts	227	161
packages/transformers/webextension/src/WebExtensionTransformer.ts	210	80
packages/core/integration-tests/test/css-modules.ts	191	107
packages/core/core/src/requests/TargetRequest.ts	190	133
packages/core/integration-tests/test/react-refresh.ts	188	65
packages/core/integration-tests/test/babel.ts	187	115

---

This report was generated by the Type Coverage GitHub Action

[github-actions]

📊 Benchmark Results

⚠️ Performance regressions detected!

📊 Benchmark Results

Overall Performance

Test	Duration	JS Memory Peak	Native Memory Peak	vs Baseline	Status
Three.js Real Repository (JS)	18.05s	2.08GB	2.30GB	+6.73% duration, -3.04% JS memory	🔴 Regression
Three.js Real Repository (V3)	34.78s	3.64GB	4.13GB	+38.88% duration, +2.86% JS memory	🔴 Regression

🔍 Detailed Phase Analysis

Three.js Real Repository (JS)

Phase	Duration (avg)	Duration (p95)	Memory Peak (avg)	Memory Peak (p95)
resolving	9.38s	9.88s	1.45GB	1.73GB
transforming	9.37s	9.88s	1.45GB	1.73GB
bundling	9.15s	9.67s	1.45GB	1.74GB
bundled	8.73s	9.25s	1.55GB	1.83GB
packaging	6.27s	6.68s	1.66GB	1.96GB
optimizing	6.09s	6.48s	2.08GB	2.29GB

Three.js Real Repository (V3)

Phase	Duration (avg)	Duration (p95)	Memory Peak (avg)	Memory Peak (p95)
bundling	22.17s	30.56s	2.92GB	3.37GB
bundled	21.70s	30.12s	3.10GB	3.54GB
packaging	8.36s	8.58s	3.25GB	3.70GB
optimizing	8.21s	8.41s	3.64GB	4.13GB

💾 Unified Memory Analysis

Three.js Real Repository (JS) Memory Statistics

Memory Type	Metric	Min	Mean	Median	P95	P99	Max	Std Dev
JavaScript	RSS	1.34GB	1.88GB	1.87GB	2.29GB	2.29GB	2.29GB	291.67MB
	Heap Used	80.12MB	87.26MB	85.11MB	99.11MB	99.11MB	99.11MB	7.52MB
	Heap Total	93.85MB	130.89MB	133.89MB	181.91MB	181.91MB	181.91MB	30.09MB
	External	38.88MB	105.96MB	173.02MB	173.02MB	173.02MB	173.02MB	67.06MB
Native (Rust)	Physical Memory	1.66GB	1.87GB	1.89GB	2.11GB	2.24GB	2.30GB	146.09MB
	Virtual Memory	30.04GB	30.75GB	30.75GB	31.04GB	31.21GB	31.24GB	206.78MB

Sample Counts: JS: 14, Native: 294

Three.js Real Repository (V3) Memory Statistics

Memory Type	Metric	Min	Mean	Median	P95	P99	Max	Std Dev
JavaScript	RSS	2.59GB	3.40GB	3.40GB	4.13GB	4.13GB	4.13GB	429.39MB
	Heap Used	77.37MB	78.01MB	78.02MB	78.51MB	78.51MB	78.51MB	0.33MB
	Heap Total	84.95MB	89.32MB	88.20MB	100.70MB	100.70MB	100.70MB	4.54MB
	External	179.71MB	182.60MB	185.49MB	185.49MB	185.49MB	185.49MB	2.89MB
Native (Rust)	Physical Memory	3.21GB	3.51GB	3.57GB	3.88GB	4.03GB	4.13GB	235.61MB
	Virtual Memory	30.82GB	33.33GB	33.64GB	33.93GB	34.15GB	34.21GB	824.07MB

Sample Counts: JS: 14, Native: 701

🖥️ Environment

• Node.js: v22.17.0
• Platform: linux (x64)
• CPU: AMD EPYC 7763 64-Core Processor
• Total Memory: 15.62GB
• Git SHA: d2275be
• Branch: mszczepanski/full-native-tokens-transformer
• Timestamp: 2025-11-27T22:47:16.129Z

[marcins]

Hmm, looks like v3 macOS tests are reliably hanging after this change, will need to investigate further (see if I can repro locally)..

Worked eventually?