Optimise `pkg` bundling so that it is not bundling useless markdown files (combine with esbuild?)

[CMCDragonkai]

Specification

The pkg bundling appears to be bundling useless markdown files. There is a lack of an ignore configuration that we can apply to pkg so that useless markdown files are not packaged as well. The assets and scripts are "inclusive" configuration, we need "exclusive" configuration.

This should reduce our file size by quite a bit. On top of that, we should be applying minification to our JS that is being bundled.

One way to solve this, is to make use of a JS bundler first, before using pkg. See the problem is that pkg is a fairly basic bundler, focused on creating a usable executable. But it's not a good "module bundler", and there are better tools to do module bundling.

Suppose we were to combine esbuild (or webpack or rollup) with pkg, where esbuild runs first to package the executable into 1 JS file while doing minification. Then pkg only has to deal with that 1 file.

However we still need to consider things that actually cannot be bundled into that 1 JS file. These are all the assets like images, native addons, JSON configuration files, protobuf files. Furthermore there's also scripts. In the case of module bundling, some of these can be "redirected" like JSON. Others however must remain as actual files on the filesystem. This is where pkg can come in save the day. It just needs to maintain its virtual filesystem, and we just need to configure the assets and scripts appropriately.

So right now the scripts/pkg.js is just the beginning. Once a module bundler is involved, the generation of the pkg configuration JSON is likely to become more complex to deal with all these other assets.

Additional context

• WIP: Demonstrating Native Addons TypeScript-Demo-Lib#38 (comment) - initial detection of pkg bundling files like CHANGELOG.md
• https://github.com/vercel/ncc - vercel also works on the ncc project, can it also be used in place of esbuild?

Tasks

1. ...
2. ...
3. ...

[CMCDragonkai]

If we are going to integrate a bundler like esbuild or webpack, then this may not need to be done. Instead it will be the bundler's job to put together a proper bundle with everything required that would then be "bundled" again by pkg into an executable file.

[CMCDragonkai]

I'm also curious about pkg virtual filesystem, does it end up unpacking the virtual filesystem on a real filesystem in a temporary directory before running node?

The only other alternative would be to patch the require/import calls so that can load in-memory chunks from the executable and present them as files.

[CMCDragonkai]

The bundler can do things like minification and obfuscation in front of pkg. Most importantly it has to deal with all the possible assets. It would have to sit in front of tsc. The tsc will continue to produce a dist directory as this is required for npm publishing, the bundler would then use the dist directory and then trace all the require, static import, dynamic import, and dynamic read file calls.

It's important for us to keep track of all assets that may be read dynamically and configure them in a central place. This includes any JSON (tsc deals with this already?), protobuf, object files (for native addons), text files (like certificates)... etc.

Web apps also have CSS and images, but these are a separate concern.

[CMCDragonkai]

It would be good for the debugging process if the CI/CD process can produce a report/diagram for what will exist in the bundle and its layout structure. The bot can take that and produce nice visuals.

[CMCDragonkai]

The lack of support for ESM modules is also concerning. Right now we have a problem with pkg bundling the tslib, it doesn't really make any sense: MatrixAI/Polykey#379 (comment). So another reason why we need to have a purpose-built bundler before passing into pkg. If that is done, then pkg just ultimately handles non-JS assets (perhaps non-JSON assets too), and native addons and the virtual filesystem and the bundling of the node binary.

[tegefaulkes]

Issue relating to this is here.
MatrixAI/Polykey#387

[CMCDragonkai]

Trying out nextjs on Polykey-Enterprise as its a web application. Interestingly it doesn't use esbuild.

It has its own special bundler called Turbopack written in Rust. Funny how we end up writing these things in Rust to support TypeScript applications.

Esbuild may still be relevant for Polykey though - specifically for Polykey-CLI and Polykey-Desktop. Because this repo Polykey will focus on staying as a library, so we will move pkg and esbuild and all the other stuff out of Polykey repo. Instead Polykey-CLI will do the bundling into a CLI executable, while Polykey-Desktop will do other bundling specific for Electron.

That will reduce the amount of build CI complexity for Polykey, and speed up builds for Polykey-CLI.

[CMCDragonkai]

This might be relevant https://esbuild.github.io/api/#keep-names.

Cross posting this MatrixAI/TypeScript-Demo-Lib#32 (comment)