feat(internal/librarian/nodejs): use pnpm install#6110
Conversation
There was a problem hiding this comment.
Code Review
This pull request migrates the Node.js tool installation logic from npm to pnpm, updating the configuration schema, internal structs, and installation procedures. The changes include activating pnpm via corepack and dynamically setting the global binary directory. Feedback suggests defining the hardcoded pnpm version as a constant and using environment variables instead of persistent global configuration commands to avoid side effects on the host environment.
…bal enable dependency
…andardizing on pnpm
… to preserve package names and baselines
… Node APIs during migration
…de sidekick test suites to pass TestGolangCILint
coryan
left a comment
There was a problem hiding this comment.
Those extra return statements after t.Fatal() and after t.Fatalf() seems wrong, please revert.
JoeWang1127
left a comment
There was a problem hiding this comment.
Could you revert the changes in sidekick directory? They seems redundant.
|
Could you fix the unit tests? |
3b8ca9a to
9ef6f40
Compare
TLDR
This PR refactors our Node.js installer inside librarian. We migrated from our current practice of using the
npmCLI wrapper to the pnpm toolchain utilizingcorepackto establish absolute package linkage determinism.By using pnpm and adhering to the lock files installation, we can create a deterministic installation environment. Matching our tool versions to those specified in the lockfiles allows us to match our current generation pipeline output.
Context: Why should we transition to pnpm instead of npm?
npminstallations are deterministic when consuming local project files underpackage-lock.json. However, our current practice of performing dynamic global package installations inside librarian (e.g., executingnpm install -g gapic-tools@1.0.5on host environments) does not respect lockfiles, floating transitives dynamically. Standardizing onpnpmresolves this by strictly respectingpnpm-lock.yamlacross our source-built target tools.pnpmdirectly reflects and aligns our local generation tools with the hermetic sandboxing configurations we use upstream in our current bazel-bot pipeline and postprocessor.pnpmin the future due to its first-class workspace support and content-addressable storage design tailored for monorepos.npmbinary during tool bootstrapping, we successfully transitioned our installer to Node.js's built-incorepackwrapper.corepack enable pnpmfirst.corepack prepare pnpm@7.32.2 --activate.process.execPathfolder path directly in JavaScript (node -e "console.log(require('path').dirname(process.execPath))") to configurepnpm'sglobal-bin-dir. This runs completely independently of standardnpm configcalls.npm linkwithpnpm link --globalto bind the compiled typescript generator to our shared global virtual store.Why do should we explicitly specify protobufjs?
During local runs of the generator we created against
google-cloud-secretmanager, the generatedprotos.jsfile introduced a substantial diff containing recursive depth limits (long > $util.recursionLimit) and prototype pollution validations (keys[i] !== "__proto__").We traced this discrepancy back to a peer dependency resolution mismatch:
protobufjs-cli, which depends onprotobufjsas a peer dependency. In floated global registry environments, this peer dependency resolves to the latestprotobufjs@7.6.0, which automatically injects these recursion and security checks into the generated JavaScript AST.pnpmnot address this?:pnpm install),pnpm's virtual store correctly isolates peer dependencies according to the locks.gapic-tools(which contains thecompileProtoscompilation shim) is installed from the NPM registry globally (pnpm add -g), published packages do not bundle a lockfile.pnpmdynamically resolves the floated peer range^7.5.3underprotobufjs-clito the latest minor version (7.6.0), triggering the security and depth limit checks.protobufjssub-dependency bundled inside the sandbox (specifically7.5.4/7.5.5), generating AST files without these checks.protobufjs-cli@1.2.0andprotobufjs@7.5.5inside our tools configuration block inlibrarian.yaml.