Skip to content

feat(internal/librarian/nodejs): use pnpm install#6110

Merged
quirogas merged 32 commits into
googleapis:mainfrom
quirogas:feat/use-pnpm-install
May 23, 2026
Merged

feat(internal/librarian/nodejs): use pnpm install#6110
quirogas merged 32 commits into
googleapis:mainfrom
quirogas:feat/use-pnpm-install

Conversation

@quirogas
Copy link
Copy Markdown
Contributor

@quirogas quirogas commented May 21, 2026

TLDR

This PR refactors our Node.js installer inside librarian. We migrated from our current practice of using the npm CLI wrapper to the pnpm toolchain utilizing corepack to establish absolute package linkage determinism.

By using pnpm and adhering to the lock files installation, we can create a deterministic installation environment. Matching our tool versions to those specified in the lockfiles allows us to match our current generation pipeline output.


Context: Why should we transition to pnpm instead of npm?

  1. Deterministic Dependency Isolation: Standard npm installations are deterministic when consuming local project files under package-lock.json. However, our current practice of performing dynamic global package installations inside librarian (e.g., executing npm install -g gapic-tools@1.0.5 on host environments) does not respect lockfiles, floating transitives dynamically. Standardizing on pnpm resolves this by strictly respecting pnpm-lock.yaml across our source-built target tools.
  2. Infrastructure Alignment: Moving to pnpm directly reflects and aligns our local generation tools with the hermetic sandboxing configurations we use upstream in our current bazel-bot pipeline and postprocessor.
  3. Monorepo Support & Forward Outlook: The Node.js platform team has expressed a strategic intent to migrate libraries and generators entirely to pnpm in the future due to its first-class workspace support and content-addressable storage design tailored for monorepos.
  4. Complete npm Decoupling: To eliminate the runtime footprint of the npm binary during tool bootstrapping, we successfully transitioned our installer to Node.js's built-in corepack wrapper.
    • Our installer now automatically creates and maps standard executable bin shims globally using corepack enable pnpm first.
    • Our installer activates the target version globally via corepack prepare pnpm@7.32.2 --activate.
    • Our installer queries Node's native process.execPath folder path directly in JavaScript (node -e "console.log(require('path').dirname(process.execPath))") to configure pnpm's global-bin-dir. This runs completely independently of standard npm config calls.
  5. Clean Global Links: We replaced npm link with pnpm link --global to bind the compiled typescript generator to our shared global virtual store.

Why do should we explicitly specify protobufjs?

During local runs of the generator we created against google-cloud-secretmanager, the generated protos.js file introduced a substantial diff containing recursive depth limits (long > $util.recursionLimit) and prototype pollution validations (keys[i] !== "__proto__").

We traced this discrepancy back to a peer dependency resolution mismatch:

  • The Mismatch: Upstream compiler shims rely on protobufjs-cli, which depends on protobufjs as a peer dependency. In floated global registry environments, this peer dependency resolves to the latest protobufjs@7.6.0, which automatically injects these recursion and security checks into the generated JavaScript AST.
  • Does pnpm not address this?:
    • Normally, when executing project-level installations inside folders with locked files (pnpm install), pnpm's virtual store correctly isolates peer dependencies according to the locks.
    • However, because gapic-tools (which contains the compileProtos compilation shim) is installed from the NPM registry globally (pnpm add -g), published packages do not bundle a lockfile.
    • In the absence of a lockfile, pnpm dynamically resolves the floated peer range ^7.5.3 under protobufjs-cli to the latest minor version (7.6.0), triggering the security and depth limit checks.
  • The Current Pipeline: However, our current bazel-bot pipeline runfiles are hermetically isolated. During historical compilation, the peer dependency resolved to the only available, older protobufjs sub-dependency bundled inside the sandbox (specifically 7.5.4 / 7.5.5), generating AST files without these checks.
  • The Aligned Solution:
    • To resolve this peer range drift and align with our pipeline output, we added direct peer pins for protobufjs-cli@1.2.0 and protobufjs@7.5.5 inside our tools configuration block in librarian.yaml.
    • This forces the global pnpm virtual store to link the peer dependencies directly to the correct pre-patched versions, resolving the version-mismatch checks completely (leaving only benign monorepo root namespace registry adjustments!).

@quirogas quirogas self-assigned this May 21, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request migrates the Node.js tool installation logic from npm to pnpm, updating the configuration schema, internal structs, and installation procedures. The changes include activating pnpm via corepack and dynamically setting the global binary directory. Feedback suggests defining the hardcoded pnpm version as a constant and using environment variables instead of persistent global configuration commands to avoid side effects on the host environment.

Comment thread internal/librarian/nodejs/install.go Outdated
Comment thread internal/librarian/nodejs/install.go Outdated
@quirogas quirogas requested a review from JoeWang1127 May 21, 2026 21:12
@quirogas quirogas marked this pull request as ready for review May 21, 2026 21:12
@quirogas quirogas requested a review from a team as a code owner May 21, 2026 21:12
@quirogas quirogas requested review from a team as code owners May 22, 2026 06:04
coryan
coryan previously requested changes May 22, 2026
Copy link
Copy Markdown
Contributor

@coryan coryan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those extra return statements after t.Fatal() and after t.Fatalf() seems wrong, please revert.

Comment thread internal/sidekick/rust/annotate_method_test.go Outdated
Comment thread internal/librarian/nodejs/generate.go Outdated
Comment thread internal/librarian/nodejs/install.go Outdated
Comment thread internal/sidekick/parser/discovery/discovery_test.go Outdated
Copy link
Copy Markdown
Contributor

@JoeWang1127 JoeWang1127 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you revert the changes in sidekick directory? They seems redundant.

Comment thread internal/librarian/nodejs/install.go
Comment thread internal/librarian/nodejs/install.go Outdated
Comment thread internal/librarian/nodejs/install_test.go
@JoeWang1127
Copy link
Copy Markdown
Contributor

Could you fix the unit tests?

@coryan coryan dismissed their stale review May 22, 2026 18:34

No longer affects Rust.

@quirogas quirogas force-pushed the feat/use-pnpm-install branch from 3b8ca9a to 9ef6f40 Compare May 22, 2026 18:39
@quirogas quirogas requested a review from JoeWang1127 May 22, 2026 20:38
@quirogas quirogas enabled auto-merge (squash) May 22, 2026 20:48
Comment thread internal/librarian/nodejs/generate.go Outdated
Comment thread internal/librarian/nodejs/install.go Outdated
Comment thread internal/librarian/nodejs/install.go
Comment thread internal/librarian/nodejs/install.go
Comment thread internal/librarian/nodejs/install.go Outdated
Comment thread internal/config/language.go Outdated
Comment thread internal/librarian/nodejs/install.go
@quirogas quirogas requested review from JoeWang1127 and coryan and removed request for coryan May 22, 2026 22:31
@quirogas quirogas merged commit 17b85b3 into googleapis:main May 23, 2026
30 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants