Skip to content

Batch worker fs tree walks into one spawn_blocking per directory - #2611

Open
amankrx wants to merge 1 commit into
TraceMachina:mainfrom
amankrx:fix/worker-materialization-batch-spawn-blocking
Open

Batch worker fs tree walks into one spawn_blocking per directory#2611
amankrx wants to merge 1 commit into
TraceMachina:mainfrom
amankrx:fix/worker-materialization-batch-spawn-blocking

Conversation

@amankrx

@amankrx amankrx commented Jul 23, 2026

Copy link
Copy Markdown
Member

Description

The worker's input-materialization walks issued a tokio::fs call (each a spawn_blocking on tokio's blocking pool) per file. Materializing an action's input tree touches thousands of files, and at high action concurrency the per-file blocking-pool handoff dominates on many-core hosts while CPU and I/O sit near idle.

Batch both recursive walks to one blocking task per directory:

  • hardlink_directory_tree_recursive: read the directory, hardlink files and recreate symlinks inline, create subdirectories, and return them for the async layer to recurse into.
  • set_perms_recursive (set_readonly_recursive / set_dir_writable_recursive): convert perms_fn to a synchronous closure and chmod each directory's entries plus the directory itself in a single blocking task, recursing per subdirectory.

Fixes #2609

Type of change

Please delete options that aren't relevant.

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Please also list any relevant details for your test configuration

Checklist

  • Updated documentation if needed
  • Tests added/amended
  • bazel test //... passes locally
  • PR is contained in a single commit, using git amend see some docs

This change is Reviewable

@vercel

vercel Bot commented Jul 23, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
nativelink Ready Ready Preview, Comment Jul 23, 2026 12:13pm
nativelink-aidm Ready Ready Preview, Comment Jul 23, 2026 12:13pm

Request Review

@palfrey palfrey left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is failing due to uses of tokio::task::spawn_blocking and should be using spawn_blocking! instead

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

per-file spawn_blocking + per-process global mutexes: lock-handoff storms on many-core hosts

2 participants