Skip to content

fix: make store-based barrier reusable across updates - #107

Merged
weixiao-huang merged 2 commits into
MoonshotAI:mainfrom
RunningLeon:fix/reusable-store-barrier
Sep 15, 2026
Merged

weixiao-huang merged 2 commits into
MoonshotAI:mainfrom
RunningLeon:fix/reusable-store-barrier

Conversation

@RunningLeon

Copy link
Copy Markdown
Contributor

fix: make the shared-store barrier reusable across updates in #106

Summary

  • add an independent generation counter for ParameterServer.store_based_barrier();
  • use a unique _store_based_barrier() group name for every invocation;
  • keep the long-lived root TCPStore unchanged and shared;
  • add CPU-only regression coverage for two consecutive barriers on the same TCPStore.

@koriyoshi2041 koriyoshi2041 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the repeated-barrier path on the exact head. The generation counter is independent of process-group creation, every rank deterministically advances it once per successful update barrier (including auto_pg=False), and the shared-store regression exercises two consecutive generations. Locally, tests/test_store_barrier.py passes 2/2; focused Ruff check/format and diff hygiene also pass.

@RunningLeon

Copy link
Copy Markdown
Contributor Author

@koriyoshi2041 hi, is there a plan to merge this PR?

@koriyoshi2041

Copy link
Copy Markdown
Contributor

I reviewed this as an external contributor and don't have merge rights in this repository, so I can't give a maintainer merge timeline. The PR is still at the exact head I approved (300f5c2), both hosted checks are green, and I found no correctness blocker in the repeated shared-store barrier path. A repository maintainer will need to make the merge decision.

Comment thread checkpoint_engine/ps.py
Comment thread checkpoint_engine/ps.py

@koriyoshi2041 koriyoshi2041 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the requested key cleanup at exact head c63195f. Rank 0 only deletes generation N after completing N+2; completion of N+2 means every rank has entered that later barrier, so no rank can still depend on N's counter or last_worker key. Cleanup failure remains non-fatal and bounded to a warning.

Locally, the real shared-TCPStore regression passes all 10 generations and verifies generations 1–8 are removed while 9–10 remain (2/2 focused tests). Ruff check and format also pass on both changed files. Approving the updated head.

@weixiao-huang
weixiao-huang merged commit 84631a2 into MoonshotAI:main Sep 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants