Skip to content

perf(workers): use the final VM sweep during shutdown - #154

Merged
steipete merged 1 commit into
mainfrom
claude/worker-heap-destruction-resume
Oct 9, 2026
Merged

steipete merged 1 commit into
mainfrom
claude/worker-heap-destruction-resume

Conversation

@steipete

@steipete steipete commented Oct 9, 2026 •

Copy link
Copy Markdown

Worker shutdown performs a full tracing collection immediately before mandatory VM destruction. Move that preliminary collection to the main-process-only caller; Workers retain native/environment cleanup, code deletion, VM reference draining and last-chance finalization. No VM reuse or deferred acknowledged cleanup is introduced. This is rebased on the finalized ownership in #152 and the merged EventNames cleanup in #153.

The heap-specific regression fails the unchanged baseline at the extra full collection and passes both Worker exit routes. The current-head Mac run passes 271 Worker destruction/N-API/finalizer tests; the original eight-resource fixture remains 8/8 with zero fds, and WAL durability/reopen checks pass. Three OpenClaw cleanup/retirement files pass 13 tests on Node and 13 on the integration. The earlier frozen integration's surrounding Worker suites pass 303 tests with 20 existing gated skips.

Fixed paired integration measurements use the same frozen Bun/engine source and build settings in both arms. The companion engine change is openclaw/WebKit#31; no standalone-Bun attribution is inferred from combined numbers.

Loaded Worker, median ms Before Engine-only Combined Node 26.10.0
Linux synthetic catalog 22.216 20.461 14.380 2.804
Linux real catalog-module import 37.367 37.329 23.896 4.989
Mac synthetic catalog 6.178 6.191 4.790 1.174
Mac real catalog-module import 19.171 18.573 16.071 2.185

Each component cell has 20 observations in fixed forward/reverse order. Empty Mac point estimates are mixed; no universal speedup or Node parity is claimed. Separate phase logs show one preliminary collection before and zero after. Some destructor work moves into the final sweep rather than disappearing.

Full Gateway before/after AB/BA keeps the workload and 2,000 ms grace unchanged: 80/80 Linux workloads and 82 completed Mac workloads exit cleanly under the bound, with no kill attempt. Two Mac setup failures remain recorded. Linux medians improve 912.05→806.92 ms without trace and 997.33→874.01 ms with trace; Mac trace-off improves modestly, while trace-on has a small mixed regression. Both baseline and candidate meet the bound in this sample, so historical deadline misses are not proved eliminated by this change.

Calibrated ASAN tests retain the stronger ten-resource callback/thread/null-env/reentry assertions through 100 Workers. The frozen before/after comparison has only the known cache/parser leak sites. A fully rebuilt local composition with the separate cache/parser fixes passes 1,000 Workers on both routes and 100 retained-addon Workers per route under ASAN/LSAN, using only the inherited console suppression. Its unprimed exact-key string finalizes once with null env on the creator thread. No new suppression or lifetime-policy exception is introduced; the parser fix remains separate.

P2 review is scoped-clean. At head 3851747d2e747f7f92fed8cc9b0d1d7ce1bc4e2d, Linux/macOS native CI and Windows x64/ARM64 builds, smoke tests and compatibility tests pass, as do formatting and source/JavaScript lint. Windows attempt 1 stopped in provisioning on Docker Hub manifest/token timeouts before compilation or tests; only those interrupted infrastructure jobs were retried at the same head. The workflow used publish=false, with signing/publication skipped and no tag created. This draft remains unmerged and publishes no runtime. Upstream searches found no equivalent Worker-only collection-removal fix to port.

Synthetic fixture and measurement procedure

Save these as make-graph.py and loaded-worker.mjs in a fresh directory. The graph has 1,200 modules with 24 exported functions each; the Worker retains another 100,000 objects. Startup/import is outside the parent termination interval. sqlite adds the same graph plus 2,000 in-memory rows; empty imports no graph.

from pathlib import Path
out=Path('graph')
out.mkdir(exist_ok=True)
for i in range(1200):
 p=out/f'm{i}.mjs'
 p.write_text('\n'.join([
  f'export const metadata = {{id: "provider-{i}", name: "Model {i}", contextWindow: 128000, input: ["text", "image"], cost: {{input: 0.1, output: 0.3}}}};',
  *[f'export function normalize{j}(x) {{ return {{...metadata, value: String(x).replace(/provider-{i}-field-{j}/g, "field"), key: {j}}}; }}' for j in range(24)],
  'export const schema = new Map(Object.entries(metadata));',
 ]))
(out/'entry.mjs').write_text('\n'.join(f'import * as m{i} from "./m{i}.mjs";' for i in range(1200))+'\nexport const catalog = ['+','.join(f'm{i}' for i in range(1200))+'];\n')
import { Worker, isMainThread, parentPort, workerData } from 'node:worker_threads';
import { performance } from 'node:perf_hooks';
import { writeSync } from 'node:fs';
const [mode='terminate', countText='30', shape='catalog', moduleURL=''] = process.argv.slice(2);
if (!isMainThread) {
  const held = workerData.shape === 'empty' ? {} : workerData.moduleURL ? await import(workerData.moduleURL) : await import('./graph/entry.mjs');
  const objects = Array.from({length: workerData.shape === 'empty' ? 0 : 100000}, (_,i)=>({id:i,value:`model-${i}`,metadata:held, next:null}));
  for(let i=1;i<objects.length;i++) objects[i-1].next=objects[i];
  globalThis.held={held,objects};
  if (workerData.shape === 'sqlite') {
    const {DatabaseSync} = await import('node:sqlite');
    const db = new DatabaseSync(':memory:');
    db.exec('CREATE TABLE entries (id INTEGER PRIMARY KEY, body TEXT); BEGIN');
    const insert = db.prepare('INSERT INTO entries VALUES (?, ?)');
    for(let i=0;i<2000;i++) insert.run(i,`model-${i}`);
    db.exec('COMMIT');
    globalThis.database={db,insert,read:db.prepare('SELECT * FROM entries')};
  }
  parentPort.on('message', () => process.exit(0));
  parentPort.postMessage({ready:true});
} else {
  const rows=[];
  for(let i=0;i<Number(countText);i++) {
    const w=new Worker(new URL(import.meta.url), {workerData:{shape,moduleURL}});
    let ready=false;
    const exited=new Promise((resolve,reject)=>{w.once('exit', code=>ready?resolve(code):reject(new Error(`early exit ${code}`)));w.once('error',reject);});
    await Promise.race([new Promise(resolve=>w.once('message',m=>{if(m.ready){ready=true;resolve();}})),exited]);
    if(mode==='main-exit') { writeSync(1, 'READY_MAIN_EXIT\n'); process.exit(0); }
    const start=performance.now();
    const code = mode==='terminate' ? await w.terminate() : (w.postMessage('exit'), await exited);
    rows.push({i,ms:performance.now()-start,code});
  }
  console.log(JSON.stringify({runtime:process.versions,mode,shape,moduleURL,rows}));
}

Generate once with python3 make-graph.py. For each runtime, invoke loaded-worker.mjs terminate 1 catalog (or worker-exit, empty, sqlite). Use fresh processes and private HOME/state/temp directories. Our comparison uses 20 fixed blocks per route/shape, with before/engine-only/combined/Node order on odd blocks and the reverse on even blocks; logging and profilers are disabled. Retain every attempt rather than replacing failures. Node reference is 26.10.0. These reduced fixtures do not replace full Gateway validation.

Workers already finish environment and native-resource cleanup before retiring the VM. Avoid tracing and sweeping the heap immediately before its mandatory last-chance finalization, while preserving main-process collection and VM reference draining. Cover both Worker exit routes with heap-specific collection diagnostics.
@steipete
steipete force-pushed the claude/worker-heap-destruction-resume branch from d8549f6 to 3851747 Compare October 9, 2026 21:24
@steipete
steipete marked this pull request as ready for review October 9, 2026 23:03
@steipete
steipete merged commit a683b27 into main Oct 9, 2026
26 of 28 checks passed
@steipete
steipete deleted the claude/worker-heap-destruction-resume branch October 9, 2026 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant