[perf] async/await parallelization opportunities — plan loop + chunk batch + file ops

[jlin53882]

Analysis Summary

This issue documents async/await parallelization opportunities found in memory-lancedb-pro.

Unit Test Proof Results

Test	Sequential	Parallel	Speedup	Priority
memory-compactor plan loop	943ms	94ms	10.0x	HIGH
store.ts doFlush chunk	156ms	62ms	2.5x	MEDIUM
self-improvement-files ensureFile	94ms	46ms	2.0x	LOW

Test file: test/async-parallel-simple.mjs

Related Issues

• Issue [perf] MMR O(n³)→O(n²) + Event Loop Blocking 防治（P2+P3 合併追蹤） #763 - MMR O(n3) to O(n2) + Event Loop Blocking prevention
• Issue [BUG] 100 concurrent bulkStore() calls still timeout — updateQueue prevents errors but not throughput #690 - 100 concurrent bulkStore() timeout
• Issue [BUG] Regex fallback (agent_end hook) uses per-item store.store() causing lock timeout under high-frequency auto-capture #675 - per-item store.store() timeout

Recommended Fixes

1. memory-compactor.ts plan loop (HIGH priority)

Location: src/memory-compactor.ts:196-230

Current:

for (const plan of plans) {
  const vector = await embedder.embedPassage(plan.merged.text);
  await store.store({...});
  for (const m of members) {
    await store.delete(m.id);
  }
}

Proposed:

await Promise.all(plans.map(async (plan) => {
  const vector = await embedder.embedPassage(plan.merged.text);
  await store.store({...});
  await Promise.all(plan.memberIndices.map(m => store.delete(m.id)));
}));

2. store.ts doFlush chunk (MEDIUM priority)

Location: src/store.ts:647-654

Current:

for (const chunk of chunks) {
  await this.runWithFileLock(async () => {
    await this.table!.add(chunk);
  });
}

Proposed (batch parallel):

const BATCH_SIZE = 3;
for (let i = 0; i < chunks.length; i += BATCH_SIZE) {
  const batch = chunks.slice(i, i + BATCH_SIZE);
  await Promise.all(batch.map(chunk => 
    this.runWithFileLock(() => this.table!.add(chunk))
  );
}

3. self-improvement-files.ts ensureFile (LOW priority)

Location: src/self-improvement-files.ts:69-70

Current:

await ensureFile(join(learningsDir, "LEARNINGS.md"), DEFAULT_LEARNINGS_TEMPLATE);
await ensureFile(join(learningsDir, "ERRORS.md"), DEFAULT_ERRORS_TEMPLATE);

Proposed:

await Promise.all([
  ensureFile(join(learningsDir, "LEARNINGS.md"), DEFAULT_LEARNINGS_TEMPLATE),
  ensureFile(join(learningsDir, "ERRORS.md"), DEFAULT_ERRORS_TEMPLATE)
]);

[jlin53882]

最新更新：PR #787 包含完整修復

程式碼修復已完成

1. memory-compactor.ts - 行 51-79

• 使用 Promise.all 並行處理 plans
• 使用 Promise.all 並行處理 members delete
• 測試結果：15.7x speedup (1107ms → 70ms)

2. self-improvement-files.ts - 行 69-73

• 使用 Promise.all 並行處理 ensureFile
• 測試結果：2.0x speedup (93ms → 48ms)

單元測試驗證

`
▶ memory-compactor plan loop parallelization
✔ parallel should be faster than sequential (1178ms)
Sequential: 1107ms
Parallel: 70ms
Speedup: 15.7x

▶ self-improvement-files ensureFile parallelization
✔ parallel should be faster than sequential (140ms)
Sequential: 93ms
Parallel: 48ms
Speedup: 2.0x

ℹ tests 2
ℹ pass 2
ℹ fail 0
`

檔案變更

• src/memory-compactor.ts - 并行化 plan loop
• src/self-improvement-files.ts - 并行化 file operations
• est/async-parallelization.test.mjs - 带断言的测试

相關 Issue

• [perf] async/await parallelization opportunities — plan loop + chunk batch + file ops #785 (本 PR)
• [perf] MMR O(n³)→O(n²) + Event Loop Blocking 防治（P2+P3 合併追蹤） #763
• [BUG] 100 concurrent bulkStore() calls still timeout — updateQueue prevents errors but not throughput #690
• [BUG] Regex fallback (agent_end hook) uses per-item store.store() causing lock timeout under high-frequency auto-capture #675