Release the mutation lock before emitting the result

[em3s]

Summary

Mutation lock release was fire-and-forget, so a completed response did not imply a released lock. doFinally runs after the result has propagated to the caller, and the .subscribe() inside it only submits the release to a scheduler queue before returning. A second mutation on the same edge can therefore arrive while the lock is still held, exhaust its retry budget (50 × 100ms = 5s), and fail with LockAcquisitionFailed. The gap is 1–2ms normally, but grows to seconds when boundedElastic is saturated.

The lag was already instrumented but never fixed: AbstractLabel.lockMonitor logs an error for every lock not released within 2 seconds.

Changes

Applied identically to AbstractLabel (v2 path) and V2BackedTableBinding (v3 path).

	As-is	To-be
Ordering	emit result, release later	release, then emit result
Release issued	after the result propagates (doFinally)	before the result is emitted
Meaning of a completed response	lock may still be held	lock release confirmed
Success path	doFinally { release.subscribe() }	delayUntil { release }
Error path	same doFinally	onErrorResume { release.then(error) }
Cancel path	same doFinally	doFinally(CANCEL) — unchanged
Failed release	dropped (onErrorDropped)	warn log; the mutation still succeeds
Lock hold time	until the release completes	shorter, since the release is issued earlier
Consecutive writes to one edge	fail after 5s under load	succeed immediately
Response latency	no release wait	includes one release CAS round trip

No call sites changed. Public API, exception types, response payloads and storage format are unchanged. The wait cannot become unbounded — mutationRequestTimeout already wraps the whole mutation. The trade-off is that storage degradation now surfaces as latency instead of failure.

Verified: no double release (the success, error and cancel paths are mutually exclusive), and a release cannot delete another holder's lock (cad is a conditional delete against the holder's own value).

How to Test

./gradlew :engine:test :server:test

Measured on a larger change set that exercises this path: without the fix the full suite failed 2 of 4 runs; with it, 3 consecutive runs passed.

Production impact can be assessed by grepping for was not released within — if it fires, this fixes an active issue rather than a latent one.

AI Assistance

• This PR was written largely with AI assistance.
  • Tool / model: claude code (opus 5)

[em3s]

Optimistic