Handle in-flight lock transactions

[ezforce]

Summary

This change fixes a lock recovery gap in the broker and market lock flow.

When a lock transaction has already been accepted by the node or mempool, the RPC send path can return already known style errors even though the transaction is still live. In v1.2.2, the broker treats that as a hard failure, marks the order as skipped, clears the lock cache, and never reconciles the on-chain lock state. If the transaction later lands, the order can remain locked by us without ever entering proving, which can lead to a slash.

This PR changes that behavior so in-flight lock submissions are reconciled instead of being dropped.

Changes

• classify already known style lock send failures as in-flight lock submissions
• share the in-flight lock reconciliation path between lock_request() and lock_request_with_signature()
• query RequestLocked before treating an in-flight lock submission as a failure
• recover the order as locked when the on-chain locker is our prover
• return RequestAlreadyLocked when another prover already holds the lock
• return LockTxInFlight when the lock transaction cannot be reconciled yet
• keep the broker lock cache entry alive while the lock state is still unresolved
• avoid inserting Skipped rows for unresolved in-flight lock submissions
• allow the next broker tick to recover the order into proving once the on-chain lock becomes visible

Root Cause

The previous flow failed in two places:

1. call.send().await errors on lock submission were propagated directly, even for recoverable transaction-in-flight cases.
2. the broker treated those errors as terminal, inserted Skipped, and invalidated the lock cache before it had a chance to observe that the chain was already locked by this prover.

That combination made a successful lock transaction unrecoverable from the broker point of view.

Testing

Ran on main:

• cargo test -p broker lock_order_recovers_when_request_is_already_locked_by_us_on_chain -- --nocapture
• cargo test -p broker already_known_ -- --nocapture

Added regression coverage for:

• recovering a lock when the request is already locked by us on-chain
• keeping the order in the lock cache when an already known error occurs without a visible broadcast yet
• recovering an order into PendingProving after an already known error once the lock becomes visible on-chain

Backport

The same fix has been backported to release-1.2 in:

• codex/release-1.2-handle-in-flight-locks
• commit 8b0080d5

Note: local release-1.2 broker test execution is currently blocked by a pre-existing deploy_hit_points() failure (HTTP error 502 with empty body) during test environment setup. The same failure also reproduces in an existing unmodified broker test, so it is not introduced by this patch.

---

Note

Medium Risk
Changes lock transaction error-handling and broker state transitions around on-chain locking; incorrect classification could cause missed locks or repeated retries, impacting proving/slashing behavior.

Overview
Improves resilience to in-flight lock transactions where RPC send returns "already known"/similar errors even though the tx may still land.

The market client now classifies these send failures as recoverable, attempts to reconcile by querying RequestLocked, and returns a new MarketError::LockTxInFlight when the lock can’t be confirmed yet (or RequestAlreadyLocked if another prover won). The broker consumes this new error/state: it can recover orders already locked by this prover, defers unresolved locks by keeping them in the lock cache (avoiding Skipped DB rows), and retries on the next tick; adds targeted regression tests with an injected provider that simulates these RPC failures.

^{Written by Cursor Bugbot for commit 727d9b5. This will update automatically on new commits. Configure here.}