Skip to content

Update v1 to v2 storage migration tool#271

Open
Vui-Chee wants to merge 24 commits intomainfrom
vui-chee/reth-v2.0.0-sf-patch-test
Open

Update v1 to v2 storage migration tool#271
Vui-Chee wants to merge 24 commits intomainfrom
vui-chee/reth-v2.0.0-sf-patch-test

Conversation

@Vui-Chee
Copy link
Copy Markdown
Contributor

@Vui-Chee Vui-Chee commented Apr 28, 2026

Upstream reth finally created a tool for v2 migration. So port over with custom genesis height.

Took ~30 minutes for mainnet snapshot to be migrated. Note, the latest tool does not write to rocksdb.

Testing Procedure

  1. download snapshot data (mainnet)
  2. run devnet using this PR with v1 storage with merge genesis settings active
  3. custom genesis file + state root calculation will be generated during 3-op-init.sh
  4. stop the op-reth-seq node
  5. using this, perform v1 to v2 migration using the new custom chain xlayer-devnet over the snapshot metadata (under ./data/op-reth-seq)
  6. setup storage v2 in settings before run
  7. start the op-reth-seq node (obviously restart op-node as well)

sieniven and others added 23 commits April 27, 2026 22:55
Bump reth to okx/reth@056289cd (v2.0.0), align all alloy/revm/op-alloy
deps to versions used by reth, bump optimism submodule, drop obsolete
op-alloy crates.io patches, and bump toolchain to 1.94.0.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migrate xlayer-reth call sites for the upstream reth v2.0.0 and op-reth
v2.1.0 upgrade:

- Replace removed `PayloadBuilderAttributes` trait usage with
  `OpPayloadBuilderAttributes` struct + `OpPayloadAttrs` engine API
  wrapper; flatten nested `payload_attributes.X` field accesses.
- Drop the removed `TaskSpawner` / `TokioTaskExecutor` generics in
  flashblocks builder, generator, handler, RPC subscription, and
  monitor in favor of concrete `reth_tasks::TaskExecutor`.
- Rewrite `DefaultPayloadBuilder` `PayloadBuilder` impl to mirror
  upstream `OpPayloadBuilder` (`Attributes = OpPayloadAttrs`, convert
  to internal builder attrs in `try_build`).
- Plumb `runtime: reth_tasks::Runtime` through bin/tools commands and
  the new `import_blocks_from_file` arg.
- Add `cache_for`, `sparse_trie_handle_for`, and a separate
  `WaitForCaches` impl on `XLayerEngineValidator` to satisfy the new
  `EngineValidator` trait surface; return the new
  `(ExecutedBlock, Option<Box<ExecutionTimingStats>>)` tuple.
- Update `OpFlashblockPayloadMetadata` construction for the new
  `Option<BTreeMap>` fields plus `access_list`.
- Migrate validator state-root pipeline to the new payload_processor
  surface: keep `Arc<TrieUpdates>` end-to-end into
  `DeferredTrieData::pending`, switch from the now-private
  `ChangesetCache::insert` to `register_pending`/`resolve`, set
  `ExecutionEnv.gas_used`, and box the `OnStateHook` from the
  optional state hook.
- Preserve `WithEncoded` bytes through `flashblock_tx_iterator` so
  the prewarm path uses `FromTxWithEncoded` instead of recomputing
  `tx.encoded_2718()` per transaction via `FromRecoveredTx`.
- Add `PostExec` arms for `OpReceipt` and `OpTypedTransaction`,
  pairing them with the existing `Deposit` arm in `Signer::sign_tx`
  (system transactions, never user-signed).
- Implement `PoolTransaction::consensus_ref` on `MockFbTransaction`
  mirroring upstream's `unimplemented!()` mock pattern.
- Replace the removed `set_state_clear_flag` call (Spurious Dragon
  empty-account handling is always on in revm 12.0).
- Derive next-block number from `provider.last_block_number() + 1`
  in `monitor::handle` since `Events::Attributes` no longer carries
  parent hash.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Rust badge was bumped in the v2.0.0 upgrade commit but the Reth
badge link/version was missed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bump okx/reth pin to ccbef3f and update deps/optimism submodule.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…or migrate port

Pulled in to support porting upstream's `reth db migrate-v2` command
(crates/cli/commands/src/db/migrate_v2.rs) into bin/tools/legacy_migrate.

- reth-codecs: Compact bound on Receipt for static file write
- reth-prune-types: PruneSegment enum for prune-checkpoint lookup
- reth-stages-types: StageId enum for execution-stage tip
- reth-static-file: canonical Receipts::copy_to_static_files helper
Replaces the local hand-rolled MDBX→static-file/RocksDB migration with
upstream reth's `db migrate-v2` (crates/cli/commands/src/db/migrate_v2.rs)
adapted for XLayer:

  * wrapped in our `LegacyMigrateCommand<C>` so it works with
    `EnvironmentArgs<OpChainSpec>` and our `runtime: reth_tasks::Runtime`
  * floors `first_block` at `genesis_block` for AccountChangeSets,
    StorageChangeSets and Receipts so a non-zero genesis (XLayer mainnet)
    doesn't waste time scanning empty MDBX keys
  * gates the leading `ensure_at_block(first_block - 1)` on
    `first_block > genesis_block` and bridges via
    `increment_block(genesis_block)` first, because upstream's
    `ensure_at_block` opens fresh writers with `increment_block(0)` —
    which fails when `next_block_number` starts at a non-zero genesis

The new flow follows upstream phases verbatim:
  Phase 0 — preflight (idempotency + Execution-stage tip + empty segments)
  Phase 1 — migrate AccountChangeSets / StorageChangeSets to static files
  Phase 2 — migrate Receipts via `Segment::copy_to_static_files`
  Phase 3 — write `StorageSettings::v2()`
  Phase 4 — clear recomputable tables + reset stage checkpoints
  Phase 5 — compact MDBX with `mdbx_env_copy(MDBX_CP_COMPACT)` + atomic
            backup-and-swap

The previous local design copied indices to RocksDB during migration and
ran static-file + RocksDB phases in parallel. Following upstream we now
just clear those indices and let the pipeline rebuild them on next start.
This removes ~430 lines of custom code (migrate.rs + progress.rs) and
inherits future fixes from upstream.

Cargo.toml: switch `reth-codecs` from git to published `version = "0.1.0"`
to match how the okx fork declares it.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add [patch."https://github.com/paradigmxyz/reth"] in workspace root so
  reth-* crates pulled transitively via deps/optimism/rust path crates
  resolve to the same okx/reth rev pinned in [workspace.dependencies],
  collapsing the lockfile to a single reth source.
- Bump deps/optimism submodule to 73460092 (sync upstream).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…VE patch

The local fork was at 1.6.3 from when the workspace shipped reth v1.11.3.
The reth v2.0.0 alignment bumped alloy-rpc-types-engine to 1.8.x, leaving
the 1.6.3 fork outside semver range and silently unused — jsonwebtoken
fell back to the vulnerable 9.3.1 from crates.io.

Replace the fork's contents with upstream alloy-rpc-types-engine 1.8.3,
re-apply the jsonwebtoken pin to 10.3.0 with the rust_crypto backend, and
add the missing [patch.crates-io] header above the security patches so
both the alloy-rpc-types-engine and libp2p-yamux overrides actually take
effect. Cargo.lock now resolves jsonwebtoken to 10.3.0, libp2p-yamux to
the local fork (no yamux 0.12.x), and emits no patch-unused warnings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bump okx/reth rev from ccbef3f8 to 2de4f640 across both
[workspace.dependencies] and [patch."https://github.com/paradigmxyz/reth"]
so the workspace stays on a single reth source. Cargo.lock fully migrated
to the new rev (103 refs, no stragglers on the old one).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* niven/reth-v2.0.0:
  chore(deps): switch okx/reth tip to storage v2 static file fix
  fix(deps): rebase alloy-rpc-types-engine fork onto 1.8.3 to restore CVE patch
  chore(deps): patch reth to okx fork and bump optimism submodule
  chore: fmt
Replace Runtime::test() (test helper with 2 worker threads, 2 rayon
threads, 16 max blocking tasks) with a production-grade Runtime built
via RuntimeBuilder using RuntimeConfig::default() and attached to the
existing #[tokio::main] handle. This gives full CPU-derived rayon pool
sizing for import/export/gen-genesis/legacy-migrate operations without
spawning a second tokio runtime.

Addresses PR #270 review comment on bin/tools/main.rs:63.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Align workspace deps to reth v2.1.0 (okx/reth@044b173):

- reth-primitives-traits 0.1.0 → 0.3.0
- revm 36.0.0 → 38.0.0; op-revm 17.0.0 → 19.0.0
- alloy-evm 0.30.0 → 0.33.0; alloy-op-evm 0.30.0 → 0.31.0
- alloy-hardforks 0.4.5 → 0.4.7
- alloy-{consensus,contract,eips,genesis,json-rpc,network,provider,
  rpc-client,rpc-types-{beacon,engine,eth},serde,signer-local,
  transport,transport-http} 1.8.2 → 2.0.0
- tokio 1.44.2 → 1.51.1; tokio-tungstenite 0.26.2 → 0.28.0
- metrics 0.24.1 → 0.24.0
- README badge: reth v2.0.0 → v2.1.0

Drop the local alloy-rpc-types-engine fork: the v1.8.3 jsonwebtoken
CVE patch is no longer needed since alloy-rpc-types-engine 2.0.0
ships with jsonwebtoken 10.3.0 upstream.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- alloy-consensus 2.0 added block_access_list_hash and slot_number to
  Header; populate as None in flashblocks builder and assemble paths
  (matches op-reth/crates/evm/src/build.rs)
- alloy-rpc-types-eth 2.0 added block_timestamp to TransactionInfo;
  use ..Default::default() to match reth v2.1.0 RPC convention
- alloy-network 2.0 split TransactionRequest::build off into the
  NetworkTransactionBuilder trait; import it in e2e test utils
- revm 38 deprecated ExecutionResult::gas_used / Gas::used after the
  EIP-8037 state-gas split; switch flashblocks builder, context, and
  handler to tx_gas_used()

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* niven/reth-v2.0.0:
  fix(deps): adapt to alloy 2.0 / revm 38 API changes
  chore(deps): bump xlayer-reth to reth v2.1.0
  chore: update optimism submodule to opnode v1.17.0
  fix(tools): use production runtime in xlayer-reth-tools CLI
@Vui-Chee Vui-Chee changed the base branch from niven/reth-v2.0.0 to main May 2, 2026 13:01
* main:
  Merge : upgrade xlayer-reth with reth v2.1.0 and op-reth v2.1.0
@Vui-Chee Vui-Chee marked this pull request as ready for review May 4, 2026 02:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants