Skip to content

Add blaster test for endpoint performance evaluation - #855

Open
cjonas9 wants to merge 128 commits into
mainfrom
blaster-test
Open

Add blaster test for endpoint performance evaluation#855
cjonas9 wants to merge 128 commits into
mainfrom
blaster-test

Conversation

@cjonas9

@cjonas9 cjonas9 commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

What

Adds an endpoint load-test (using stellar-rpc-blaster) leg to the GHA perf-eval suite. Because we want to run this load test on a current, up-to-date RPC node, we chain the RPC this leg blasts at to the existing backfill leg (see #830). Generally, the flow of this established by this branch is as follows:

  • The backfill box keeps its RPC daemon serving after publishing its result.
  • A second, newly spawned EC2 box builds stellar-rpc-blaster, waits out the server's catchup, runs a serial per-endpoint sweep, and puts the per-endpoint latency percentiles into the sticky PR comment.
  • Coordination between the blaster and the serving box is exclusively through GHA/the coordinator. The server's IP flows through a reusable-workflow output into the blaster box's user-data (TARGET_RPC), and the blaster job's cleanup terminates both boxes (adopt_run_label). This helped me avoid any the side-channel coordination/"box rendezvous" state and logic my earlier attempts had.

[DONE] Please also see stellar-rpc-blaster#40, which this branch treats as if it's been merged.

Why

This is part of epic #706 and seeks to add endpoint performance evaluation to our release evaluation suite. This is an especially pertinent component of RPC evaluation and allows us to ensure endpoint reliability between releases in a convenient and consistent manner.

Known limitations

N/A

cjonas9 added 30 commits May 8, 2026 21:32
@github-actions

github-actions Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

🧪 Performance Evaluation Test #6

Commit: 63f672405a6c (blaster-test)
Run: https://github.com/stellar/stellar-rpc/actions/runs/29766497768

✅ Apply-load ingestion — verdict: ok

📈 Ingest load test — 63f6724

Profile Ledgers ms/ledger p50 / p95 / p99 ms max ms
load-test-ledgers-v27-oz 1000 1195.782 1114.654 / 1628.072 / 1912.494 2797.257
load-test-ledgers-v27-sac 1000 1097.863 1107.721 / 1180.668 / 1242.022 1304.395
load-test-ledgers-v27-soroswap 1000 792.883 804.143 / 868.145 / 925.887 1036.038
Metric Value
Ledgers replayed 3000
Initial DB ledger count 120960
Throughput 0.94 ledgers/sec
Elapsed wall-clock 3189.479s
Ingest busy-time 3086.529s (96.8% utilization)
Per-ledger p50 / p95 / p99 1050.909 / 1424.810 / 1728.353 ms
Golden DB fetch+decompress 2440s
stellar-core v27.0.0
Workflow run #29766497768-1

✅ Backfill ingestion — verdict: ok

⏳ Backfill ingestion — 63f672405a6c

Metric Value
Ledgers ingested 17280 ([63551296 -> 63568575])
Retention window 17280
Wall-clock 40m0s
Ledgers/sec 7.2

✅ Endpoint load test — verdict: ok

🎯 Endpoint load test — 63f672405a6c

Serial blast per endpoint (ramp-up 2m, duration 3m, blaster 5b6e13046218) against the backfilled RPC (ledgers [63551606, 63568885], handoff wait 1458s).

Endpoint Target RPS Requests Errors p50 (ms) p95 (ms) p99 (ms) p99.9 (ms)
getFeeStats 15 2075 0 (0.0%) 0.6 14.1 107.5 167.9
getHealth 15 2388 0 (0.0%) 0.6 14.1 98.2 166.4
getLatestLedger 15 2377 0 (0.0%) 147.7 438.3 1110.0 1444.9
getLedgerEntries 15 2074 0 (0.0%) 2.2 41.7 112.2 203.3
getLedgers (limit=1) 1 179 0 (0.0%) 151.2 333.8 370.2 445.7
getNetwork 15 2387 0 (0.0%) 26.0 37.8 100.5 170.6
getTransaction 15 2074 0 (0.0%) 28.1 41.2 128.8 236.9
getTransactions (limit=1) 1 179 0 (0.0%) 62.3 109.0 202.6 728.6
getVersionInfo 15 2376 0 (0.0%) 26.0 39.1 99.0 165.4
Performance Evaluation Test #5

🧪 Performance Evaluation Test #5

Commit: 3e7d884fbc44 (blaster-test)
Run: https://github.com/stellar/stellar-rpc/actions/runs/29599452130

✅ Apply-load ingestion — verdict: ok

📈 Ingest load test — 3e7d884

Profile Ledgers ms/ledger p50 / p95 / p99 ms max ms
load-test-ledgers-v27-oz 1000 1199.569 1117.701 / 1625.765 / 1911.731 3348.337
load-test-ledgers-v27-sac 1000 1098.926 1108.832 / 1181.936 / 1239.448 1305.659
load-test-ledgers-v27-soroswap 1000 792.978 804.172 / 868.226 / 927.601 1039.809
Metric Value
Ledgers replayed 3000
Initial DB ledger count 120960
Throughput 0.94 ledgers/sec
Elapsed wall-clock 3189.397s
Ingest busy-time 3091.473s (96.9% utilization)
Per-ledger p50 / p95 / p99 1051.685 / 1420.369 / 1732.817 ms
Golden DB fetch+decompress 2429s
stellar-core v27.0.0
Workflow run #29599452130-1

✅ Backfill ingestion — verdict: ok

⏳ Backfill ingestion — 3e7d884fbc44

Metric Value
Ledgers ingested 17280 ([63504576 -> 63521855])
Retention window 17280
Wall-clock 42m34s
Ledgers/sec 6.8

✅ Endpoint load test — verdict: ok

🎯 Endpoint load test — 3e7d884fbc44

Serial blast per endpoint (ramp-up 2m, duration 3m, blaster 73c0b3fac325) against the backfilled RPC (ledgers [63504947, 63522226], handoff wait 1624s).

Endpoint Target RPS Requests Errors p50 (ms) p95 (ms) p99 (ms) p99.9 (ms)
getFeeStats 15 2075 0 (0.0%) 0.8 16.2 99.8 154.9
getHealth 15 2387 0 (0.0%) 0.7 16.3 100.2 186.5
getLatestLedger 15 2386 0 (0.0%) 160.8 222.1 257.8 353.0
getLedgerEntries 15 2074 0 (0.0%) 1.9 51.1 110.3 127.2
getLedgers (limit=1) 1 180 0 (0.0%) 268.8 390.9 410.4 584.2
getNetwork 15 2388 0 (0.0%) 26.1 43.8 101.0 151.6
getTransaction 15 2075 0 (0.0%) 32.2 47.4 86.0 163.5
getTransactions (limit=1) 1 179 0 (0.0%) 66.1 84.9 96.3 169.3
getVersionInfo 15 2372 0 (0.0%) 25.3 42.8 83.3 150.5
Performance Evaluation Test #4

🧪 Performance Evaluation Test #4

Commit: abec2aee4b2a (blaster-test)
Run: https://github.com/stellar/stellar-rpc/actions/runs/29569224962

✅ Apply-load ingestion — verdict: ok

📈 Ingest load test — abec2ae

Profile Ledgers ms/ledger p50 / p95 / p99 ms max ms
load-test-ledgers-v27-oz 1000 1197.687 1115.520 / 1631.881 / 1914.386 2873.723
load-test-ledgers-v27-sac 1000 1097.612 1107.593 / 1182.116 / 1238.701 1304.028
load-test-ledgers-v27-soroswap 1000 791.723 802.842 / 867.478 / 924.323 1036.498
Metric Value
Ledgers replayed 3000
Initial DB ledger count 120960
Throughput 0.94 ledgers/sec
Elapsed wall-clock 3189.587s
Ingest busy-time 3087.022s (96.8% utilization)
Per-ledger p50 / p95 / p99 1052.000 / 1416.615 / 1728.359 ms
Golden DB fetch+decompress 2431s
stellar-core v27.0.0
Workflow run #29569224962-1

✅ Backfill ingestion — verdict: ok

⏳ Backfill ingestion — abec2aee4b2a

Metric Value
Ledgers ingested 17280 ([63499392 -> 63516671])
Retention window 17280
Wall-clock 41m3s
Ledgers/sec 7.0

✅ Endpoint load test — verdict: ok

🎯 Endpoint load test — abec2aee4b2a

Serial blast per endpoint (ramp-up 2m, duration 3m, blaster f6085c38900f) against the backfilled RPC (ledgers [63499764, 63517043], handoff wait 1473s).

Endpoint Target RPS Requests Errors p50 (ms) p95 (ms) p99 (ms) p99.9 (ms)
getFeeStats 15 2074 0 (0.0%) 1.3 10.9 99.2 157.1
getHealth 15 2387 0 (0.0%) 1.1 10.7 96.6 148.2
getLatestLedger 15 2387 0 (0.0%) 157.7 230.0 308.0 379.4
getLedgerEntries 15 2074 0 (0.0%) 3.7 47.4 106.0 133.2
getLedgers (limit=1) 1 179 0 (0.0%) 175.6 378.9 405.2 470.3
getNetwork 15 2388 0 (0.0%) 24.9 32.7 50.7 137.2
getTransaction 15 2075 0 (0.0%) 31.9 42.0 119.8 197.0
getTransactions (limit=1) 1 180 0 (0.0%) 65.6 81.5 147.7 308.2
getVersionInfo 15 2373 0 (0.0%) 25.9 34.0 59.2 131.3
Performance Evaluation Test #3

🧪 Performance Evaluation Test #3

Commit: c6cd9d492ff1 (blaster-test)
Run: https://github.com/stellar/stellar-rpc/actions/runs/29283375076

✅ Apply-load ingestion — verdict: ok

📈 Ingest load test — c6cd9d4

Profile Ledgers ms/ledger p50 / p95 / p99 ms max ms
load-test-ledgers-v27-oz 1000 1206.737 1115.160 / 1648.651 / 1977.222 4634.191
load-test-ledgers-v27-sac 1000 1097.183 1106.764 / 1179.606 / 1233.021 1307.135
load-test-ledgers-v27-soroswap 1000 791.507 802.920 / 868.582 / 923.094 1030.999
Metric Value
Ledgers replayed 3000
Initial DB ledger count 120960
Throughput 0.94 ledgers/sec
Elapsed wall-clock 3196.746s
Ingest busy-time 3095.426s (96.8% utilization)
Per-ledger p50 / p95 / p99 1050.728 / 1413.879 / 1843.836 ms
Golden DB fetch+decompress 2430s
stellar-core v27.0.0
Workflow run #29283375076-1

✅ Backfill ingestion — verdict: ok

⏳ Backfill ingestion — c6cd9d492ff1

Metric Value
Ledgers ingested 8640 ([63454976 -> 63463615])
Retention window 8640
Wall-clock 20m57s
Ledgers/sec 6.9

✅ Endpoint load test — verdict: ok

🎯 Endpoint load test — c6cd9d492ff1

Serial blast per endpoint (ramp-up 2m, duration 3m, blaster f6085c38900f) against the backfilled RPC (ledgers [63455276, 63463915], catchup 1458s).

Endpoint Target RPS Requests Errors p50 (ms) p95 (ms) p99 (ms) p99.9 (ms)
getEvents (limit=1) 1 82 24 (29.3%) 58.6 10010.6 10010.6 10018.8
getFeeStats 0 0 0 (0.0%) 0.0 0.0 0.0 0.0
getHealth 0 0 0 (0.0%) 0.0 0.0 0.0 0.0
getLatestLedger 0 0 0 (0.0%) 0.0 0.0 0.0 0.0
getLedgerEntries 0 0 0 (0.0%) 0.0 0.0 0.0 0.0
getLedgers (limit=1) 0 0 0 (0.0%) 0.0 0.0 0.0 0.0
getNetwork 0 0 0 (0.0%) 0.0 0.0 0.0 0.0
getTransaction 0 0 0 (0.0%) 0.0 0.0 0.0 0.0
getTransactions (limit=1) 0 0 0 (0.0%) 0.0 0.0 0.0 0.0
getVersionInfo 0 0 0 (0.0%) 0.0 0.0 0.0 0.0
Performance Evaluation Test #2

🧪 Performance Evaluation Test #2

Commit: ee0f5cbf53b2 (blaster-test)
Run: https://github.com/stellar/stellar-rpc/actions/runs/29133837755

✅ Apply-load ingestion — verdict: ok

📈 Ingest load test — ee0f5cb

Profile Ledgers ms/ledger p50 / p95 / p99 ms max ms
load-test-ledgers-v27-oz 1000 1197.033 1115.398 / 1629.591 / 1911.283 3290.848
load-test-ledgers-v27-sac 1000 1096.878 1106.531 / 1177.999 / 1239.813 1303.008
load-test-ledgers-v27-soroswap 1000 791.185 802.614 / 867.703 / 919.590 1035.681
Metric Value
Ledgers replayed 3000
Initial DB ledger count 120960
Throughput 0.94 ledgers/sec
Elapsed wall-clock 3190.229s
Ingest busy-time 3085.096s (96.7% utilization)
Per-ledger p50 / p95 / p99 1051.225 / 1414.314 / 1747.520 ms
Golden DB fetch+decompress 2425s
stellar-core v27.0.0
Workflow run #29133837755-1

✅ Backfill ingestion — verdict: ok

⏳ Backfill ingestion — ee0f5cbf53b2

Metric Value
Ledgers ingested 8640 ([63412992 -> 63421631])
Retention window 8640
Wall-clock 17m46s
Ledgers/sec 8.1

✅ Endpoint load test — verdict: ok

🎯 Endpoint load test — ee0f5cbf53b2

Serial blast per endpoint (ramp-up 2m, duration 3m, blaster 1c495b3c612c) against the backfilled RPC (ledgers [63413289, 63421928], catchup 1337s).

Endpoint Target RPS Requests Errors p50 (ms) p95 (ms) p99 (ms) p99.9 (ms)
getEvents (limit=1) 1 179 7 (3.9%) 18.3 7475.2 10002.4 10010.6
getFeeStats 15 2074 0 (0.0%) 0.6 13.1 96.4 179.1
getHealth 15 2387 0 (0.0%) 0.5 13.2 139.5 208.1
getLatestLedger 15 2385 0 (0.0%) 136.1 180.5 267.3 354.3
getLedgerEntries 15 2075 0 (0.0%) 1.4 47.6 108.4 162.9
getLedgers (limit=1) 1 180 0 (0.0%) 160.1 359.9 378.1 382.5
getNetwork 15 2387 0 (0.0%) 22.2 34.3 71.0 138.9
getTransaction 15 2075 0 (0.0%) 27.2 38.6 105.3 191.7
getTransactions (limit=1) 1 179 0 (0.0%) 61.5 77.2 105.9 166.7
getVersionInfo 15 2376 0 (0.0%) 22.7 42.5 1012.2 2392.1
Performance Evaluation Test #1

🧪 Performance Evaluation Test #1

Commit: 6defac9a0c11 (blaster-test)
Run: https://github.com/stellar/stellar-rpc/actions/runs/29133628286

❌ Apply-load ingestion — verdict: fail

Load test failed before producing a result body.

❌ Backfill ingestion — verdict: fail

Load test failed before producing a result body.

❌ Endpoint load test — verdict: none

No result object published (leg timed out or failed before publishing). See the run logs.

on:
push:
branches: [release/**]
# temporary scaffolding: before merge to main, replace with line

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove before merge->main

@socket-security

socket-security Bot commented Jul 17, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedgolang/​github.com/​caarlos0/​env/​v11@​v11.4.196100100100100

View full report

@cjonas9
cjonas9 marked this pull request as ready for review July 17, 2026 16:44
Copilot AI review requested due to automatic review settings July 17, 2026 16:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a chained endpoint load-test leg to the release performance-evaluation workflow.

Changes:

  • Keeps the backfill RPC instance serving for a successor EC2 job.
  • Builds and runs stellar-rpc-blaster, then reports endpoint latency statistics.
  • Extends shared harness and workflow coordination for chained instances.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
go.mod Adds environment parsing dependency.
go.sum Records dependency checksums.
perf-eval/harness/leg.go Adds shared leg setup and publishing.
perf-eval/harness/health.go Adds RPC health polling.
perf-eval/harness/harness.go Adds boot-relative deadlines.
endpoint-load-test/testdata/endpoints.toml Defines endpoint load profiles.
endpoint-load-test/runner/results.go Summarizes blaster results.
endpoint-load-test/runner/results_test.go Tests result rendering.
endpoint-load-test/runner/main.go Defines the blaster runner command.
endpoint-load-test/runner/instantiate.go Builds and executes the blaster.
endpoint-load-test/run-blaster.sh Bootstraps the endpoint test.
backfill-test/testdata/backfill-pubnet.toml.tmpl Makes the RPC bind address configurable.
backfill-test/runner/serve.go Keeps the RPC instance serving.
backfill-test/runner/main.go Documents handoff mode.
backfill-test/runner/instantiate.go Adds serving lifecycle and refactors configuration.
backfill-test/run-backfill.sh Documents serving behavior.
.github/workflows/load-test-coordinator.yml Chains backfill and blaster jobs.
.github/workflows/ec2-leg.yml Supports instance handoff and adoption.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/load-test-coordinator.yml
Comment on lines +39 to +42
[endpoints.getLedgers]
rps = 1
start_rps = 1
limit = 1

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the way that it is because getEvents's response time currently is so variable depending on its query pattern that legitimately, we have it causing a whole blast to fail with rps=1. The query pattern will get an overhaul after we receive endpoint usage data and reconstruct it from that.

Comment thread .github/workflows/ec2-leg.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Release eval: job for measuring request load Release eval: job to compare endpoint benchmarks

3 participants