Add blaster test for endpoint performance evaluation - #855
Conversation
🧪 Performance Evaluation Test #6Commit: ✅ Apply-load ingestion — verdict: ok📈 Ingest load test —
|
| Profile | Ledgers | ms/ledger | p50 / p95 / p99 ms | max ms |
|---|---|---|---|---|
| load-test-ledgers-v27-oz | 1000 | 1195.782 | 1114.654 / 1628.072 / 1912.494 | 2797.257 |
| load-test-ledgers-v27-sac | 1000 | 1097.863 | 1107.721 / 1180.668 / 1242.022 | 1304.395 |
| load-test-ledgers-v27-soroswap | 1000 | 792.883 | 804.143 / 868.145 / 925.887 | 1036.038 |
| Metric | Value |
|---|---|
| Ledgers replayed | 3000 |
| Initial DB ledger count | 120960 |
| Throughput | 0.94 ledgers/sec |
| Elapsed wall-clock | 3189.479s |
| Ingest busy-time | 3086.529s (96.8% utilization) |
| Per-ledger p50 / p95 / p99 | 1050.909 / 1424.810 / 1728.353 ms |
| Golden DB fetch+decompress | 2440s |
| stellar-core | v27.0.0 |
| Workflow run | #29766497768-1 |
✅ Backfill ingestion — verdict: ok
⏳ Backfill ingestion — 63f672405a6c
| Metric | Value |
|---|---|
| Ledgers ingested | 17280 ([63551296 -> 63568575]) |
| Retention window | 17280 |
| Wall-clock | 40m0s |
| Ledgers/sec | 7.2 |
✅ Endpoint load test — verdict: ok
🎯 Endpoint load test — 63f672405a6c
Serial blast per endpoint (ramp-up 2m, duration 3m, blaster 5b6e13046218) against the backfilled RPC (ledgers [63551606, 63568885], handoff wait 1458s).
| Endpoint | Target RPS | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | p99.9 (ms) |
|---|---|---|---|---|---|---|---|
| getFeeStats | 15 | 2075 | 0 (0.0%) | 0.6 | 14.1 | 107.5 | 167.9 |
| getHealth | 15 | 2388 | 0 (0.0%) | 0.6 | 14.1 | 98.2 | 166.4 |
| getLatestLedger | 15 | 2377 | 0 (0.0%) | 147.7 | 438.3 | 1110.0 | 1444.9 |
| getLedgerEntries | 15 | 2074 | 0 (0.0%) | 2.2 | 41.7 | 112.2 | 203.3 |
| getLedgers (limit=1) | 1 | 179 | 0 (0.0%) | 151.2 | 333.8 | 370.2 | 445.7 |
| getNetwork | 15 | 2387 | 0 (0.0%) | 26.0 | 37.8 | 100.5 | 170.6 |
| getTransaction | 15 | 2074 | 0 (0.0%) | 28.1 | 41.2 | 128.8 | 236.9 |
| getTransactions (limit=1) | 1 | 179 | 0 (0.0%) | 62.3 | 109.0 | 202.6 | 728.6 |
| getVersionInfo | 15 | 2376 | 0 (0.0%) | 26.0 | 39.1 | 99.0 | 165.4 |
Performance Evaluation Test #5
🧪 Performance Evaluation Test #5
Commit: 3e7d884fbc44 (blaster-test)
Run: https://github.com/stellar/stellar-rpc/actions/runs/29599452130
✅ Apply-load ingestion — verdict: ok
📈 Ingest load test — 3e7d884
| Profile | Ledgers | ms/ledger | p50 / p95 / p99 ms | max ms |
|---|---|---|---|---|
| load-test-ledgers-v27-oz | 1000 | 1199.569 | 1117.701 / 1625.765 / 1911.731 | 3348.337 |
| load-test-ledgers-v27-sac | 1000 | 1098.926 | 1108.832 / 1181.936 / 1239.448 | 1305.659 |
| load-test-ledgers-v27-soroswap | 1000 | 792.978 | 804.172 / 868.226 / 927.601 | 1039.809 |
| Metric | Value |
|---|---|
| Ledgers replayed | 3000 |
| Initial DB ledger count | 120960 |
| Throughput | 0.94 ledgers/sec |
| Elapsed wall-clock | 3189.397s |
| Ingest busy-time | 3091.473s (96.9% utilization) |
| Per-ledger p50 / p95 / p99 | 1051.685 / 1420.369 / 1732.817 ms |
| Golden DB fetch+decompress | 2429s |
| stellar-core | v27.0.0 |
| Workflow run | #29599452130-1 |
✅ Backfill ingestion — verdict: ok
⏳ Backfill ingestion — 3e7d884fbc44
| Metric | Value |
|---|---|
| Ledgers ingested | 17280 ([63504576 -> 63521855]) |
| Retention window | 17280 |
| Wall-clock | 42m34s |
| Ledgers/sec | 6.8 |
✅ Endpoint load test — verdict: ok
🎯 Endpoint load test — 3e7d884fbc44
Serial blast per endpoint (ramp-up 2m, duration 3m, blaster 73c0b3fac325) against the backfilled RPC (ledgers [63504947, 63522226], handoff wait 1624s).
| Endpoint | Target RPS | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | p99.9 (ms) |
|---|---|---|---|---|---|---|---|
| getFeeStats | 15 | 2075 | 0 (0.0%) | 0.8 | 16.2 | 99.8 | 154.9 |
| getHealth | 15 | 2387 | 0 (0.0%) | 0.7 | 16.3 | 100.2 | 186.5 |
| getLatestLedger | 15 | 2386 | 0 (0.0%) | 160.8 | 222.1 | 257.8 | 353.0 |
| getLedgerEntries | 15 | 2074 | 0 (0.0%) | 1.9 | 51.1 | 110.3 | 127.2 |
| getLedgers (limit=1) | 1 | 180 | 0 (0.0%) | 268.8 | 390.9 | 410.4 | 584.2 |
| getNetwork | 15 | 2388 | 0 (0.0%) | 26.1 | 43.8 | 101.0 | 151.6 |
| getTransaction | 15 | 2075 | 0 (0.0%) | 32.2 | 47.4 | 86.0 | 163.5 |
| getTransactions (limit=1) | 1 | 179 | 0 (0.0%) | 66.1 | 84.9 | 96.3 | 169.3 |
| getVersionInfo | 15 | 2372 | 0 (0.0%) | 25.3 | 42.8 | 83.3 | 150.5 |
Performance Evaluation Test #4
🧪 Performance Evaluation Test #4
Commit: abec2aee4b2a (blaster-test)
Run: https://github.com/stellar/stellar-rpc/actions/runs/29569224962
✅ Apply-load ingestion — verdict: ok
📈 Ingest load test — abec2ae
| Profile | Ledgers | ms/ledger | p50 / p95 / p99 ms | max ms |
|---|---|---|---|---|
| load-test-ledgers-v27-oz | 1000 | 1197.687 | 1115.520 / 1631.881 / 1914.386 | 2873.723 |
| load-test-ledgers-v27-sac | 1000 | 1097.612 | 1107.593 / 1182.116 / 1238.701 | 1304.028 |
| load-test-ledgers-v27-soroswap | 1000 | 791.723 | 802.842 / 867.478 / 924.323 | 1036.498 |
| Metric | Value |
|---|---|
| Ledgers replayed | 3000 |
| Initial DB ledger count | 120960 |
| Throughput | 0.94 ledgers/sec |
| Elapsed wall-clock | 3189.587s |
| Ingest busy-time | 3087.022s (96.8% utilization) |
| Per-ledger p50 / p95 / p99 | 1052.000 / 1416.615 / 1728.359 ms |
| Golden DB fetch+decompress | 2431s |
| stellar-core | v27.0.0 |
| Workflow run | #29569224962-1 |
✅ Backfill ingestion — verdict: ok
⏳ Backfill ingestion — abec2aee4b2a
| Metric | Value |
|---|---|
| Ledgers ingested | 17280 ([63499392 -> 63516671]) |
| Retention window | 17280 |
| Wall-clock | 41m3s |
| Ledgers/sec | 7.0 |
✅ Endpoint load test — verdict: ok
🎯 Endpoint load test — abec2aee4b2a
Serial blast per endpoint (ramp-up 2m, duration 3m, blaster f6085c38900f) against the backfilled RPC (ledgers [63499764, 63517043], handoff wait 1473s).
| Endpoint | Target RPS | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | p99.9 (ms) |
|---|---|---|---|---|---|---|---|
| getFeeStats | 15 | 2074 | 0 (0.0%) | 1.3 | 10.9 | 99.2 | 157.1 |
| getHealth | 15 | 2387 | 0 (0.0%) | 1.1 | 10.7 | 96.6 | 148.2 |
| getLatestLedger | 15 | 2387 | 0 (0.0%) | 157.7 | 230.0 | 308.0 | 379.4 |
| getLedgerEntries | 15 | 2074 | 0 (0.0%) | 3.7 | 47.4 | 106.0 | 133.2 |
| getLedgers (limit=1) | 1 | 179 | 0 (0.0%) | 175.6 | 378.9 | 405.2 | 470.3 |
| getNetwork | 15 | 2388 | 0 (0.0%) | 24.9 | 32.7 | 50.7 | 137.2 |
| getTransaction | 15 | 2075 | 0 (0.0%) | 31.9 | 42.0 | 119.8 | 197.0 |
| getTransactions (limit=1) | 1 | 180 | 0 (0.0%) | 65.6 | 81.5 | 147.7 | 308.2 |
| getVersionInfo | 15 | 2373 | 0 (0.0%) | 25.9 | 34.0 | 59.2 | 131.3 |
Performance Evaluation Test #3
🧪 Performance Evaluation Test #3
Commit: c6cd9d492ff1 (blaster-test)
Run: https://github.com/stellar/stellar-rpc/actions/runs/29283375076
✅ Apply-load ingestion — verdict: ok
📈 Ingest load test — c6cd9d4
| Profile | Ledgers | ms/ledger | p50 / p95 / p99 ms | max ms |
|---|---|---|---|---|
| load-test-ledgers-v27-oz | 1000 | 1206.737 | 1115.160 / 1648.651 / 1977.222 | 4634.191 |
| load-test-ledgers-v27-sac | 1000 | 1097.183 | 1106.764 / 1179.606 / 1233.021 | 1307.135 |
| load-test-ledgers-v27-soroswap | 1000 | 791.507 | 802.920 / 868.582 / 923.094 | 1030.999 |
| Metric | Value |
|---|---|
| Ledgers replayed | 3000 |
| Initial DB ledger count | 120960 |
| Throughput | 0.94 ledgers/sec |
| Elapsed wall-clock | 3196.746s |
| Ingest busy-time | 3095.426s (96.8% utilization) |
| Per-ledger p50 / p95 / p99 | 1050.728 / 1413.879 / 1843.836 ms |
| Golden DB fetch+decompress | 2430s |
| stellar-core | v27.0.0 |
| Workflow run | #29283375076-1 |
✅ Backfill ingestion — verdict: ok
⏳ Backfill ingestion — c6cd9d492ff1
| Metric | Value |
|---|---|
| Ledgers ingested | 8640 ([63454976 -> 63463615]) |
| Retention window | 8640 |
| Wall-clock | 20m57s |
| Ledgers/sec | 6.9 |
✅ Endpoint load test — verdict: ok
🎯 Endpoint load test — c6cd9d492ff1
Serial blast per endpoint (ramp-up 2m, duration 3m, blaster f6085c38900f) against the backfilled RPC (ledgers [63455276, 63463915], catchup 1458s).
| Endpoint | Target RPS | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | p99.9 (ms) |
|---|---|---|---|---|---|---|---|
| getEvents (limit=1) | 1 | 82 | 24 (29.3%) | 58.6 | 10010.6 | 10010.6 | 10018.8 |
| getFeeStats | 0 | 0 | 0 (0.0%) | 0.0 | 0.0 | 0.0 | 0.0 |
| getHealth | 0 | 0 | 0 (0.0%) | 0.0 | 0.0 | 0.0 | 0.0 |
| getLatestLedger | 0 | 0 | 0 (0.0%) | 0.0 | 0.0 | 0.0 | 0.0 |
| getLedgerEntries | 0 | 0 | 0 (0.0%) | 0.0 | 0.0 | 0.0 | 0.0 |
| getLedgers (limit=1) | 0 | 0 | 0 (0.0%) | 0.0 | 0.0 | 0.0 | 0.0 |
| getNetwork | 0 | 0 | 0 (0.0%) | 0.0 | 0.0 | 0.0 | 0.0 |
| getTransaction | 0 | 0 | 0 (0.0%) | 0.0 | 0.0 | 0.0 | 0.0 |
| getTransactions (limit=1) | 0 | 0 | 0 (0.0%) | 0.0 | 0.0 | 0.0 | 0.0 |
| getVersionInfo | 0 | 0 | 0 (0.0%) | 0.0 | 0.0 | 0.0 | 0.0 |
Performance Evaluation Test #2
🧪 Performance Evaluation Test #2
Commit: ee0f5cbf53b2 (blaster-test)
Run: https://github.com/stellar/stellar-rpc/actions/runs/29133837755
✅ Apply-load ingestion — verdict: ok
📈 Ingest load test — ee0f5cb
| Profile | Ledgers | ms/ledger | p50 / p95 / p99 ms | max ms |
|---|---|---|---|---|
| load-test-ledgers-v27-oz | 1000 | 1197.033 | 1115.398 / 1629.591 / 1911.283 | 3290.848 |
| load-test-ledgers-v27-sac | 1000 | 1096.878 | 1106.531 / 1177.999 / 1239.813 | 1303.008 |
| load-test-ledgers-v27-soroswap | 1000 | 791.185 | 802.614 / 867.703 / 919.590 | 1035.681 |
| Metric | Value |
|---|---|
| Ledgers replayed | 3000 |
| Initial DB ledger count | 120960 |
| Throughput | 0.94 ledgers/sec |
| Elapsed wall-clock | 3190.229s |
| Ingest busy-time | 3085.096s (96.7% utilization) |
| Per-ledger p50 / p95 / p99 | 1051.225 / 1414.314 / 1747.520 ms |
| Golden DB fetch+decompress | 2425s |
| stellar-core | v27.0.0 |
| Workflow run | #29133837755-1 |
✅ Backfill ingestion — verdict: ok
⏳ Backfill ingestion — ee0f5cbf53b2
| Metric | Value |
|---|---|
| Ledgers ingested | 8640 ([63412992 -> 63421631]) |
| Retention window | 8640 |
| Wall-clock | 17m46s |
| Ledgers/sec | 8.1 |
✅ Endpoint load test — verdict: ok
🎯 Endpoint load test — ee0f5cbf53b2
Serial blast per endpoint (ramp-up 2m, duration 3m, blaster 1c495b3c612c) against the backfilled RPC (ledgers [63413289, 63421928], catchup 1337s).
| Endpoint | Target RPS | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | p99.9 (ms) |
|---|---|---|---|---|---|---|---|
| getEvents (limit=1) | 1 | 179 | 7 (3.9%) | 18.3 | 7475.2 | 10002.4 | 10010.6 |
| getFeeStats | 15 | 2074 | 0 (0.0%) | 0.6 | 13.1 | 96.4 | 179.1 |
| getHealth | 15 | 2387 | 0 (0.0%) | 0.5 | 13.2 | 139.5 | 208.1 |
| getLatestLedger | 15 | 2385 | 0 (0.0%) | 136.1 | 180.5 | 267.3 | 354.3 |
| getLedgerEntries | 15 | 2075 | 0 (0.0%) | 1.4 | 47.6 | 108.4 | 162.9 |
| getLedgers (limit=1) | 1 | 180 | 0 (0.0%) | 160.1 | 359.9 | 378.1 | 382.5 |
| getNetwork | 15 | 2387 | 0 (0.0%) | 22.2 | 34.3 | 71.0 | 138.9 |
| getTransaction | 15 | 2075 | 0 (0.0%) | 27.2 | 38.6 | 105.3 | 191.7 |
| getTransactions (limit=1) | 1 | 179 | 0 (0.0%) | 61.5 | 77.2 | 105.9 | 166.7 |
| getVersionInfo | 15 | 2376 | 0 (0.0%) | 22.7 | 42.5 | 1012.2 | 2392.1 |
Performance Evaluation Test #1
🧪 Performance Evaluation Test #1
Commit: 6defac9a0c11 (blaster-test)
Run: https://github.com/stellar/stellar-rpc/actions/runs/29133628286
❌ Apply-load ingestion — verdict: fail
Load test failed before producing a result body.
❌ Backfill ingestion — verdict: fail
Load test failed before producing a result body.
❌ Endpoint load test — verdict: none
No result object published (leg timed out or failed before publishing). See the run logs.
| on: | ||
| push: | ||
| branches: [release/**] | ||
| # temporary scaffolding: before merge to main, replace with line |
There was a problem hiding this comment.
remove before merge->main
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
There was a problem hiding this comment.
Pull request overview
Adds a chained endpoint load-test leg to the release performance-evaluation workflow.
Changes:
- Keeps the backfill RPC instance serving for a successor EC2 job.
- Builds and runs
stellar-rpc-blaster, then reports endpoint latency statistics. - Extends shared harness and workflow coordination for chained instances.
Reviewed changes
Copilot reviewed 17 out of 18 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
go.mod |
Adds environment parsing dependency. |
go.sum |
Records dependency checksums. |
perf-eval/harness/leg.go |
Adds shared leg setup and publishing. |
perf-eval/harness/health.go |
Adds RPC health polling. |
perf-eval/harness/harness.go |
Adds boot-relative deadlines. |
endpoint-load-test/testdata/endpoints.toml |
Defines endpoint load profiles. |
endpoint-load-test/runner/results.go |
Summarizes blaster results. |
endpoint-load-test/runner/results_test.go |
Tests result rendering. |
endpoint-load-test/runner/main.go |
Defines the blaster runner command. |
endpoint-load-test/runner/instantiate.go |
Builds and executes the blaster. |
endpoint-load-test/run-blaster.sh |
Bootstraps the endpoint test. |
backfill-test/testdata/backfill-pubnet.toml.tmpl |
Makes the RPC bind address configurable. |
backfill-test/runner/serve.go |
Keeps the RPC instance serving. |
backfill-test/runner/main.go |
Documents handoff mode. |
backfill-test/runner/instantiate.go |
Adds serving lifecycle and refactors configuration. |
backfill-test/run-backfill.sh |
Documents serving behavior. |
.github/workflows/load-test-coordinator.yml |
Chains backfill and blaster jobs. |
.github/workflows/ec2-leg.yml |
Supports instance handoff and adoption. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| [endpoints.getLedgers] | ||
| rps = 1 | ||
| start_rps = 1 | ||
| limit = 1 |
There was a problem hiding this comment.
This is the way that it is because getEvents's response time currently is so variable depending on its query pattern that legitimately, we have it causing a whole blast to fail with rps=1. The query pattern will get an overhaul after we receive endpoint usage data and reconstruct it from that.
What
Adds an endpoint load-test (using
stellar-rpc-blaster) leg to the GHA perf-eval suite. Because we want to run this load test on a current, up-to-date RPC node, we chain the RPC this leg blasts at to the existing backfill leg (see #830). Generally, the flow of this established by this branch is as follows:rpsenvelopes are put here inendpoints.toml, which we should change once we have more data on what a realistic request load profile looks like. See Create RPC request body corpus builder #860.TARGET_RPC), and the blaster job's cleanup terminates both boxes (adopt_run_label). This helped me avoid any the side-channel coordination/"box rendezvous" state and logic my earlier attempts had.[DONE] Please also see stellar-rpc-blaster#40, which this branch treats as if it's been merged.
Why
This is part of epic #706 and seeks to add endpoint performance evaluation to our release evaluation suite. This is an especially pertinent component of RPC evaluation and allows us to ensure endpoint reliability between releases in a convenient and consistent manner.
Known limitations
N/A