Add Go benchmark leg in release performance evaluation suite - #861
Merged
Conversation
cjonas9
changed the title
cjonas9
changed the title
cjonas9
marked this pull request as ready for review
July 15, 2026 18:07
There was a problem hiding this comment.
Pull request overview
Adds an EC2 performance-evaluation leg comparing Go benchmarks between the latest release and candidate revision.
Changes:
- Runs audited benchmarks and generates benchstat comparisons.
- Publishes benchmark reports and raw logs to S3.
- Consolidates common leg environment handling and fixes
BenchmarkTransactionFetch.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/load-test-coordinator.yml |
Adds the Go benchmark matrix leg. |
perf-eval/go-bench/runner/main.go |
Defines the benchmark runner command. |
perf-eval/go-bench/runner/instantiate.go |
Executes, compares, and publishes benchmarks. |
perf-eval/go-bench/run-go-bench.sh |
Bootstraps the benchmark leg. |
perf-eval/harness/harness.go |
Centralizes common environment values. |
perf-eval/harness/gather.go |
Refactors environment parsing. |
perf-eval/ingest-load-test/runner/instantiate.go |
Uses centralized environment values. |
perf-eval/ingest-load-test/run-load-test.sh |
Updates bootstrap documentation. |
perf-eval/backfill-test/runner/instantiate.go |
Uses centralized environment values. |
perf-eval/backfill-test/run-backfill.sh |
Updates bootstrap documentation. |
internal/db/transaction_test.go |
Corrects benchmark iteration setup. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
cjonas9
commented
Jul 17, 2026
Shaptic
reviewed
Jul 17, 2026
Contributor
🧪 Performance Evaluation Test #3Commit: ✅ Go endpoint benchmarks — verdict: okBaseline benchstat: baseline vs candidateRaw benchmark logs ( Performance Evaluation Test #2🧪 Performance Evaluation Test #2Commit: ✅ Apply-load ingestion — verdict: ok📈 Ingest load test —
|
| Profile | Ledgers | ms/ledger | p50 / p95 / p99 ms | max ms |
|---|---|---|---|---|
| load-test-ledgers-v27-oz | 1000 | 1197.764 | 1116.266 / 1627.458 / 1901.660 | 3007.699 |
| load-test-ledgers-v27-sac | 1000 | 1097.440 | 1107.470 / 1181.305 / 1240.057 | 1304.350 |
| load-test-ledgers-v27-soroswap | 1000 | 792.805 | 804.091 / 868.589 / 926.207 | 1034.326 |
| Metric | Value |
|---|---|
| Ledgers replayed | 3000 |
| Initial DB ledger count | 120960 |
| Throughput | 0.94 ledgers/sec |
| Elapsed wall-clock | 3188.602s |
| Ingest busy-time | 3088.009s (96.8% utilization) |
| Per-ledger p50 / p95 / p99 | 1053.457 / 1421.015 / 1731.059 ms |
| Golden DB fetch+decompress | 2435s |
| stellar-core | v27.0.0 |
| Workflow run | #29768807299-1 |
✅ Backfill ingestion — verdict: ok
⏳ Backfill ingestion — 00b3733b8796
| Metric | Value |
|---|---|
| Ledgers ingested | 17280 ([63551616 -> 63568895]) |
| Retention window | 17280 |
| Wall-clock | 39m29s |
| Ledgers/sec | 7.3 |
✅ Go endpoint benchmarks — verdict: ok
Baseline v27.1.1 (7e7128153191) vs candidate 00b3733b8796 — -benchmem -count=10, both refs sequentially on one box.
❌ Baseline bench run failed in github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/db; its rows lack a base column.
benchstat: baseline vs candidate
goos: linux
goarch: amd64
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/db
cpu: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
GetLedgerRange-8 967.4n ± 2% 959.1n ± 1% ~ (p=0.123 n=10)
BatchGetLedgers-8 759.2µ ± 1% 756.2µ ± 2% ~ (p=0.631 n=10)
TransactionFetch-8 59.59µ ± 1%
geomean 27.10µ 35.09µ -0.63%
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
GetLedgerRange-8 16.00 ± 0% 16.00 ± 0% ~ (p=1.000 n=10) ¹
BatchGetLedgers-8 466.7Ki ± 0% 466.7Ki ± 0% ~ (p=0.697 n=10)
TransactionFetch-8 11.71Ki ± 0%
geomean 2.700Ki 4.404Ki +0.00%
¹ all samples are equal
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
GetLedgerRange-8 4.000 ± 0% 4.000 ± 0% ~ (p=1.000 n=10) ¹
BatchGetLedgers-8 2.914k ± 0% 2.914k ± 0% ~ (p=1.000 n=10) ¹
TransactionFetch-8 162.0 ± 0%
geomean 108.0 123.6 +0.00%
¹ all samples are equal
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/feewindow
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
ComputeFeeDistribution/computeFeeDistribution-8 8.728µ ± 0% 8.611µ ± 0% -1.35% (p=0.000 n=10)
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 216.2µ ± 1% 218.3µ ± 1% ~ (p=0.089 n=10)
geomean 43.44µ 43.36µ -0.18%
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
ComputeFeeDistribution/computeFeeDistribution-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 565.4Ki ± 0% 565.4Ki ± 0% ~ (p=0.178 n=10)
geomean ² +0.00% ²
¹ all samples are equal
² summaries must be >0 to compute geomean
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
ComputeFeeDistribution/computeFeeDistribution-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 30.00 ± 0% 30.00 ± 0% ~ (p=1.000 n=10) ¹
geomean ² +0.00% ²
¹ all samples are equal
² summaries must be >0 to compute geomean
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/methods
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
GetEventsTopicFilters-8 2.588m ± 1% 2.600m ± 1% ~ (p=0.123 n=10)
GetEvents-8 164.0µ ± ∞ ¹ 162.7µ ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 1.323m ± ∞ ¹ 1.354m ± ∞ ¹ ~ (p=1.000 n=1) ²
JSONTransactions/JSON_format-8 8.518µ ± 1% 8.498µ ± 1% ~ (p=0.436 n=10)
JSONTransactions/XDR_format-8 2.830µ ± 0% 2.837µ ± 0% ~ (p=0.137 n=10)
GetProtocolVersion-8 43.84µ ± ∞ ¹ 43.50µ ± ∞ ¹ ~ (p=1.000 n=1) ²
geomean 91.66µ 91.86µ +0.21%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
GetEventsTopicFilters-8 187.5Ki ± 0% 187.4Ki ± 0% ~ (p=1.000 n=10)
GetEvents-8 42.97Ki ± ∞ ¹ 42.96Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 1.316Mi ± ∞ ¹ 1.316Mi ± ∞ ¹ ~ (p=1.000 n=1) ²
JSONTransactions/JSON_format-8 2.000Ki ± 0% 2.000Ki ± 0% ~ (p=1.000 n=10) ³
JSONTransactions/XDR_format-8 1.336Ki ± 0% 1.336Ki ± 0% ~ (p=1.000 n=10) ³
GetProtocolVersion-8 7.454Ki ± ∞ ¹ 7.453Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
geomean 24.50Ki 24.50Ki -0.01%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
³ all samples are equal
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
GetEventsTopicFilters-8 1.521k ± 0% 1.520k ± 0% ~ (p=0.984 n=10)
GetEvents-8 621.0 ± ∞ ¹ 621.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 5.322k ± ∞ ¹ 5.322k ± ∞ ¹ ~ (p=1.000 n=1) ²
JSONTransactions/JSON_format-8 25.00 ± 0% 25.00 ± 0% ~ (p=1.000 n=10) ²
JSONTransactions/XDR_format-8 23.00 ± 0% 23.00 ± 0% ~ (p=1.000 n=10) ²
GetProtocolVersion-8 120.0 ± ∞ ¹ 120.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
geomean 265.1 265.0 -0.02%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/preflight
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
GetPreflight-8 807.9µ ± 1% 804.8µ ± 1% ~ (p=0.247 n=10)
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
GetPreflight-8 52.06Ki ± 0% 52.06Ki ± 0% ~ (p=0.598 n=10)
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
GetPreflight-8 290.0 ± 0% 290.0 ± 0% ~ (p=1.000 n=10)
Raw benchmark logs (s3://stellar-rpc-ci-load-test/runs/29768807299/go-bench/): baseline.txt, benchstat.txt, candidate.txt
Performance Evaluation Test #1
🧪 Performance Evaluation Test #1
Commit: e4f105124fab (go-bench-test)
Run: https://github.com/stellar/stellar-rpc/actions/runs/29617565817
✅ Apply-load ingestion — verdict: ok
📈 Ingest load test — e4f1051
| Profile | Ledgers | ms/ledger | p50 / p95 / p99 ms | max ms |
|---|---|---|---|---|
| load-test-ledgers-v27-oz | 1000 | 1199.447 | 1116.930 / 1629.925 / 1913.659 | 3412.891 |
| load-test-ledgers-v27-sac | 1000 | 1097.635 | 1107.995 / 1182.107 / 1241.205 | 1300.731 |
| load-test-ledgers-v27-soroswap | 1000 | 792.496 | 803.648 / 869.068 / 922.988 | 1038.398 |
| Metric | Value |
|---|---|
| Ledgers replayed | 3000 |
| Initial DB ledger count | 120960 |
| Throughput | 0.94 ledgers/sec |
| Elapsed wall-clock | 3189.010s |
| Ingest busy-time | 3089.577s (96.9% utilization) |
| Per-ledger p50 / p95 / p99 | 1051.392 / 1421.570 / 1733.807 ms |
| Golden DB fetch+decompress | 2428s |
| stellar-core | v27.0.0 |
| Workflow run | #29617565817-1 |
✅ Backfill ingestion — verdict: ok
⏳ Backfill ingestion — e4f105124fab
| Metric | Value |
|---|---|
| Ledgers ingested | 17280 ([63507776 -> 63525055]) |
| Retention window | 17280 |
| Wall-clock | 40m52s |
| Ledgers/sec | 7.0 |
❌ Go endpoint benchmarks — verdict: none
No result object published (leg timed out or failed before publishing). See the run logs.
cjonas9
commented
Jul 20, 2026
Shaptic
approved these changes
Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds leg of performance evaluation suite that runs the existing Stellar RPC Go benchmarks and compares their outputs from version to version. In particular, this PR introduces a leg that is spawned by the coordinator as an EC2 instance, runs
BenchmarkGetLedgers,BenchmarkGetEventsTopicFilters,BenchmarkGetEvents,BenchmarkJSONTransactions,BenchmarkGetProtocolVersion,BenchmarkGetPreflight,BenchmarkComputeFeeDistribution,BenchmarkGetLedgerRange,BenchmarkBatchGetLedgers, andBenchmarkTransactionFetchon both the candidate ref and the previous release ref, captures the go benchmark logs andbenchstat, then ships the results to S3 for the coordinator.Also, just for posterity, some minor cleanup edits are made to DRY multiple legs that have shown themselves to require some common code.
Why
This is part of epic #706 and mostly seeks to ensure dependable benchmark execution, artifact capture, and comparison. Nothing else ever ran the benchmark suite, and having it as part of our pre-release evaluation is a sensible place to check.
Known limitations
N/A