[system test] [perf] Performance perf report into PR #12124

see-quick · 2025-11-11T12:14:21Z

Type of change

Enhancement / new feature

Description

This PR adds a perf report, which might look like this [1] (ignore those numbers; important is the format). This report would always be appended to the PR as a message when performance tests are triggered.

Currently, I am using only one agent (i.e., ubuntu-latest) for testing purposes on my fork, but the plan is to use both x64 and arm-based agents.

[1] - see-quick#15 (comment)

Checklist

Write tests
Make sure all tests pass
Update documentation

codecov · 2025-11-11T13:22:14Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.78%. Comparing base (4503119) to head (42a8dd1).
⚠️ Report is 19 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12124      +/-   ##
============================================
- Coverage     74.81%   74.78%   -0.04%     
- Complexity     6619     6624       +5     
============================================
  Files           377      377              
  Lines         25329    25349      +20     
  Branches       3394     3398       +4     
============================================
+ Hits          18951    18957       +6     
- Misses         4991     5007      +16     
+ Partials       1387     1385       -2

see 12 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

.github/workflows/system-tests.yml

...test/src/test/java/io/strimzi/systemtest/performance/UserOperatorScalabilityPerformance.java

...est/src/test/java/io/strimzi/systemtest/performance/TopicOperatorScalabilityPerformance.java

...test/src/test/java/io/strimzi/systemtest/performance/UserOperatorScalabilityPerformance.java

.github/actions/systemtests/perf-report/action.yml

.github/actions/utils/add-to-summary/action.yml

.github/actions/utils/create-warning/action.yml

.github/actions/utils/list-artifacts/action.yml

Frawless · 2025-11-13T18:12:49Z

.github/actions/systemtests/run-perf-report/action.yml

+
+          ${{ steps.generate_report.outputs.summary }}
+
+    - name: Add performance report to job summary


This is cool, maybe there could be some usage for common sts execution as well?

see-quick · 2025-11-18T15:08:45Z

/gha run pipeline=performance

github-actions · 2025-11-18T15:09:26Z

⏳ System test verification started: link

The following 2 job(s) will be executed:

performance-amd64 (oracle-vm-8cpu-32gb-x86-64)
performance-arm64 (oracle-vm-8cpu-32gb-arm64)

Tests will start after successful build completion.

github-actions · 2025-11-18T16:52:49Z

❌ System test verification failed: link

see-quick · 2025-11-19T09:14:26Z

/gha run pipeline=performance

github-actions · 2025-11-19T09:15:05Z

⏳ System test verification started: link

The following 2 job(s) will be executed:

performance-amd64 (oracle-vm-8cpu-32gb-x86-64)
performance-arm64 (oracle-vm-8cpu-32gb-arm64)

Tests will start after successful build completion.

Signed-off-by: see-quick <[email protected]>

github-actions · 2025-11-21T10:16:05Z

🎉 System test verification passed: link

Signed-off-by: see-quick <[email protected]>

see-quick · 2025-11-24T11:47:50Z

/gha run pipeline=performance

github-actions · 2025-11-24T11:48:21Z

⏳ System test verification started: link

The following 2 job(s) will be executed:

performance-amd64 (oracle-vm-8cpu-32gb-x86-64)
performance-arm64 (oracle-vm-8cpu-32gb-arm64)

Tests will start after successful build completion.

github-actions · 2025-11-24T13:24:46Z

🎉 System test verification passed: link

Signed-off-by: see-quick <[email protected]>

.github/tests/workflows/perf-report-template.yaml

.github/actions/systemtests/run-perf-report/action.yml

Frawless · 2025-11-25T07:46:07Z

.github/actions/systemtests/run-perf-report/generate-report.js

+/**
+ * Find the latest timestamped results directory
+ */
+function findLatestResultsDir(baseDir) {


there are multiple results from one run that you need to find the latest?

Also, this was mainly for local testing where I had multiple directories and runs:

├── 2025-11-20-15-57-41 │ └── user-operator │ └── latencyUseCase │ ├── users-1000-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-2000-tp--cache--bq--bb-100-bt-100-utp- │ └── users-3000-tp--cache--bq--bb-100-bt-100-utp- ├── 2025-11-20-16-10-15 │ └── user-operator │ └── latencyUseCase │ ├── users-1000-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-2000-tp--cache--bq--bb-100-bt-100-utp- │ └── users-3000-tp--cache--bq--bb-100-bt-100-utp- ├── 2025-11-20-16-35-19 │ └── user-operator │ └── latencyUseCase │ ├── users-1000-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-2000-tp--cache--bq--bb-100-bt-100-utp- │ └── users-3000-tp--cache--bq--bb-100-bt-100-utp- ├── 2025-11-20-17-02-58 │ └── user-operator │ └── latencyUseCase │ ├── users-1000-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-2000-tp--cache--bq--bb-100-bt-100-utp- │ └── users-3000-tp--cache--bq--bb-100-bt-100-utp- ├── 2025-11-20-17-15-26 │ └── user-operator │ └── latencyUseCase │ ├── users-1000-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-2000-tp--cache--bq--bb-100-bt-100-utp- │ └── users-3000-tp--cache--bq--bb-100-bt-100-utp- ├── 2025-11-21-09-50-34 │ ├── topic-operator │ │ └── scalabilityUseCase │ │ ├── max-batch-size-100-max-linger-time-100-with-clients-false-number-of-topics-2 │ │ └── max-batch-size-100-max-linger-time-100-with-clients-false-number-of-topics-3 │ └── user-operator │ ├── latencyUseCase │ │ ├── users-10-tp--cache--bq--bb-100-bt-100-utp- │ │ ├── users-20-tp--cache--bq--bb-100-bt-100-utp- │ │ └── users-30-tp--cache--bq--bb-100-bt-100-utp- │ └── scalabilityUseCase │ ├── users-10-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-12-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-14-tp--cache--bq--bb-100-bt-100-utp- │ └── users-16-tp--cache--bq--bb-100-bt-100-utp- └── 2025-11-24-12-16-22 └── user-operator └── latencyUseCase ├── users-1000-tp--cache--bq--bb-100-bt-100-utp- ├── users-2000-tp--cache--bq--bb-100-bt-100-utp- └── users-3000-tp--cache--bq--bb-100-bt-100-utp-

so it will always pick latest. I think I can re-name that method (to something like that findTimestampedResultsDir``)? But in general per one architecture there should be just only ONE timestamp`. If more architectures are run then we handle it differently... (we don't care about that here).

.github/actions/systemtests/run-perf-report/generate-report.js

.github/tests/workflows/perf-report-template.yaml

Signed-off-by: see-quick <[email protected]>

see-quick · 2025-12-01T10:14:35Z

/gha run pipeline=performance

github-actions · 2025-12-01T10:15:15Z

⏳ System test verification started: link

The following 2 job(s) will be executed:

performance-amd64 (oracle-vm-8cpu-32gb-x86-64)
performance-arm64 (oracle-vm-8cpu-32gb-arm64)

Tests will start after successful build completion.

github-actions · 2025-12-01T11:31:41Z

🎉 System test verification passed: link

Frawless

The changes are fine form my POV. I am not sure with 450 lines of js to generate the report especially when we have something similar already in the tests. AFAIU it is not trivial to re-use it. I wonder what others think about it. Otherwise I am fine with it as long as it will be used :)

see-quick · 2025-12-02T14:00:05Z

Basically, there are two approaches... the first one is (where we would have +500 LOC to merge two or more results from architectures):

Performance Test Results

Test Run: 2025-11-12 20:47

Topic Operator

Use Case: scalabilityUseCase

Configuration:

MAX QUEUE SIZE: 2147483647
MAX BATCH SIZE (ms): 100
MAX BATCH LINGER (ms): 100
PROCESS TYPE: TOPIC-CONCURRENT

Results:

#	NUMBER OF TOPICS	NUMBER OF EVENTS	Reconciliation interval (ms) [AMD64]	Reconciliation interval (ms) [ARM64]
1	2	8	10229	10167
2	32	98	11505	10504
3	125	375	42367	41202
4	250	750	74596	72361

User Operator

Use Case: scalabilityUseCase

Configuration:

WORK_QUEUE_SIZE: 1024
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Reconciliation interval (ms) [AMD64]	Reconciliation interval (ms) [ARM64]
1	10	10472	10797
2	100	33036	33851
3	200	54940	55822
4	500	133782	135474

Use Case: latencyUseCase

Configuration:

WORK_QUEUE_SIZE: 2048
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Min Latency (ms) [AMD64]	Min Latency (ms) [ARM64]	Max Latency (ms) [AMD64]	Max Latency (ms) [ARM64]	Average Latency (ms) [AMD64]	Average Latency (ms) [ARM64]	P50 Latency (ms) [AMD64]	P50 Latency (ms) [ARM64]	P95 Latency (ms) [AMD64]	P95 Latency (ms) [ARM64]	P99 Latency (ms) [AMD64]	P99 Latency (ms) [ARM64]
1	110	12	14	69	103	27.78	25.03	26	22	39	45	54	79
2	200	11	15	75	66	29.93	27.13	28	25	48	45	75	59
3	300	10	12	61	98	26.0	25.53	26	23	41	41	50	89

see-quick · 2025-12-02T14:01:06Z

or second one have (but with no need + 450LOC of javascript) => but with price with a lot of redudancy from my POV:

Performance Test Results

AMD64

Test Run: 2025-11-18 14:43

Topic Operator

Use Case: latencyUseCase

Configuration:

WORK_QUEUE_SIZE: 2048
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Min Latency (ms)	Max Latency (ms)	Average Latency (ms)	P50 Latency (ms)	P95 Latency (ms)	P99 Latency (ms)
1	110	14	75	27.51	24	51	73
2	300	14	51	24.24	22	38	49
3	200	13	72	25.56	23	44	57

Use Case: scalabilityUseCase

Configuration:

MAX QUEUE SIZE: 2147483647
MAX BATCH SIZE (ms): 100
MAX BATCH LINGER (ms): 100
PROCESS TYPE: TOPIC-CONCURRENT

Results:

#	NUMBER OF TOPICS	NUMBER OF EVENTS	Reconciliation interval (ms)
1	2	8	10130
2	32	98	10441
3	125	375	41369
4	250	750	71977

User Operator

Use Case: latencyUseCase

Configuration:

WORK_QUEUE_SIZE: 2048
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Min Latency (ms)	Max Latency (ms)	Average Latency (ms)	P50 Latency (ms)	P95 Latency (ms)	P99 Latency (ms)
1	110	14	75	27.51	24	51	73
2	200	13	72	25.56	23	44	57
3	300	14	51	24.24	22	38	49

Use Case: scalabilityUseCase

Configuration:

WORK_QUEUE_SIZE: 1024
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Reconciliation interval (ms)
1	10	10641
2	100	33107
3	200	54157
4	500	134269

ARCH (another arch)

Test Run: 2025-11-18 14:49

Topic Operator

Use Case: latencyUseCase

Configuration:

WORK_QUEUE_SIZE: 2048
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Min Latency (ms)	Max Latency (ms)	Average Latency (ms)	P50 Latency (ms)	P95 Latency (ms)	P99 Latency (ms)
1	110	14	75	27.51	24	51	73
2	300	14	51	24.24	22	38	49
3	200	13	72	25.56	23	44	57

Use Case: scalabilityUseCase

Configuration:

MAX QUEUE SIZE: 2147483647
MAX BATCH SIZE (ms): 100
MAX BATCH LINGER (ms): 100
PROCESS TYPE: TOPIC-CONCURRENT

Results:

#	NUMBER OF TOPICS	NUMBER OF EVENTS	Reconciliation interval (ms)
1	2	8	10130
2	32	98	10441
3	125	375	41369
4	250	750	71977

User Operator

Use Case: latencyUseCase

Configuration:

WORK_QUEUE_SIZE: 2048
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Min Latency (ms)	Max Latency (ms)	Average Latency (ms)	P50 Latency (ms)	P95 Latency (ms)	P99 Latency (ms)
1	110	14	75	27.51	24	51	73
2	200	13	72	25.56	23	44	57
3	300	14	51	24.24	22	38	49

Use Case: scalabilityUseCase

Configuration:

WORK_QUEUE_SIZE: 1024
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Reconciliation interval (ms)
1	10	10641
2	100	33107
3	200	54157
4	500	134269

im-konge · 2025-12-04T11:31:47Z

@see-quick I approved it, but I agree that we should have something for the regular STs as well - it would be nice to have.
Regarding the 450 lines of code in the JS generator -> yeah I think that it's maybe "overkill", but I'm not sure if there is some easier way and have just one thing that generates the report?

im-konge · 2025-12-04T11:44:56Z

For the JS generator - my idea is following:

What about creating two reports in Java rather than just one? The first one would be the full one, second one would be shorter - basically what you are doing in the JS generator. It would be named like some-report-full and some-report-short and then you would be able to just pick up these short reports and connect them in one MD/output in the GHA. For that you can definitely have some few lines in JS - just to take report from the amd64 pipeline (add heading for that architecture + then just paste the short report without any customization) and arm64 pipeline. The generator of the report in Java anyway has all the info, so "duplicating" it to shorter report would be trivial. Plus you would be able to re-use that in other CIs or even locally if you would run the tests - so output would be this short report, if user would like to see more, you can paste some path to the full report file.

WDYT?

see-quick · 2025-12-04T14:37:05Z

For the JS generator - my idea is following:

What about creating two reports in Java rather than just one? The first one would be the full one, second one would be shorter - basically what you are doing in the JS generator. It would be named like some-report-full and some-report-short and then you would be able to just pick up these short reports and connect them in one MD/output in the GHA. For that you can definitely have some few lines in JS - just to take report from the amd64 pipeline (add heading for that architecture + then just paste the short report without any customization) and arm64 pipeline. The generator of the report in Java anyway has all the info, so "duplicating" it to shorter report would be trivial. Plus you would be able to re-use that in other CIs or even locally if you would run the tests - so output would be this short report, if user would like to see more, you can paste some path to the full report file.

WDYT?

Hmmm, interesting and how do you imagine this short/compact report? If, for instance, this is a full-report:

## Performance Test Results

## AMD64

**Test Run:** `2025-11-18 14:43`

## Topic Operator

**Use Case:** latencyUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 2048
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Min Latency (ms) | Max Latency (ms) | Average Latency (ms) | P50 Latency (ms) | P95 Latency (ms) | P99 Latency (ms) |
|---|---|---|---|---|---|---|---|
| 1 | 110 | 14 | 75 | 27.51 | 24 | 51 | 73 |
| 2 | 300 | 14 | 51 | 24.24 | 22 | 38 | 49 |
| 3 | 200 | 13 | 72 | 25.56 | 23 | 44 | 57 |

**Use Case:** scalabilityUseCase

**Configuration:**
- MAX QUEUE SIZE: 2147483647
- MAX BATCH SIZE (ms): 100
- MAX BATCH LINGER (ms): 100
- PROCESS TYPE: TOPIC-CONCURRENT

**Results:**

| # | NUMBER OF TOPICS | NUMBER OF EVENTS | Reconciliation interval (ms) |
|---|---|---|---|
| 1 | 2 | 8 | 10130 |
| 2 | 32 | 98 | 10441 |
| 3 | 125 | 375 | 41369 |
| 4 | 250 | 750 | 71977 |

## User Operator

**Use Case:** latencyUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 2048
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Min Latency (ms) | Max Latency (ms) | Average Latency (ms) | P50 Latency (ms) | P95 Latency (ms) | P99 Latency (ms) |
|---|---|---|---|---|---|---|---|
| 1 | 110 | 14 | 75 | 27.51 | 24 | 51 | 73 |
| 2 | 200 | 13 | 72 | 25.56 | 23 | 44 | 57 |
| 3 | 300 | 14 | 51 | 24.24 | 22 | 38 | 49 |

**Use Case:** scalabilityUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 1024
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Reconciliation interval (ms) |
|---|---|---|
| 1 | 10 | 10641 |
| 2 | 100 | 33107 |
| 3 | 200 | 54157 |
| 4 | 500 | 134269 |

maybe something like this?

Performance Test Results

Test Run: 2025-11-18 14:43 | Architecture: AMD64

Topic Operator

Scalability (2-250 topics): Reconciliation 10s-72s

User Operator

Latency (110-300 users): Avg 24-28ms, P99 49-73ms
Scalability (10-500 users): Reconciliation 10s-134s

This is what you mean by shorter or any different?

im-konge · 2025-12-04T14:49:58Z

For the JS generator - my idea is following:
What about creating two reports in Java rather than just one? The first one would be the full one, second one would be shorter - basically what you are doing in the JS generator. It would be named like some-report-full and some-report-short and then you would be able to just pick up these short reports and connect them in one MD/output in the GHA. For that you can definitely have some few lines in JS - just to take report from the amd64 pipeline (add heading for that architecture + then just paste the short report without any customization) and arm64 pipeline. The generator of the report in Java anyway has all the info, so "duplicating" it to shorter report would be trivial. Plus you would be able to re-use that in other CIs or even locally if you would run the tests - so output would be this short report, if user would like to see more, you can paste some path to the full report file.
WDYT?

Hmmm, interesting and how do you imagine this short/compact report? If, for instance, this is a full-report:
## Performance Test Results

## AMD64

**Test Run:** `2025-11-18 14:43`

## Topic Operator

**Use Case:** latencyUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 2048
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Min Latency (ms) | Max Latency (ms) | Average Latency (ms) | P50 Latency (ms) | P95 Latency (ms) | P99 Latency (ms) |
|---|---|---|---|---|---|---|---|
| 1 | 110 | 14 | 75 | 27.51 | 24 | 51 | 73 |
| 2 | 300 | 14 | 51 | 24.24 | 22 | 38 | 49 |
| 3 | 200 | 13 | 72 | 25.56 | 23 | 44 | 57 |

**Use Case:** scalabilityUseCase

**Configuration:**
- MAX QUEUE SIZE: 2147483647
- MAX BATCH SIZE (ms): 100
- MAX BATCH LINGER (ms): 100
- PROCESS TYPE: TOPIC-CONCURRENT

**Results:**

| # | NUMBER OF TOPICS | NUMBER OF EVENTS | Reconciliation interval (ms) |
|---|---|---|---|
| 1 | 2 | 8 | 10130 |
| 2 | 32 | 98 | 10441 |
| 3 | 125 | 375 | 41369 |
| 4 | 250 | 750 | 71977 |

## User Operator

**Use Case:** latencyUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 2048
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Min Latency (ms) | Max Latency (ms) | Average Latency (ms) | P50 Latency (ms) | P95 Latency (ms) | P99 Latency (ms) |
|---|---|---|---|---|---|---|---|
| 1 | 110 | 14 | 75 | 27.51 | 24 | 51 | 73 |
| 2 | 200 | 13 | 72 | 25.56 | 23 | 44 | 57 |
| 3 | 300 | 14 | 51 | 24.24 | 22 | 38 | 49 |

**Use Case:** scalabilityUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 1024
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Reconciliation interval (ms) |
|---|---|---|
| 1 | 10 | 10641 |
| 2 | 100 | 33107 |
| 3 | 200 | 54157 |
| 4 | 500 | 134269 |
maybe something like this?

Performance Test Results

Test Run: 2025-11-18 14:43 | Architecture: AMD64

Topic Operator

Scalability (2-250 topics): Reconciliation 10s-72s

User Operator

Latency (110-300 users): Avg 24-28ms, P99 49-73ms

Scalability (10-500 users): Reconciliation 10s-134s

This is what you mean by shorter or any different?

TBH, now I'm a bit confused.
I thought that there is bigger report in Java, so you are re-generating the report from GHA using that JS generator, which takes less info from that "full report".

see-quick · 2025-12-04T15:00:17Z

For the JS generator - my idea is following:
What about creating two reports in Java rather than just one? The first one would be the full one, second one would be shorter - basically what you are doing in the JS generator. It would be named like some-report-full and some-report-short and then you would be able to just pick up these short reports and connect them in one MD/output in the GHA. For that you can definitely have some few lines in JS - just to take report from the amd64 pipeline (add heading for that architecture + then just paste the short report without any customization) and arm64 pipeline. The generator of the report in Java anyway has all the info, so "duplicating" it to shorter report would be trivial. Plus you would be able to re-use that in other CIs or even locally if you would run the tests - so output would be this short report, if user would like to see more, you can paste some path to the full report file.
WDYT?

Hmmm, interesting and how do you imagine this short/compact report? If, for instance, this is a full-report:
## Performance Test Results

## AMD64

**Test Run:** `2025-11-18 14:43`

## Topic Operator

**Use Case:** latencyUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 2048
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Min Latency (ms) | Max Latency (ms) | Average Latency (ms) | P50 Latency (ms) | P95 Latency (ms) | P99 Latency (ms) |
|---|---|---|---|---|---|---|---|
| 1 | 110 | 14 | 75 | 27.51 | 24 | 51 | 73 |
| 2 | 300 | 14 | 51 | 24.24 | 22 | 38 | 49 |
| 3 | 200 | 13 | 72 | 25.56 | 23 | 44 | 57 |

**Use Case:** scalabilityUseCase

**Configuration:**
- MAX QUEUE SIZE: 2147483647
- MAX BATCH SIZE (ms): 100
- MAX BATCH LINGER (ms): 100
- PROCESS TYPE: TOPIC-CONCURRENT

**Results:**

| # | NUMBER OF TOPICS | NUMBER OF EVENTS | Reconciliation interval (ms) |
|---|---|---|---|
| 1 | 2 | 8 | 10130 |
| 2 | 32 | 98 | 10441 |
| 3 | 125 | 375 | 41369 |
| 4 | 250 | 750 | 71977 |

## User Operator

**Use Case:** latencyUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 2048
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Min Latency (ms) | Max Latency (ms) | Average Latency (ms) | P50 Latency (ms) | P95 Latency (ms) | P99 Latency (ms) |
|---|---|---|---|---|---|---|---|
| 1 | 110 | 14 | 75 | 27.51 | 24 | 51 | 73 |
| 2 | 200 | 13 | 72 | 25.56 | 23 | 44 | 57 |
| 3 | 300 | 14 | 51 | 24.24 | 22 | 38 | 49 |

**Use Case:** scalabilityUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 1024
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Reconciliation interval (ms) |
|---|---|---|
| 1 | 10 | 10641 |
| 2 | 100 | 33107 |
| 3 | 200 | 54157 |
| 4 | 500 | 134269 |
maybe something like this?

Performance Test Results

Test Run: 2025-11-18 14:43 | Architecture: AMD64

Topic Operator

Scalability (2-250 topics): Reconciliation 10s-72s

User Operator

Latency (110-300 users): Avg 24-28ms, P99 49-73ms

Scalability (10-500 users): Reconciliation 10s-134s

This is what you mean by shorter or any different?
TBH, now I'm a bit confused. I thought that there is bigger report in Java, so you are re-generating the report from GHA using that JS generator, which takes less info from that "full report".

That report is per-architecture, because each architecture generates it.

so you are re-generating the report from GHA using that JS generator

I am just merging architectures to reduce redundancy over those both architectures so instead of having

## Performance Test Results

## AMD64

**Test Run:** `2025-11-18 14:43`

## Topic Operator

**Use Case:** latencyUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 2048
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Min Latency (ms) | Max Latency (ms) | Average Latency (ms) | P50 Latency (ms) | P95 Latency (ms) | P99 Latency (ms) |
|---|---|---|---|---|---|---|---|
| 1 | 110 | 14 | 75 | 27.51 | 24 | 51 | 73 |
| 2 | 300 | 14 | 51 | 24.24 | 22 | 38 | 49 |
| 3 | 200 | 13 | 72 | 25.56 | 23 | 44 | 57 |

**Use Case:** scalabilityUseCase

**Configuration:**
- MAX QUEUE SIZE: 2147483647
- MAX BATCH SIZE (ms): 100
- MAX BATCH LINGER (ms): 100
- PROCESS TYPE: TOPIC-CONCURRENT

**Results:**

| # | NUMBER OF TOPICS | NUMBER OF EVENTS | Reconciliation interval (ms) |
|---|---|---|---|
| 1 | 2 | 8 | 10130 |
| 2 | 32 | 98 | 10441 |
| 3 | 125 | 375 | 41369 |
| 4 | 250 | 750 | 71977 |

## User Operator

**Use Case:** latencyUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 2048
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Min Latency (ms) | Max Latency (ms) | Average Latency (ms) | P50 Latency (ms) | P95 Latency (ms) | P99 Latency (ms) |
|---|---|---|---|---|---|---|---|
| 1 | 110 | 14 | 75 | 27.51 | 24 | 51 | 73 |
| 2 | 200 | 13 | 72 | 25.56 | 23 | 44 | 57 |
| 3 | 300 | 14 | 51 | 24.24 | 22 | 38 | 49 |

**Use Case:** scalabilityUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 1024
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Reconciliation interval (ms) |
|---|---|---|
| 1 | 10 | 10641 |
| 2 | 100 | 33107 |
| 3 | 200 | 54157 |
| 4 | 500 | 134269 |

## Another arch

with the same fields but different valeus (because diff architecture)

and with that JavaScript generator, I just merged those two arches, which eventually gives you this:

## Performance Test Results

**Test Run:** `2025-11-12 20:47`

## Topic Operator

**Use Case:** scalabilityUseCase

**Configuration:**
- MAX QUEUE SIZE: 2147483647
- MAX BATCH SIZE (ms): 100
- MAX BATCH LINGER (ms): 100
- PROCESS TYPE: TOPIC-CONCURRENT

**Results:**

| # | NUMBER OF TOPICS | NUMBER OF EVENTS | Reconciliation interval (ms) [AMD64] | Reconciliation interval (ms) [ARM64] |
|---|---|---|---|---|
| 1 | 2 | 8 | 10229 | 10167 |
| 2 | 32 | 98 | 11505 | 10504 |
| 3 | 125 | 375 | 42367 | 41202 |
| 4 | 250 | 750 | 74596 | 72361 |

## User Operator

**Use Case:** scalabilityUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 1024
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Reconciliation interval (ms) [AMD64] | Reconciliation interval (ms) [ARM64] |
|---|---|---|---|
| 1 | 10 | 10472 | 10797 |
| 2 | 100 | 33036 | 33851 |
| 3 | 200 | 54940 | 55822 |
| 4 | 500 | 133782 | 135474 |

**Use Case:** latencyUseCase

**Configuration:**
- WORK_QUEUE_SIZE: 2048
- BATCH_MAXIMUM_BLOCK_SIZE: 100
- BATCH_MAXIMUM_BLOCK_TIME_MS: 100

**Results:**

| # | NUMBER OF KAFKA USERS | Min Latency (ms) [AMD64] | Min Latency (ms) [ARM64] | Max Latency (ms) [AMD64] | Max Latency (ms) [ARM64] | Average Latency (ms) [AMD64] | Average Latency (ms) [ARM64] | P50 Latency (ms) [AMD64] | P50 Latency (ms) [ARM64] | P95 Latency (ms) [AMD64] | P95 Latency (ms) [ARM64] | P99 Latency (ms) [AMD64] | P99 Latency (ms) [ARM64] |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 110 | 12 | 14 | 69 | 103 | 27.78 | 25.03 | 26 | 22 | 39 | 45 | 54 | 79 |
| 2 | 200 | 11 | 15 | 75 | 66 | 29.93 | 27.13 | 28 | 25 | 48 | 45 | 75 | 59 |
| 3 | 300 | 10 | 12 | 61 | 98 | 26.0 | 25.53 | 26 | 23 | 41 | 41 | 50 | 89 |

---

im-konge · 2025-12-05T13:13:35Z

After offline conversation with @see-quick (and finally understanding what is going on) I think that there is not much we can actually do about it - in case that we really want to have the reports as the comments. I had another idea to just leave comment saying that the jobs were (or weren't) successful and then links to the reports (generated by the Java generator) for each of the architecture, so anyone can have a look at them without much post-processing.

But if we want to keep it this way, I guess we will have to live with the 450 LoC on the JS generator.

see-quick added this to the 0.50.0 milestone Nov 11, 2025

see-quick self-assigned this Nov 11, 2025

see-quick added System tests performance labels Nov 11, 2025

see-quick requested review from a team and Frawless November 11, 2025 12:15

see-quick force-pushed the perf-report-st branch from 6ef694a to aaf5e72 Compare November 13, 2025 09:23

Frawless reviewed Nov 13, 2025

View reviewed changes

see-quick force-pushed the perf-report-st branch from 7fa2d0c to cea51a6 Compare November 18, 2025 15:02

see-quick added 15 commits November 19, 2025 10:41

[system test] [perf] Performance perf report into PR

54ea727

Signed-off-by: see-quick <[email protected]>

correct doc after rebase

e24a841

Signed-off-by: see-quick <[email protected]>

also add oracle agent instead ubuntu latest

af0ee3a

Signed-off-by: see-quick <[email protected]>

check 2 agents

83de883

Signed-off-by: see-quick <[email protected]>

correct javadoc

975dc3d

Signed-off-by: see-quick <[email protected]>

move logic stuff to actions and also lower the creation

cdee595

Signed-off-by: see-quick <[email protected]>

do not merge perf tests within workflow

9425e2f

Signed-off-by: see-quick <[email protected]>

some fix

bfd5afa

Signed-off-by: see-quick <[email protected]>

have epthemeral storage

60af3fc

Signed-off-by: see-quick <[email protected]>

also imports

2288b8a

Signed-off-by: see-quick <[email protected]>

try both arches to display

1df141f

Signed-off-by: see-quick <[email protected]>

[system test] [perf] Performance perf report into PR

76c1da9

Signed-off-by: see-quick <[email protected]>

refactor a bit

6beb9a6

Signed-off-by: see-quick <[email protected]>

[system test] [perf] Performance perf report into PR 4

eee3dd3

Signed-off-by: see-quick <[email protected]>

fix actions

44c0c95

Signed-off-by: see-quick <[email protected]>

see-quick added 2 commits November 24, 2025 09:51

keep IN: and OUT: parameters within results-table.txt

84ee02b

Signed-off-by: see-quick <[email protected]>

adjust cpus specific to GHA runners

ab130c5

Signed-off-by: see-quick <[email protected]>

see-quick added 2 commits November 24, 2025 14:29

modify input tests in perf report

4423660

Signed-off-by: see-quick <[email protected]>

update perf-report expected results

c5a05b1

Signed-off-by: see-quick <[email protected]>

see-quick requested a review from Frawless November 24, 2025 13:41

Frawless reviewed Nov 25, 2025

View reviewed changes

see-quick added 5 commits December 1, 2025 10:22

Jakub review + fix tests + re-name .txt to .md + refactor generation.js

a347815

Signed-off-by: see-quick <[email protected]>

remove ref from run-perf-report step

a4f356e

Signed-off-by: see-quick <[email protected]>

udpate workflow of tests

c5c416c

Signed-off-by: see-quick <[email protected]>

try

55fcd5f

Signed-off-by: see-quick <[email protected]>

a few leftover and ready to review again

42a8dd1

Signed-off-by: see-quick <[email protected]>

Frawless approved these changes Dec 2, 2025

View reviewed changes

see-quick requested review from a team and im-konge December 2, 2025 13:55

im-konge approved these changes Dec 4, 2025

View reviewed changes


		${{ steps.generate_report.outputs.summary }}

		- name: Add performance report to job summary

[system test] [perf] Performance perf report into PR #12124

Are you sure you want to change the base?

[system test] [perf] Performance perf report into PR #12124

Uh oh!

Conversation

see-quick commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change

Description

Checklist

Uh oh!

codecov bot commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Frawless Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

see-quick commented Nov 18, 2025

Uh oh!

github-actions bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 18, 2025

Uh oh!

see-quick commented Nov 19, 2025

Uh oh!

github-actions bot commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 21, 2025

Uh oh!

see-quick commented Nov 24, 2025

Uh oh!

github-actions bot commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Frawless Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

see-quick Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

see-quick commented Dec 1, 2025

Uh oh!

github-actions bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 1, 2025

Uh oh!

Frawless left a comment

Choose a reason for hiding this comment

Uh oh!

see-quick commented Dec 2, 2025

Performance Test Results

Topic Operator

User Operator

Uh oh!

see-quick commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Test Results

AMD64

Topic Operator

User Operator

see-quick commented Nov 11, 2025 •

edited

Loading

codecov bot commented Nov 11, 2025 •

edited

Loading

github-actions bot commented Nov 18, 2025 •

edited

Loading

github-actions bot commented Nov 19, 2025 •

edited

Loading

github-actions bot commented Nov 24, 2025 •

edited

Loading

github-actions bot commented Dec 1, 2025 •

edited

Loading

see-quick commented Dec 2, 2025 •

edited

Loading

see-quick commented Dec 4, 2025 •

edited

Loading