Test benchmarking-platform changes #9960

sarahchen6 · 2025-11-13T21:13:51Z

What Does This Do

Test benchmarking-platform changes -- in particular the changes relate to running benchmarks against the latest shared commit of a PR's branch and master, rather than against the latest commit on master which is what we're doing right now.

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any useful labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

datadog-official · 2025-11-13T21:30:42Z

🎯 Code Coverage
• Patch Coverage: 100.00%
• Total Coverage: 59.65% (+0.01%)

View detailed report

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: d28bd84 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

pr-commenter · 2025-11-14T23:24:53Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	sarahchen6/test-benchmark-changes
git_commit_date	1763128431	1763155712
git_commit_sha	`b436de8`	`d28bd84`
release_version	1.56.0-SNAPSHOT~b436de808f	1.56.0-SNAPSHOT~d28bd84ee2

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1763162610	1763162610
ci_job_id	1237071039	1237071039
ci_pipeline_id	82536699	82536699
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-itx6i0h2 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-itx6i0h2 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 7 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.56.0-SNAPSHOT~d28bd84ee2, baseline=1.56.0-SNAPSHOT~b436de808f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.103 s) : 0, 1102923
Total [baseline] (10.763 s) : 0, 10763109
Agent [candidate] (1.099 s) : 0, 1099001
Total [candidate] (10.709 s) : 0, 10708676
section appsec
Agent [baseline] (1.287 s) : 0, 1286860
Total [baseline] (11.005 s) : 0, 11005277
Agent [candidate] (1.279 s) : 0, 1279381
Total [candidate] (11.106 s) : 0, 11106443
section iast
Agent [baseline] (1.238 s) : 0, 1238254
Total [baseline] (11.215 s) : 0, 11214871
Agent [candidate] (1.238 s) : 0, 1238385
Total [candidate] (11.187 s) : 0, 11186513
section profiling
Agent [baseline] (1.228 s) : 0, 1228175
Total [baseline] (11.049 s) : 0, 11049140
Agent [candidate] (1.236 s) : 0, 1236304
Total [candidate] (11.068 s) : 0, 11067602

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.103 s	-
Agent	appsec	1.287 s	183.937 ms (16.7%)
Agent	iast	1.238 s	135.331 ms (12.3%)
Agent	profiling	1.228 s	125.252 ms (11.4%)
Total	tracing	10.763 s	-
Total	appsec	11.005 s	242.167 ms (2.2%)
Total	iast	11.215 s	451.762 ms (4.2%)
Total	profiling	11.049 s	286.031 ms (2.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.099 s	-
Agent	appsec	1.279 s	180.38 ms (16.4%)
Agent	iast	1.238 s	139.384 ms (12.7%)
Agent	profiling	1.236 s	137.303 ms (12.5%)
Total	tracing	10.709 s	-
Total	appsec	11.106 s	397.766 ms (3.7%)
Total	iast	11.187 s	477.836 ms (4.5%)
Total	profiling	11.068 s	358.925 ms (3.4%)

gantt
    title petclinic - break down per module: candidate=1.56.0-SNAPSHOT~d28bd84ee2, baseline=1.56.0-SNAPSHOT~b436de808f

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.468 ms) : 0, 1468
crashtracking [candidate] (1.452 ms) : 0, 1452
BytebuddyAgent [baseline] (709.461 ms) : 0, 709461
BytebuddyAgent [candidate] (705.624 ms) : 0, 705624
GlobalTracer [baseline] (248.032 ms) : 0, 248032
GlobalTracer [candidate] (247.984 ms) : 0, 247984
AppSec [baseline] (32.163 ms) : 0, 32163
AppSec [candidate] (32.226 ms) : 0, 32226
Debugger [baseline] (64.295 ms) : 0, 64295
Debugger [candidate] (64.236 ms) : 0, 64236
Remote Config [baseline] (637.189 µs) : 0, 637
Remote Config [candidate] (631.812 µs) : 0, 632
Telemetry [baseline] (8.223 ms) : 0, 8223
Telemetry [candidate] (8.239 ms) : 0, 8239
Flare Poller [baseline] (3.693 ms) : 0, 3693
Flare Poller [candidate] (3.718 ms) : 0, 3718
section appsec
crashtracking [baseline] (1.471 ms) : 0, 1471
crashtracking [candidate] (1.447 ms) : 0, 1447
BytebuddyAgent [baseline] (734.084 ms) : 0, 734084
BytebuddyAgent [candidate] (729.837 ms) : 0, 729837
GlobalTracer [baseline] (241.143 ms) : 0, 241143
GlobalTracer [candidate] (240.287 ms) : 0, 240287
IAST [baseline] (25.076 ms) : 0, 25076
IAST [candidate] (24.797 ms) : 0, 24797
AppSec [baseline] (175.075 ms) : 0, 175075
AppSec [candidate] (174.151 ms) : 0, 174151
Debugger [baseline] (61.89 ms) : 0, 61890
Debugger [candidate] (61.004 ms) : 0, 61004
Remote Config [baseline] (782.152 µs) : 0, 782
Remote Config [candidate] (661.662 µs) : 0, 662
Telemetry [baseline] (8.377 ms) : 0, 8377
Telemetry [candidate] (8.346 ms) : 0, 8346
Flare Poller [baseline] (3.788 ms) : 0, 3788
Flare Poller [candidate] (3.86 ms) : 0, 3860
section iast
crashtracking [baseline] (1.45 ms) : 0, 1450
crashtracking [candidate] (1.475 ms) : 0, 1475
BytebuddyAgent [baseline] (829.569 ms) : 0, 829569
BytebuddyAgent [candidate] (829.77 ms) : 0, 829770
GlobalTracer [baseline] (238.353 ms) : 0, 238353
GlobalTracer [candidate] (238.168 ms) : 0, 238168
IAST [baseline] (29.206 ms) : 0, 29206
IAST [candidate] (30.805 ms) : 0, 30805
AppSec [baseline] (32.162 ms) : 0, 32162
AppSec [candidate] (30.521 ms) : 0, 30521
Debugger [baseline] (60.905 ms) : 0, 60905
Debugger [candidate] (61.056 ms) : 0, 61056
Remote Config [baseline] (541.179 µs) : 0, 541
Remote Config [candidate] (547.209 µs) : 0, 547
Telemetry [baseline] (7.693 ms) : 0, 7693
Telemetry [candidate] (7.764 ms) : 0, 7764
Flare Poller [baseline] (3.463 ms) : 0, 3463
Flare Poller [candidate] (3.476 ms) : 0, 3476
section profiling
crashtracking [baseline] (1.46 ms) : 0, 1460
crashtracking [candidate] (1.452 ms) : 0, 1452
BytebuddyAgent [baseline] (729.899 ms) : 0, 729899
BytebuddyAgent [candidate] (734.678 ms) : 0, 734678
GlobalTracer [baseline] (222.125 ms) : 0, 222125
GlobalTracer [candidate] (223.041 ms) : 0, 223041
AppSec [baseline] (32.248 ms) : 0, 32248
AppSec [candidate] (32.494 ms) : 0, 32494
Debugger [baseline] (62.891 ms) : 0, 62891
Debugger [candidate] (63.864 ms) : 0, 63864
Remote Config [baseline] (661.785 µs) : 0, 662
Remote Config [candidate] (668.747 µs) : 0, 669
Telemetry [baseline] (7.986 ms) : 0, 7986
Telemetry [candidate] (8.133 ms) : 0, 8133
Flare Poller [baseline] (3.817 ms) : 0, 3817
Flare Poller [candidate] (3.843 ms) : 0, 3843
ProfilingAgent [baseline] (97.415 ms) : 0, 97415
ProfilingAgent [candidate] (98.01 ms) : 0, 98010
Profiling [baseline] (98.015 ms) : 0, 98015
Profiling [candidate] (98.601 ms) : 0, 98601

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.56.0-SNAPSHOT~d28bd84ee2, baseline=1.56.0-SNAPSHOT~b436de808f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.097 s) : 0, 1097097
Total [baseline] (8.823 s) : 0, 8823463
Agent [candidate] (1.098 s) : 0, 1097919
Total [candidate] (8.81 s) : 0, 8810410
section iast
Agent [baseline] (1.236 s) : 0, 1236283
Total [baseline] (9.536 s) : 0, 9535819
Agent [candidate] (1.244 s) : 0, 1243954
Total [candidate] (9.566 s) : 0, 9566055

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.097 s	-
Agent	iast	1.236 s	139.186 ms (12.7%)
Total	tracing	8.823 s	-
Total	iast	9.536 s	712.356 ms (8.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.098 s	-
Agent	iast	1.244 s	146.035 ms (13.3%)
Total	tracing	8.81 s	-
Total	iast	9.566 s	755.645 ms (8.6%)

gantt
    title insecure-bank - break down per module: candidate=1.56.0-SNAPSHOT~d28bd84ee2, baseline=1.56.0-SNAPSHOT~b436de808f

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.465 ms) : 0, 1465
crashtracking [candidate] (1.449 ms) : 0, 1449
BytebuddyAgent [baseline] (704.725 ms) : 0, 704725
BytebuddyAgent [candidate] (705.136 ms) : 0, 705136
GlobalTracer [baseline] (247.702 ms) : 0, 247702
GlobalTracer [candidate] (248.232 ms) : 0, 248232
AppSec [baseline] (32.153 ms) : 0, 32153
AppSec [candidate] (32.5 ms) : 0, 32500
Debugger [baseline] (63.707 ms) : 0, 63707
Debugger [candidate] (63.364 ms) : 0, 63364
Remote Config [baseline] (628.978 µs) : 0, 629
Remote Config [candidate] (632.212 µs) : 0, 632
Telemetry [baseline] (8.249 ms) : 0, 8249
Telemetry [candidate] (8.114 ms) : 0, 8114
Flare Poller [baseline] (3.642 ms) : 0, 3642
Flare Poller [candidate] (3.631 ms) : 0, 3631
section iast
crashtracking [baseline] (1.462 ms) : 0, 1462
crashtracking [candidate] (1.474 ms) : 0, 1474
BytebuddyAgent [baseline] (829.336 ms) : 0, 829336
BytebuddyAgent [candidate] (834.709 ms) : 0, 834709
GlobalTracer [baseline] (237.374 ms) : 0, 237374
GlobalTracer [candidate] (238.645 ms) : 0, 238645
IAST [baseline] (27.72 ms) : 0, 27720
IAST [candidate] (27.82 ms) : 0, 27820
AppSec [baseline] (33.869 ms) : 0, 33869
AppSec [candidate] (33.848 ms) : 0, 33848
Debugger [baseline] (60.006 ms) : 0, 60006
Debugger [candidate] (60.54 ms) : 0, 60540
Remote Config [baseline] (534.179 µs) : 0, 534
Remote Config [candidate] (559.045 µs) : 0, 559
Telemetry [baseline] (7.646 ms) : 0, 7646
Telemetry [candidate] (7.768 ms) : 0, 7768
Flare Poller [baseline] (3.406 ms) : 0, 3406
Flare Poller [candidate] (3.539 ms) : 0, 3539

Load

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	sarahchen6/test-benchmark-changes
git_commit_date	1763128431	1763155712
git_commit_sha	`b436de8`	`d28bd84`
release_version	1.56.0-SNAPSHOT~b436de808f	1.56.0-SNAPSHOT~d28bd84ee2

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1763160583	1763160583
ci_job_id	1237048877	1237048877
ci_pipeline_id	82536699	82536699
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-1wl3vmd4 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-1wl3vmd4 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.56.0-SNAPSHOT~d28bd84ee2, baseline=1.56.0-SNAPSHOT~b436de808f
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.746 s) : 14746000, 14746000
.   : milestone, 14746000,
appsec (14.702 s) : 14702000, 14702000
.   : milestone, 14702000,
iast (18.04 s) : 18040000, 18040000
.   : milestone, 18040000,
iast_GLOBAL (17.94 s) : 17940000, 17940000
.   : milestone, 17940000,
profiling (15.02 s) : 15020000, 15020000
.   : milestone, 15020000,
tracing (14.704 s) : 14704000, 14704000
.   : milestone, 14704000,
section candidate
no_agent (15.541 s) : 15541000, 15541000
.   : milestone, 15541000,
appsec (14.581 s) : 14581000, 14581000
.   : milestone, 14581000,
iast (17.995 s) : 17995000, 17995000
.   : milestone, 17995000,
iast_GLOBAL (18.086 s) : 18086000, 18086000
.   : milestone, 18086000,
profiling (15.416 s) : 15416000, 15416000
.   : milestone, 15416000,
tracing (14.482 s) : 14482000, 14482000
.   : milestone, 14482000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.746 s [14.746 s, 14.746 s]	-
appsec	14.702 s [14.702 s, 14.702 s]	-44.0 ms (-0.3%)
iast	18.04 s [18.04 s, 18.04 s]	3.294 s (22.3%)
iast_GLOBAL	17.94 s [17.94 s, 17.94 s]	3.194 s (21.7%)
profiling	15.02 s [15.02 s, 15.02 s]	274.0 ms (1.9%)
tracing	14.704 s [14.704 s, 14.704 s]	-42.0 ms (-0.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.541 s [15.541 s, 15.541 s]	-
appsec	14.581 s [14.581 s, 14.581 s]	-960.0 ms (-6.2%)
iast	17.995 s [17.995 s, 17.995 s]	2.454 s (15.8%)
iast_GLOBAL	18.086 s [18.086 s, 18.086 s]	2.545 s (16.4%)
profiling	15.416 s [15.416 s, 15.416 s]	-125.0 ms (-0.8%)
tracing	14.482 s [14.482 s, 14.482 s]	-1.059 s (-6.8%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.56.0-SNAPSHOT~d28bd84ee2, baseline=1.56.0-SNAPSHOT~b436de808f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.477 ms) : 1466, 1489
.   : milestone, 1477,
appsec (3.713 ms) : 3494, 3932
.   : milestone, 3713,
iast (2.207 ms) : 2144, 2271
.   : milestone, 2207,
iast_GLOBAL (2.252 ms) : 2189, 2316
.   : milestone, 2252,
profiling (2.054 ms) : 2004, 2105
.   : milestone, 2054,
tracing (2.034 ms) : 1984, 2083
.   : milestone, 2034,
section candidate
no_agent (1.476 ms) : 1464, 1487
.   : milestone, 1476,
appsec (3.719 ms) : 3501, 3938
.   : milestone, 3719,
iast (2.214 ms) : 2151, 2277
.   : milestone, 2214,
iast_GLOBAL (2.261 ms) : 2197, 2325
.   : milestone, 2261,
profiling (2.061 ms) : 2010, 2112
.   : milestone, 2061,
tracing (2.031 ms) : 1981, 2081
.   : milestone, 2031,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.477 ms [1.466 ms, 1.489 ms]	-
appsec	3.713 ms [3.494 ms, 3.932 ms]	2.236 ms (151.3%)
iast	2.207 ms [2.144 ms, 2.271 ms]	729.79 µs (49.4%)
iast_GLOBAL	2.252 ms [2.189 ms, 2.316 ms]	775.083 µs (52.5%)
profiling	2.054 ms [2.004 ms, 2.105 ms]	577.08 µs (39.1%)
tracing	2.034 ms [1.984 ms, 2.083 ms]	556.402 µs (37.7%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.464 ms, 1.487 ms]	-
appsec	3.719 ms [3.501 ms, 3.938 ms]	2.243 ms (152.0%)
iast	2.214 ms [2.151 ms, 2.277 ms]	738.089 µs (50.0%)
iast_GLOBAL	2.261 ms [2.197 ms, 2.325 ms]	785.278 µs (53.2%)
profiling	2.061 ms [2.01 ms, 2.112 ms]	585.241 µs (39.7%)
tracing	2.031 ms [1.981 ms, 2.081 ms]	555.061 µs (37.6%)

Test benchmarking-platform changes

c61dca2

sarahchen6 added the tag: do not merge Do not merge changes label Nov 13, 2025

sarahchen6 added 3 commits November 14, 2025 14:44

Merge branch 'master' into sarahchen6/test-benchmark-changes

2a5f1dc

Install authanywhere in Dockerfile

4e1bd3f

Try adding in CI instead

d28bd84

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Test benchmarking-platform changes #9960

Test benchmarking-platform changes #9960

Uh oh!

sarahchen6 commented Nov 13, 2025

Uh oh!

datadog-official bot commented Nov 13, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Nov 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Test benchmarking-platform changes #9960

Are you sure you want to change the base?

Test benchmarking-platform changes #9960

Uh oh!

Conversation

sarahchen6 commented Nov 13, 2025

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

datadog-official bot commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter bot commented Nov 14, 2025

Benchmarks

Startup

Parameters

Summary

Load

Dacapo

Parameters

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datadog-official bot commented Nov 13, 2025 •

edited

Loading