Add incremental upload stress test - #1908
Conversation
GitHub Actions
Waiting
| pub chunk: usize, | ||
| } | ||
|
|
||
| impl Worker for IncrementalWriter { |
There was a problem hiding this comment.
This is exactly the same as our existing Writer? we should re-use to avoid duplication.
There was a problem hiding this comment.
Good point! I created it separately for clearer observability (incremental_writer vs writer in logs/metrics), but you're right - will reuse Writer to avoid the duplication.
| # (credential-chain probes, 404 heads, CRT ERROR on recoverable conditions) that would | ||
| # dominate a long run's CI log without signaling a real failure. | ||
| RUST_LOG: "warn,stress_tests=info,awscrt=off" | ||
| S3_BUCKET_NAME: ${{ vars.S3_BENCH_BUCKET_NAME }} |
There was a problem hiding this comment.
This needs to be changed to S3 express bucket? Check how benchmark CI is configured for S3 Express runs as a reference
| RUST_LOG: "warn,stress_tests=info,awscrt=off" | ||
| S3_BUCKET_NAME: ${{ vars.S3_BENCH_BUCKET_NAME }} | ||
| S3_BUCKET_TEST_PREFIX: ${{ vars.S3_BUCKET_BENCH_PREFIX || 'mountpoint-benchmark/' }} | ||
| S3_ENDPOINT_URL: ${{ vars.S3_BENCH_ENDPOINT_URL }} |
| @@ -0,0 +1,40 @@ | |||
| //! `incremental_upload`: 32 workers concurrently writing 64 MiB objects using incremental upload | |||
There was a problem hiding this comment.
Let's just duplicate existing sustained_writes scenario (same parameters/settings - just do incremental upload flag/config). And name it like sustained_writes_incremental_upload (for consistency with benchmark naming)
There was a problem hiding this comment.
Done! Matched sustained_writes exactly - same parameters, just added incremental_upload: true config.
Signed-off-by: Priyankakarumuru1 <prikaru@amazon.com>
ce8e44e to
b412a64
Compare
2 participants
Added a new stress test for incremental uploads. The test has 48 workers that write 100 MiB objects concurrently using incremental upload mode while sharing a 512 MiB memory limit. This mirrors the existing
sustained_writesscenario but with the incremental upload path enabled.Does this change impact existing behavior?
No. This only adds a new test
Does this change need a changelog entry? Does it require a version change?
No. Test-only change
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).