Add Azurite test support for AzureBlobStore - #2684
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
|
|
Thank you for the PR. We will check it out soon! |
Adds an embedded Azurite (Azure Storage emulator) test runner, mirroring the existing mongo_runner pattern, and closes TraceMachina#2511. Azurite has no standalone binary distribution, so the runner invokes a locally npm-installed azurite-blob directly rather than downloading one. The store is pointed at it via ExperimentalAzureSpec.sas_url rather than endpoint, since endpoint alone routes through WorkloadIdentityCredential, which Azurite cannot satisfy. A SAS signer generates both a container scoped Service SAS for test operations and an Account SAS for one time container bootstrapping, since a container SAS cannot authorize creating the container it names. Also fixes a process crash discovered while building this: Azurite logs every request to stdout, and reading that pipe only long enough to capture the startup port left it undrained afterward, causing the next write to block and crash the whole process. stdout is now drained for the process's full lifetime instead. CI: adds actions/setup-node and npm ci to native-cargo.yaml so azurite-blob is available on both OS legs before cargo test runs.
d35c7e3 to
419234e
Compare
palfrey
left a comment
palfrey
left a comment
There was a problem hiding this comment.
Thanks for doing this. There's a few issues though.
| - '.github/styles/**' | ||
| - 'web/**' | ||
| - ".github/styles/**" | ||
| - "web/**" |
There was a problem hiding this comment.
Can you please reset the format changes here
|
|
||
| - name: Install Azurite for Azure store tests | ||
| run: npm ci | ||
| working-directory: nativelink-store/tests/azurite_runner |
There was a problem hiding this comment.
Please use bun for package management, as per our existing other usage in https://github.com/TraceMachina/nativelink/blob/main/web/bun.lock
There was a problem hiding this comment.
Also, please make these steps a new reusable action, as they'll need to get added to various other workflows e.g. .github/workflows/nix.yaml and .github/workflows/coverage.yaml
| tokio = { version = "1.52.2", features = [ | ||
| "process", | ||
| "sync", | ||
| ], default-features = false } |
There was a problem hiding this comment.
Please move these further down. Our standard formatting for dependencies is the nativelink internal ones at the top, then everything else in alphabetical order after that.
| "dependencies": { | ||
| "azurite": "3.36.0" | ||
| } | ||
| } |
There was a problem hiding this comment.
This needs reformatting with the pre-commit rules. If you run the Nix setup you'll have pre-commit setup correctly as part of that.
There was a problem hiding this comment.
This needs adding to the test suite in nativelink-store/BUILD.bazel
4 participants
Summary
Adds Azurite (Azure Storage emulator) support to the test suite for
AzureBlobStore, Follows the same embedded-runner patternas
mongo_runner, adapted for Azurite's constraints.Description
Adds an embedded Azurite (Azure Storage emulator) test runner, mirroring the existing mongo_runner pattern.
Azurite has no standalone binary distribution, so the runner invokes a locally npm-installed azurite-blob directly rather than downloading one. The store is pointed at it via ExperimentalAzureSpec.sas_url rather than endpoint, since endpoint alone routes through WorkloadIdentityCredential, which Azurite cannot satisfy. A SAS signer generates both a container scoped Service SAS for test operations and an Account SAS for one time container bootstrapping, since a container SAS cannot authorize creating the container it names.
Also fixes a process crash discovered while building this: Azurite logs every request to stdout, and reading that pipe only long enough to capture the startup port left it undrained afterward, causing the next write to block and crash the whole process. stdout is now drained for the process's full lifetime instead.
Fixes #2511
Type of change
How Has This Been Tested?
Added
upload_and_get_data,upload_empty_data, andzero_len_items_exist_check, run against a realAzureBlobStoretalking to a locally-spawned Azurite instance — no mocking. Two
#[ignore]d tests document the manual verification steps used tovalidate the SAS signer and container-bootstrap logic independently
before wiring them into the automatic flow.
Note on formatting/testing: this environment doesn't have Nix
available (macOS 12/Monterey isn't supported by the current Nix
installer), so
bazel test //...and the Bazel-driven nightlyrustfmttarget couldn't be run directly. Rancargo +nightly fmtlocally as the closest available equivalent, and verified everything
via
cargo build --all --profile=smol/cargo test --all --profile=smol, matching whatnative-cargo.yaml's CI job runs.Local setup:
CI: added
actions/setup-node(pinned to v7.0.0) +npm citonative-cargo.yaml, on both OS legs, cached onnativelink-store/tests/azurite_runner/package-lock.json.Checklist
bazel test //...passes locallygit amendsee some docsThis change is