Repository navigation
SRE-1422: Cache the OpenAPI Generator jar in CI - #10523
claude[bot] wants to merge 3 commits into
Conversation
The graph client's `openapi-generator-cli` stores its jar in `~/.cache/openapi-generator-cli`. `warm-up-repo` restores that directory from the Actions cache, keyed on the generator version in `openapitools.json`, and on a miss runs the CLI once to download the jar and saves it, so codegen no longer fetches it from Maven Central.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10523 +/- ##
==========================================
- Coverage 69.03% 69.03% -0.01%
==========================================
Files 2082 2082
Lines 214463 214463
Branches 9256 9256
==========================================
- Hits 148057 148055 -2
- Misses 64519 64520 +1
- Partials 1887 1888 +1 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merging this PR will not alter performance1 warning
Comparing Footnotes |
The images whose `turbo build` runs graph client codegen downloaded the jar from Maven Central inside the build. `restore-openapi-generator` now holds the cache lookup that `warm-up-repo` used, and `docker-build-push` runs it too and passes the restored directory as the `openapi-generator-cli` build context. The Dockerfiles bind-mount that context at the generator's storage directory for the build step, and default it to an empty stage, so a build without it downloads the jar as before.
PR SummaryMedium Risk Overview
Five app Dockerfiles that run graph-client codegen add an optional Reviewed by Cursor Bugbot for commit 69f61d7. Bugbot is set up for automated code reviews on this repo. Configure here. |
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1002 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1527 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 108 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 27 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |
`checksums.txt` pins the SHA-256 of each generator version's jar. `verify-jar.sh` checks the jar with `sha256sum -c` after a cache restore, in every job and before a Docker build gets it as a build context, and after the download on a miss, before the jar is saved. A mismatch, or a configured version with no pinned checksum, fails the job.
Requested via Slack thread
🌟 Purpose
@local/hash-graph-clientcodegen runsopenapi-generator-cli, which downloads the generator jar from Maven Central whenever turbo has no cached result for the task. CI does not keep the jar between runs. When Maven Central answers HTTP 429, the download fails, codegen fails withUnable to access jarfile, and so does every job that depends on it. This ejected #10519 from the merge queue (job log).This PR caches the jar in the Actions cache, keyed on the pinned generator version, and hands it to codegen both on the runner and inside the Docker image builds, so CI codegen finds it on disk and never contacts Maven Central. Moving to a generator version published on GitHub releases (SRE-1022) removes the Maven Central dependency for good; this is the short-term fix.
🔍 Changes
openapitools.jsonsets the CLI's owngenerator-cli.storageDirto~/.cache/openapi-generator-cli. The default location isversions/inside the CLI's package innode_modules, which depends on where Yarn hoists the package. Locally, the jar now also survives a freshnode_modules.restore-openapi-generatoraction reads the version andstorageDirfrom that file with mise'syqand restores the directory from the cache underopenapi-generator-cli-<version>(openapi-generator-cli-6.6.0for the pinned 6.6.0). It skips when the repository was pruned without the graph client, and when the config has nostorageDir, as on a base-branch checkout in the bench workflow before this merges.warm-up-reporuns it in every job that installs Yarn dependencies, so no workflow opts in. On a cache miss it runsopenapi-generator-cli versionin the graph client package: the CLI downloads the jar intostorageDirbefore it starts, and the command fails unless the jar then runs, so a broken download is never saved. The first run onmainafter a version change therefore fills the cache whether or not turbo runs codegen, and branches readmain's cache. The download step iscontinue-on-error; if it fails, codegen tries the download itself, as before.docker-build-pushruns the same restore (installingyqthroughjdx/mise-action, as the build job runs noinstall-tools) and, on a hit, passes the directory to everydocker/build-push-actioncall as theopenapi-generator-clibuild context. It only reads the cache;warm-up-repofills it.turbo buildincludes@local/hash-graph-client#codegen(hash-api,hash-frontend,hash-ai-worker-ts,hash-integration-worker,brunch-agent) bind-mount that context read-write at/root/.cache/openapi-generator-clifor the build step. The builder stage runs as root, so~resolves to/root. The context defaults to an emptyscratchstage, so a build without it, such as a localdocker build, downloads the jar into the mount as before; nothing in the mount ends up in the image.hash-graphandatlasdo not run that codegen and are unchanged.restore-openapi-generator/checksums.txtpins the jar's SHA-256 per version (6.6.0:9718ff78…e4609, from the jar whose SHA-1 and MD5 match the files Maven Central publishes for it and whose PGP signature by the OpenAPI Tools key verifies).verify-jar.shchecks it withsha256sum -cafter every cache hit, so before a Docker build gets the context, and after the download on a miss, before the save. A mismatch, or a version with no pinned checksum, fails the job with an::error::naming the jar and both hashes.Review requested from Tim, who assigned SRE-1422. Please focus on the cache fill on a miss in
warm-up-repo(download,continue-on-error, save only on success) and on theopenapi-generator-clinamed build context wiring indocker-build-pushand the five Dockerfiles.🤖 Generated with Claude Code
https://claude.ai/code/session_01WERujAimwhvaFgWp15SyCJ