|
| 1 | +--- |
| 2 | +name: spec-upgrade |
| 3 | +description: Sync the OpenAPI spec from the Unstract backend into this repo and regenerate the transport layer. Use whenever the backend adds or changes a Document Studio API deployment endpoint, when the `sdk-drift` CI job goes red, when `specs/docstudio-oss.json` is behind the backend, or when someone asks to "regenerate the SDK", "refresh the spec", "pick up the new endpoint", or "release unstract-client". Reach for this even when the request only mentions the generated code or a new endpoint — the upgrade is an ordered pipeline, and skipping a step leaves the drift gate red or the spec silently stale. |
| 4 | +--- |
| 5 | + |
| 6 | +# Upgrading the spec and regenerating the client |
| 7 | + |
| 8 | +Nothing tells this repo when the backend's spec moves. Someone notices, and then |
| 9 | +runs this pipeline. The steps are ordered because each one's output is the next |
| 10 | +one's input, and because two of them are the only thing standing between a stale |
| 11 | +copy and a client that looks generated but isn't. |
| 12 | + |
| 13 | +## The pieces |
| 14 | + |
| 15 | +| Thing | Where | |
| 16 | +|---|---| |
| 17 | +| Vendored spec | `specs/docstudio-oss.json` | |
| 18 | +| Generator wrapper | `tools/gen_sdk.sh` (pins the generator, records `SPEC_SOURCE`) | |
| 19 | +| Generated transport | `src/unstract/api_deployments/_sdk_docstudio/` — never hand-edited | |
| 20 | +| Hand-written facade | `src/unstract/api_deployments/client.py` | |
| 21 | +| Drift gate | `sdk-drift` job in `.github/workflows/test.yml` | |
| 22 | +| Release | `.github/workflows/main.yml`, `workflow_dispatch` | |
| 23 | + |
| 24 | +Upstream, the spec is produced by the backend that serves these endpoints |
| 25 | +(`Zipstack/unstract`, `manage.py generate_docstudio_spec`, committed at |
| 26 | +`specs/docstudio-oss.json`). It is a copy here, not a fork. |
| 27 | + |
| 28 | +## The sequence |
| 29 | + |
| 30 | +1. **Copy the spec byte-for-byte** from the backend commit you are upgrading to. |
| 31 | + Do not reformat it and do not edit it here — the drift gate regenerates from |
| 32 | + whatever is committed, so an edit here becomes a client that no backend |
| 33 | + serves. |
| 34 | + |
| 35 | +2. **Move `SPEC_SOURCE` in `tools/gen_sdk.sh` in the same commit.** It records |
| 36 | + the backend repo, path, revision and sha256. Without that update, nothing |
| 37 | + distinguishes a current copy from one the backend has moved past — the script |
| 38 | + and the drift gate both report clean either way, which is exactly the failure |
| 39 | + this pipeline exists to prevent. |
| 40 | + |
| 41 | +3. **Regenerate:** |
| 42 | + |
| 43 | + ```bash |
| 44 | + ./tools/gen_sdk.sh |
| 45 | + ``` |
| 46 | + |
| 47 | + It rebuilds the tree from scratch with a pinned generator. If it exits |
| 48 | + non-zero on a generator warning, believe it: the generator downgrades a |
| 49 | + schema it cannot parse to a warning, drops the endpoint or model, writes the |
| 50 | + rest and exits 0 — so the warning is the only signal that an operation is |
| 51 | + missing. |
| 52 | + |
| 53 | +4. **Review the generated surface, with new files included:** |
| 54 | + |
| 55 | + ```bash |
| 56 | + git add -N -- src/unstract/api_deployments/_sdk_docstudio |
| 57 | + git diff --stat -- src/unstract/api_deployments/_sdk_docstudio |
| 58 | + git diff -- src/unstract/api_deployments/_sdk_docstudio |
| 59 | + ``` |
| 60 | + |
| 61 | + The `--stat` is the shape of the change; the full diff is the change. Read |
| 62 | + both — the operations and fields that moved only show up in the second. |
| 63 | + |
| 64 | + `git add -N` first because a plain diff cannot see a file the generator has |
| 65 | + newly created — which is precisely what a spec that grew an endpoint |
| 66 | + produces. Read the diff as the API change it represents: new modules are new |
| 67 | + operations, removed fields are removals. |
| 68 | + |
| 69 | +5. **Make the new surface reachable, in `client.py`.** Regeneration only moves |
| 70 | + the transport. If the spec added an endpoint callers should be able to use, |
| 71 | + the facade is where it becomes public API. Fixes belong here or upstream in |
| 72 | + the spec, never in the generated tree — regeneration overwrites that wholesale. |
| 73 | + |
| 74 | +6. **Run the tests:** `uv run pytest tests/`. `tests/test_compat.py` compares |
| 75 | + this client against the last released one, vendored under `tests/baseline/`. |
| 76 | + Refresh that baseline only when you mean to move the parity reference point, |
| 77 | + with `tools/refresh_baseline.sh <released-version>`; a spec upgrade on its own |
| 78 | + is not a reason to move it. |
| 79 | + |
| 80 | +## CI |
| 81 | + |
| 82 | +`sdk-drift` regenerates from the committed spec and fails on any diff, so a |
| 83 | +hand-edit of the generated tree and a spec change nobody regenerated over both |
| 84 | +fail the same way. If it is red, run step 3 and commit the result. |
| 85 | + |
| 86 | +## Versioning and release |
| 87 | + |
| 88 | +Choose the bump by what changed for callers: **major** when the spec removed or |
| 89 | +renamed something callers depend on, **minor** for new endpoints or new |
| 90 | +behaviour, **patch** for fixes that keep the surface identical. A generated diff |
| 91 | +with removals in it is the signal for major — spec upgrades produce those. |
| 92 | + |
| 93 | +Do not touch `__version__` in `src/unstract/api_deployments/__init__.py` in your |
| 94 | +PR. The in-repo value is the *last released* version; `main.yml` reads it, |
| 95 | +applies the bump you pick at dispatch time, and commits the result itself. Bumping |
| 96 | +it in the PR makes the release skip a version. |
| 97 | + |
| 98 | +Release by dispatching **Release Tag and Publish Package** on `main` and choosing |
| 99 | +the bump. It publishes to PyPI *before* it tags — a failure after publish is |
| 100 | +retried by hand against a live artifact, not by re-publishing. |
| 101 | + |
| 102 | +## Downstream |
| 103 | + |
| 104 | +`unstract-cli` pins this client exactly and vendors a copy of the same spec. |
| 105 | +After a release lands on PyPI, that pin needs bumping there; see the |
| 106 | +`bump-client-pins` skill in that repo. |
0 commit comments