Skip to content

Commit 4c341dc

Browse files
Andy-JostAndy-Jost
committed
Merge remote-tracking branch 'origin/main' into ajost/issue-2807
2 parents f1c6b82 + ad46716 commit 4c341dc

32 files changed

Lines changed: 8337 additions & 5320 deletions

‎.github/workflows/ci-pixi-lockfile-freshness-check.yml‎

Lines changed: 50 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ on:
3737
push:
3838
# `pull_request` already covers PRs, including those from forks: this check
3939
# needs no secrets or GPU runner. Watching copy-pr-bot's `pull-request/N`
40-
# mirror too would run the whole matrix a second time per PR.
40+
# mirror too would run the whole check a second time per PR.
4141
branches:
4242
- "main"
4343
paths: *lockfile_inputs
@@ -53,40 +53,13 @@ env:
5353
permissions: {}
5454

5555
jobs:
56-
# The workspace list is derived from the committed manifests so this matrix
57-
# cannot silently skip a newly added workspace (#2298).
58-
plan:
59-
name: Discover pixi workspaces
60-
if: ${{ github.repository_owner == 'nvidia' }}
61-
runs-on: ubuntu-latest
62-
timeout-minutes: 5
63-
permissions:
64-
contents: read
65-
outputs:
66-
matrix: ${{ steps.discover.outputs.matrix }}
67-
steps:
68-
- name: Checkout ${{ github.event.repository.name }}
69-
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
70-
with:
71-
fetch-depth: 1
72-
persist-credentials: false
73-
74-
- name: Discover workspaces
75-
id: discover
76-
run: |
77-
workspaces="$(python3 ci/tools/list_pixi_workspaces.py)"
78-
echo "matrix=$(jq -c '{include: .}' <<<"${workspaces}")" >> "${GITHUB_OUTPUT}"
79-
8056
lockfile-fresh:
81-
name: "pixi lock --check (${{ matrix.manifest }})"
82-
needs: plan
57+
name: pixi lock --check (all workspaces)
58+
if: ${{ github.repository_owner == 'nvidia' }}
8359
runs-on: ubuntu-latest
84-
timeout-minutes: 20
60+
timeout-minutes: 135
8561
permissions:
8662
contents: read
87-
strategy:
88-
fail-fast: false
89-
matrix: ${{ fromJSON(needs.plan.outputs.matrix) }}
9063
steps:
9164
- name: Checkout ${{ github.event.repository.name }}
9265
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
@@ -98,23 +71,57 @@ jobs:
9871
id: setup-pixi
9972
uses: ./.github/actions/setup-pixi
10073

101-
- name: Check lockfile is current with its manifest
102-
run: |
103-
if ! pixi lock --check --manifest-path "${{ matrix.manifest }}"; then
104-
echo "::error::Lockfile is stale for '${{ matrix.manifest }}'. Regenerate with: pixi lock --manifest-path ${{ matrix.manifest }}. For default-branch dependency drift, maintainers can run ${REFRESH_WORKFLOW_URL}"
105-
exit 1
106-
fi
107-
10874
# `pixi lock --check` exits 0 on a semantically current lock even when it
10975
# rewrites the file into the pinned version's canonical form (lockfile
11076
# format upgrades, platform alias renames). Without this guard that drift
11177
# is invisible: the check prints "Updated lock file" and still passes,
11278
# while every later pixi run keeps rewriting the committed file (#2298).
113-
- name: Check lockfile is byte-for-byte canonical
79+
- name: Check all lockfiles
11480
env:
11581
PIXI_VERSION: ${{ steps.setup-pixi.outputs.pixi-version }}
82+
WORKSPACE_TIMEOUT: 20m
11683
run: |
117-
if ! git diff --exit-code -- "${{ matrix.lockfile }}"; then
118-
echo "::error::pixi ${PIXI_VERSION} rewrote ${{ matrix.lockfile }} during the check, so the committed bytes are not what it generates. Regenerate with pixi ${PIXI_VERSION}: pixi lock --manifest-path ${{ matrix.manifest }}. For default-branch dependency drift, maintainers can run ${REFRESH_WORKFLOW_URL}"
119-
exit 1
120-
fi
84+
workspaces="$(python3 ci/tools/list_pixi_workspaces.py)"
85+
workspace_rows="$(
86+
jq -ce '
87+
if type != "array" or length == 0 then
88+
error("workspace inventory must be a non-empty array")
89+
elif any(.[];
90+
type != "object"
91+
or (.manifest | type != "string")
92+
or (.lockfile | type != "string")
93+
or .manifest == ""
94+
or .lockfile == ""
95+
) then
96+
error("each workspace must have non-empty string manifest and lockfile fields")
97+
else
98+
.[]
99+
end
100+
' <<<"${workspaces}"
101+
)"
102+
failed=0
103+
104+
while IFS= read -r workspace; do
105+
manifest="$(jq -r '.manifest' <<<"${workspace}")"
106+
lockfile="$(jq -r '.lockfile' <<<"${workspace}")"
107+
echo "::group::pixi lock --check (${manifest})"
108+
109+
check_status=0
110+
timeout --kill-after=1m "${WORKSPACE_TIMEOUT}" \
111+
pixi lock --check --manifest-path "${manifest}" \
112+
|| check_status=$?
113+
if ((check_status == 124 || check_status == 137)); then
114+
echo "::error::Timed out after ${WORKSPACE_TIMEOUT} while checking '${manifest}'."
115+
failed=1
116+
elif ((check_status != 0)); then
117+
echo "::error::Lockfile is stale for '${manifest}'. Regenerate with: pixi lock --manifest-path ${manifest}. For default-branch dependency drift, maintainers can run ${REFRESH_WORKFLOW_URL}"
118+
failed=1
119+
elif ! git diff --exit-code -- "${lockfile}"; then
120+
echo "::error::pixi ${PIXI_VERSION} rewrote ${lockfile} during the check, so the committed bytes are not what it generates. Regenerate with pixi ${PIXI_VERSION}: pixi lock --manifest-path ${manifest}. For default-branch dependency drift, maintainers can run ${REFRESH_WORKFLOW_URL}"
121+
failed=1
122+
fi
123+
124+
echo "::endgroup::"
125+
done <<<"${workspace_rows}"
126+
127+
exit "${failed}"

‎.github/workflows/ci-pixi-lockfile-refresh.yml‎

Lines changed: 83 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,8 @@
22
#
33
# SPDX-License-Identifier: Apache-2.0
44

5-
# Scheduled (and on-demand) pixi lockfile refresh. Each workspace gets its own
6-
# PR so a benchmark or docs environment cannot drag unrelated package churn
7-
# into review. See #2298.
5+
# Scheduled (and on-demand) pixi lockfile refresh. All workspaces are updated
6+
# in one job and, when anything changes, one PR. See #2298 and #2849.
87

98
name: "CI: pixi lockfile refresh"
109

@@ -15,14 +14,7 @@ on:
1514
# documents scheduled runs being delayed or dropped.
1615
- cron: "17 3 * * 1"
1716
timezone: "America/New_York"
18-
workflow_dispatch:
19-
inputs:
20-
package:
21-
description: >
22-
Pixi workspace id to refresh, or "all". Valid ids come from
23-
ci/tools/list_pixi_workspaces.py; an unknown id fails the run.
24-
type: string
25-
default: all
17+
workflow_dispatch: {}
2618

2719
defaults:
2820
run:
@@ -31,47 +23,17 @@ defaults:
3123
permissions: {}
3224

3325
jobs:
34-
# Job-level `if` cannot use `matrix` (evaluated before expansion), and the
35-
# workspace list must not drift from the committed manifests, so resolve the
36-
# requested workspaces here and pass them to refresh as a dynamic matrix.
37-
plan:
38-
name: Select workspaces
39-
if: ${{ github.repository_owner == 'nvidia' }}
40-
runs-on: ubuntu-latest
41-
timeout-minutes: 5
42-
permissions:
43-
contents: read
44-
outputs:
45-
matrix: ${{ steps.select.outputs.matrix }}
46-
steps:
47-
- name: Checkout ${{ github.event.repository.name }}
48-
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
49-
with:
50-
fetch-depth: 1
51-
persist-credentials: false
52-
53-
- name: Select workspaces to refresh
54-
id: select
55-
env:
56-
PACKAGE: ${{ inputs.package || 'all' }}
57-
run: |
58-
workspaces="$(python3 ci/tools/list_pixi_workspaces.py --select "${PACKAGE}")"
59-
echo "matrix=$(jq -c '{include: .}' <<<"${workspaces}")" >> "${GITHUB_OUTPUT}"
60-
6126
refresh:
62-
name: "pixi update (${{ matrix.id }})"
63-
needs: plan
27+
name: pixi update (all workspaces)
28+
if: ${{ github.repository_owner == 'nvidia' }}
6429
runs-on: ubuntu-latest
65-
timeout-minutes: 30
30+
timeout-minutes: 195
6631
permissions:
6732
contents: write
6833
pull-requests: write
6934
concurrency:
70-
group: pixi-lock-refresh-${{ matrix.branch_key }}
35+
group: pixi-lock-refresh
7136
cancel-in-progress: false
72-
strategy:
73-
fail-fast: false
74-
matrix: ${{ fromJSON(needs.plan.outputs.matrix) }}
7537
steps:
7638
- name: Checkout ${{ github.event.repository.name }}
7739
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
@@ -83,44 +45,94 @@ jobs:
8345
id: setup-pixi
8446
uses: ./.github/actions/setup-pixi
8547

86-
- name: Update lockfile without installing
87-
run: pixi update --no-install --manifest-path "${{ matrix.manifest }}"
88-
8948
# Pixi 0.73.0's update and lock-read paths can serialize the same solve
9049
# differently (#2804). Canonicalize with the same operation used by the
9150
# freshness workflow before opening the PR.
92-
- name: Canonicalize the updated lockfile
93-
run: pixi lock --check --manifest-path "${{ matrix.manifest }}"
94-
95-
- name: Verify canonicalization is byte-stable
51+
- name: Update and canonicalize all lockfiles
52+
id: update
9653
env:
97-
LOCKFILE: ${{ matrix.lockfile }}
98-
MANIFEST: ${{ matrix.manifest }}
9954
PIXI_VERSION: ${{ steps.setup-pixi.outputs.pixi-version }}
55+
WORKSPACE_TIMEOUT: 30m
10056
run: |
101-
canonical_blob="$(git hash-object "${LOCKFILE}")"
102-
pixi lock --check --manifest-path "${MANIFEST}"
103-
verified_blob="$(git hash-object "${LOCKFILE}")"
104-
if [ "${canonical_blob}" != "${verified_blob}" ]; then
105-
echo "::error::pixi ${PIXI_VERSION} rewrote ${LOCKFILE} on a second consecutive pixi lock --check; refusing to open an unstable refresh PR."
106-
exit 1
107-
fi
57+
workspaces="$(python3 ci/tools/list_pixi_workspaces.py)"
58+
workspace_rows="$(
59+
jq -ce '
60+
if type != "array" or length == 0 then
61+
error("workspace inventory must be a non-empty array")
62+
elif any(.[];
63+
type != "object"
64+
or (.manifest | type != "string")
65+
or (.lockfile | type != "string")
66+
or .manifest == ""
67+
or .lockfile == ""
68+
) then
69+
error("each workspace must have non-empty string manifest and lockfile fields")
70+
else
71+
.[]
72+
end
73+
' <<<"${workspaces}"
74+
)"
75+
76+
while IFS= read -r workspace; do
77+
manifest="$(jq -r '.manifest' <<<"${workspace}")"
78+
lockfile="$(jq -r '.lockfile' <<<"${workspace}")"
79+
echo "::group::pixi update (${manifest})"
80+
81+
workspace_status=0
82+
# The single-quoted script expands its variables in the child shell.
83+
# shellcheck disable=SC2016
84+
timeout --kill-after=1m "${WORKSPACE_TIMEOUT}" \
85+
bash --noprofile --norc -xeuo pipefail -c '
86+
manifest="$1"
87+
lockfile="$2"
88+
pixi_version="$3"
89+
90+
pixi update --no-install --manifest-path "${manifest}"
91+
pixi lock --check --manifest-path "${manifest}"
92+
93+
canonical_blob="$(git hash-object "${lockfile}")"
94+
pixi lock --check --manifest-path "${manifest}"
95+
verified_blob="$(git hash-object "${lockfile}")"
96+
if [ "${canonical_blob}" != "${verified_blob}" ]; then
97+
echo "::error::pixi ${pixi_version} rewrote ${lockfile} on a second consecutive pixi lock --check; refusing to open an unstable refresh PR."
98+
exit 1
99+
fi
100+
' bash "${manifest}" "${lockfile}" "${PIXI_VERSION}" \
101+
|| workspace_status=$?
102+
103+
if ((workspace_status == 124 || workspace_status == 137)); then
104+
echo "::error::Timed out after ${WORKSPACE_TIMEOUT} while refreshing '${manifest}'."
105+
elif ((workspace_status != 0)); then
106+
echo "::error::Failed to refresh '${manifest}' (exit ${workspace_status})."
107+
fi
108+
echo "::endgroup::"
109+
if ((workspace_status != 0)); then
110+
exit "${workspace_status}"
111+
fi
112+
done <<<"${workspace_rows}"
113+
114+
{
115+
echo "lockfiles<<EOF"
116+
jq -r '.[].lockfile' <<<"${workspaces}"
117+
echo "EOF"
118+
} >> "${GITHUB_OUTPUT}"
108119
109120
- name: Compose PR body
110121
id: pr-body
111122
env:
112-
WORKSPACE_ID: ${{ matrix.id }}
113-
MANIFEST: ${{ matrix.manifest }}
114123
PIXI_VERSION: ${{ steps.setup-pixi.outputs.pixi-version }}
115124
run: |
116125
body_path="${RUNNER_TEMP}/pixi-lock-refresh-body.md"
117126
{
118-
echo "Automated lockfile refresh for \`${WORKSPACE_ID}\` using pixi" \
127+
echo "Automated lockfile refresh for all workspaces using pixi" \
119128
"\`${PIXI_VERSION}\` (\`pixi update --no-install\`, followed by" \
120129
"byte-stable \`pixi lock --check\` canonicalization)."
121130
echo
122-
echo "This PR is workspace-scoped so other packages are not forced to review"
123-
echo "unrelated solver churn. See #2298."
131+
echo "Lockfiles included in the refresh:"
132+
python3 ci/tools/list_pixi_workspaces.py \
133+
| jq -r '.[] | "- `\(.lockfile)`"'
134+
echo
135+
echo "See #2298 and #2849."
124136
echo
125137
echo "> [!IMPORTANT]"
126138
echo "> This PR was opened with \`GITHUB_TOKEN\`. A maintainer with write access must:"
@@ -140,16 +152,13 @@ jobs:
140152
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
141153
with:
142154
token: ${{ github.token }}
143-
add-paths: ${{ matrix.lockfile }}
144-
commit-message: |
145-
chore: refresh pixi.lock for ${{ matrix.id }}
146-
147-
Scheduled pixi update --no-install for this workspace only.
155+
add-paths: ${{ steps.update.outputs.lockfiles }}
156+
commit-message: "chore: refresh pixi lockfiles"
148157
signoff: true
149158
sign-commits: true
150-
branch: ci/pixi-lock-refresh/${{ matrix.branch_key }}
159+
branch: ci/pixi-lock-refresh/all
151160
delete-branch: true
152-
title: "chore: refresh pixi.lock (${{ matrix.id }})"
161+
title: "chore: refresh pixi lockfiles"
153162
labels: |
154163
CI/CD
155164
dependencies
@@ -164,10 +173,8 @@ jobs:
164173
- name: Summarize
165174
run: |
166175
{
167-
echo "### pixi lock refresh (${{ matrix.id }})"
176+
echo "### pixi lock refresh (all workspaces)"
168177
echo ""
169-
echo "- Manifest: \`${{ matrix.manifest }}\`"
170-
echo "- Lockfile: \`${{ matrix.lockfile }}\`"
171178
echo "- Operation: \`${{ steps.cpr.outputs.pull-request-operation }}\`"
172179
if [ -n "${{ steps.cpr.outputs.pull-request-url }}" ]; then
173180
echo "- PR: ${{ steps.cpr.outputs.pull-request-url }}"

‎CONTRIBUTING.md‎

Lines changed: 8 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -219,16 +219,14 @@ The freshness check additionally fails when the check itself rewrote a lockfile.
219219
not canonical for the pinned pixi version, quietly normalizing the file instead,
220220
which leaves every later pixi run rewriting the committed lockfile.
221221

222-
A scheduled workflow (`CI: pixi lockfile refresh`)
223-
runs `pixi update --no-install` per workspace and opens a dedicated PR when that
224-
lockfile changes, so broad dependency churn is reviewed as maintenance rather
225-
than landing inside unrelated feature work. The workflow can also be dispatched
226-
manually for one workspace or for all of them. Its dispatch input and every
227-
lockfile CI matrix resolve through `ci/tools/list_pixi_workspaces.py`, which
228-
derives the workspace list from the committed manifests, so a newly added
229-
workspace is picked up without editing any workflow. Human-readable workspace
230-
IDs remain the dispatch and display names; the inventory generates separate,
231-
ref-safe keys for refresh branches and workflow concurrency.
222+
A scheduled workflow (`CI: pixi lockfile refresh`) runs
223+
`pixi update --no-install` for every workspace in one job and opens one PR with
224+
all changed lockfiles, so broad dependency churn is reviewed as maintenance
225+
rather than landing inside unrelated feature work. The workflow can also be
226+
dispatched manually. Both lockfile workflows resolve their workspace lists
227+
through `ci/tools/list_pixi_workspaces.py`, which derives the inventory from the
228+
committed manifests, so a newly added workspace is picked up without editing a
229+
workflow.
232230

233231
Refresh PRs use `GITHUB_TOKEN`. After one opens, a maintainer with write access
234232
must first select **Approve workflows to run** in the merge box, then assign

0 commit comments

Comments
 (0)