Skip to content

ci: run all workflows on CodeBuild runner - #747

Open
wangyb-A wants to merge 9 commits into
mainfrom
ci/codebuild-runner
Open

wangyb-A wants to merge 9 commits into
mainfrom
ci/codebuild-runner

Conversation

@wangyb-A

@wangyb-A wangyb-A commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Moves this repo's own CI workflows onto the CodeBuild-hosted GitHub Actions runner in the Python testing account (project github-actions-runner). The label is codebuild-github-actions-runner-${{ github.run_id }}-${{ github.run_attempt }}, so each job gets its own ephemeral runner. The project currently runs on-demand EC2 compute (aws/codebuild/standard:7.0, BUILD_GENERAL1_MEDIUM, 60-minute timeout); the workflows here were validated on both the earlier Lambda compute and the current EC2 configuration.

Scope

This PR only touches workflows that are specific to this repository. Workflows that are thin callers of shared reusable workflows are not changed here, because their runs-on lives in the dependency repository and moving them requires a change there first:

Workflow Reusable workflow owner
opentelemetry-conformance-tests.yml aws/aws-durable-execution-conformance-tests (opentelemetry-orchestrator.yml)
ai-pr-review.yml, notify.yml, issue-triage.yml, stale-issue-closer.yml aws/aws-durable-execution-ci

These will be migrated in a follow-up PR once the shared workflows accept a runner label.

Migrated

Workflow Jobs Notes
ci.yml lint-commits, build (3.11-3.14) setup-uv v10.2.0 + uv pip install hatch, pip isolation env
conformance-tests.yml discover_suites, conformance (12 suites) SAM deploy runs natively on the runner (no docker); Python 3.14 via setup-uv; runner installed with uv pip install
cloud-tests.yml example-tests (3.11-3.14) SAM build/deploy + integration tests; setup-uv for the matrix interpreter, uv pip install hatch==1.16.5
test-parser.yml test-scripts pytest over .github/scripts; setup-uv (3.13)

All migrated jobs get PIP_CONFIG_FILE=/dev/null and PYTHONPATH="" so a runner image's shared /etc/pip.conf target cannot leak into the build. The Python interpreter comes from astral-sh/setup-uv SHA-pinned to c18668ad3cf93ea998bef934396af7bb5c839dc7 (v10.2.0), which keeps the workflow independent of whichever Python versions the runner image ships.

Fork handling

conformance-tests.yml (discover_suites, conformance) and cloud-tests.yml (example-tests) are already gated by an if: that excludes fork PRs and Dependabot (github.event.pull_request.head.repo.full_name == github.repository), so they use the plain CodeBuild label.

ci.yml (lint-commits, build) and test-parser.yml (test-scripts) have no such gate and are reachable from pull_request, so they use a fork-aware runs-on expression that keeps fork PRs (and, for build, Dependabot PRs) on ubuntu-latest:

runs-on: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository && 'ubuntu-latest' || format('codebuild-github-actions-runner-{0}-{1}', github.run_id, github.run_attempt) }}

Note that for pull_request events the workflow definition comes from the PR head, so these expressions and if: gates prevent accidental routing but are not a trust boundary on their own: a PR can edit them. The boundary is the repository's "Approval for running fork pull request workflows" setting (a maintainer approves each fork run after seeing the diff) together with the CodeBuild project's webhook configuration.

Left on GitHub-hosted runners on purpose

Workflow Reason
scorecard.yml OSSF Scorecard requires GitHub-hosted runners
pypi-publish.yml Release/publish path (PyPI trusted publishing)
lambda-layer-publish.yml Release/publish path (Lambda layer publish across ~35 regions)
ecr-release.yml Release path; uses docker build / docker manifest + QEMU, to be revisited separately

Verification

  • ci.yml (build 3.11-3.14 + lint-commits): green on the runner.
  • conformance-tests.yml: green on the runner; discover_suites plus all 12 suite jobs pass (~1-3.5 min each).
  • cloud-tests.yml: SAM builds natively, deploys, and the example tests run on every Python version (3.11-3.14).
  • test-parser.yml: its path filters did not include the workflow file itself, so it had not run on this branch; the filters now include it and the run on the latest head is the first execution on the runner.

The first three are green on both Lambda and EC2 compute.

Runner-caused fixes (test configuration and scripts only, no SDK code)

  • cloud-tests.yml resolved the latest ADOT layer via gh api in the SAM deploy step. The Lambda-compute runner image had no gh CLI, so the step failed immediately. Replaced the gh api call with curl to the GitHub REST API plus python3 to extract the release body; the awk region-parsing is unchanged.
  • test/conftest.py in the examples package removes _X_AMZN_TRACE_ID from the test process. On Lambda compute the runner itself is a Lambda and sets both AWS_LAMBDA_FUNCTION_NAME and _X_AMZN_TRACE_ID; botocore's recursion-detection handler then attaches the runner's X-Amzn-Trace-Id to every Lambda invoke, the example functions joined the runner's unsampled trace, and the X-Ray span assertions (test_otel_logger_example_spans_in_xray, test_plugin_spans_in_xray_across_invocations) could not find their marker spans. Dropping the variable restores independent traces for the invoked functions.
  • tests/filesystem_serdes_test.py::test_serialization_error_handling relied on /nonexistent/readonly/path being unwritable. EC2 compute runs jobs as root, which simply created the path, so the expected SerDesError was never raised. The test now uses a regular file as the parent directory, which fails for every user.

Move lint-commits and the build matrix from ubuntu-latest to the codebuild-github-actions-runner label (CodeBuild Lambda compute in the Python testing account). The Amazon Linux runner image has no matching actions/setup-python builds, so uv provides each matrix interpreter and an activated venv; Hatch is installed with uv pip.
Move cloud-tests, conformance-tests and test-parser onto the CodeBuild-hosted runner, mirroring ci.yml: setup-uv (v10.2.0) for the interpreter, uv pip install, and PIP_CONFIG_FILE/PYTHONPATH isolation. Fork-gated jobs use the plain label; test-parser uses the fork-aware runs-on expression.
@wangyb-A wangyb-A changed the title ci: run ci.yml on the CodeBuild-hosted runner ci: run all workflows on CodeBuild runner Sep 30, 2026
Alex Wang added 5 commits September 30, 2026 03:38
The CodeBuild runner image has no gh CLI, so the SAM deploy step's gh api call to resolve the latest ADOT Python layer failed with 'gh: command not found'. Replace it with curl to the GitHub REST API plus python3 to extract the release body; the awk region parse is unchanged.
The two cloud-test X-Ray span-visibility assertions failed intermittently on the runner (traces present but the durable-execution marker span not yet indexed within the ~50s window). The same tests pass on the same runner when ingestion is fast, so widen the retry budget (3x10s -> 6x15s) to absorb slower eventual-consistency instead of failing the run.
Widening the X-Ray retry budget did not resolve the two span-visibility failures (spans still not indexed after ~110s), confirming the failure is environmental X-Ray/ADOT ingestion rather than runner-caused. Revert to keep the migration change minimal.
The CodeBuild Lambda-compute runner sets AWS_LAMBDA_FUNCTION_NAME and _X_AMZN_TRACE_ID. botocore's recursion-detection handler (add_recursion_detection_header) then attaches the runner's X-Amzn-Trace-Id to every Lambda invoke, so the example functions join the runner's unsampled trace and the two X-Ray span tests cannot find their marker spans. The same tests pass on ubuntu-latest, where neither variable is set. Removing the variable in the examples conftest restores independent traces.
The CodeBuild EC2 runner executes jobs as root, so FileSystemSerDes could create /nonexistent/readonly/path and the serialization error test did not raise. Use a regular file as the parent directory, which fails for every user.
@wangyb-A
wangyb-A marked this pull request as ready for review October 2, 2026 00:39
@wangyb-A
wangyb-A deployed to ai-pr-review-runtime October 2, 2026 00:39 — with GitHub Actions Active
@wangyb-A
wangyb-A deployed to ai-pr-review-runtime October 2, 2026 00:57 — with GitHub Actions Active
Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/test-parser.yml
@github-actions

This comment has been minimized.

- ci.yml: lint-commits and build now route fork PRs (and Dependabot for build) to ubuntu-latest, matching test-parser and the if: gates on cloud-tests/conformance; the plain CodeBuild label let fork code run inside the testing account.
- test-parser.yml: include the workflow itself in both path filters so a change to it (such as this runner migration) triggers a run; it had not executed on this branch.
@wangyb-A
wangyb-A had a problem deploying to ai-pr-review-runtime October 2, 2026 18:27 — with GitHub Actions Error
The first test-parser run on the runner failed in test_build_layer_excludes_adot_and_runtime_dependencies: build_lambda_layer.py runs `python -m pip install`, and the venv uv creates has no pip (setup-python's venv did). Install pip with the test dependencies.
@wangyb-A
wangyb-A deployed to ai-pr-review-runtime October 2, 2026 18:35 — with GitHub Actions Active
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Codex AI review

No actionable findings. Residual risk is limited to external CodeBuild project and runner-image configuration not represented in the diff.

Reviewed commit 7b87525761d3488f2f1550fa65362fbcfe898481. Workflow run

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — 7b875257 Deployed Oct 2, 2026 by wangyb-A via ai-pr-review / Codex review / Generate Codex review #1151
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant