Repository navigation
Implement Automated Line and MC/DC Coverage CI Workflows - #219165
tapiwagonga wants to merge 26 commits into
Conversation
…es and unit tests
|
Hello @tapiwagonga 👋 Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description. Frequently asked questionsHow do I add reviewers? This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically. You can also add reviewers manually using the Reviewers section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using What if there are no comments? If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers. Are any special GitHub settings required to contribute to LLVM? We only require contributors to have a public email address associated with their GitHub commits, see this section of LLVM Developer Policy for details. If you have questions, feel free to leave a comment on this PR, or ask on LLVM Discord or LLVM Discourse. Thank you, |
|
@llvm/pr-subscribers-github-workflow @llvm/pr-subscribers-libc Author: Tapiwa Gonga (tapiwagonga) ChangesI previously wrote an RFC regarding Enabling Code Coverage in LLVM-libc to address the circular dependencies and link-time failures caused by standard compiler-rt profiling in hermetic (-nostdlib) testing environments. After incorporating community feedback, the core freestanding instrumentation was merged in Code Coverage via Linux Syscalls. Following that integration, I evaluated how best to surface this coverage data automatically within the monorepo. This resulted in a second RFC proposing an Automated Code Coverage Infrastructure utilising GitHub Actions to track coverage on commit. This pull request implements that infrastructure. Developer instructions on how to run coverage locally have been published in How To Run Code Coverage. Implementation Details GitHub Actions Workflows ( Pre-commit bots ( Post-commit bots ( In-Tree Reporting Analysers (
Unit Tests
Patch is 95.24 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/219165.diff 8 Files Affected:
diff --git a/.github/workflows/libc-full-coverage.yml b/.github/workflows/libc-full-coverage.yml
new file mode 100644
index 0000000000000..ebaa9f2a39fb6
--- /dev/null
+++ b/.github/workflows/libc-full-coverage.yml
@@ -0,0 +1,143 @@
+name: Libc Full Codebase Coverage
+
+permissions:
+ contents: write # Required to publish live HTML dashboard to gh-pages branch
+
+env:
+ FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: 'true'
+
+on:
+ # Daily overnight run at 02:00 UTC
+ schedule:
+ - cron: '0 2 * * *'
+
+ # Allow manual on-demand execution from GitHub Actions UI
+ workflow_dispatch:
+
+ # Trigger on pushes to main / libc-coverage-ci-bots / post-commit-bot
+ push:
+ branches:
+ - main
+ - libc-coverage-ci-bots
+ - post-commit-bot
+ paths:
+ - 'libc/**'
+ - '.github/workflows/libc-full-coverage.yml'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ full-coverage:
+ timeout-minutes: 60
+ name: libc-full-coverage
+ runs-on: ubuntu-24.04
+ container:
+ image: ghcr.io/llvm/libc-ubuntu-24.04:latest@sha256:8fee4c9ce5a1fd095686593cd36e032c48a31b6ae575378c82accd1d86a08d59
+ options: --privileged
+ defaults:
+ run:
+ shell: bash
+ steps:
+ - name: Checkout Code
+ uses: actions/checkout@v4
+ with:
+ fetch-depth: 1
+ persist-credentials: false
+
+ - name: Setup Compiler Cache (sccache)
+ uses: hendrikmuhs/ccache-action@v1.2.23
+ with:
+ max-size: 1G
+ key: libc_coverage_unified_v2_x86_64
+ variant: sccache
+
+ - name: Configure CMake
+ run: |
+ echo "=== [STAGE 1/4] Starting CMake Configuration at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ START_TIME=$(date +%s)
+
+ export CMAKE_FLAGS="
+ -G Ninja
+ -S runtimes
+ -B build-cov
+ -DCMAKE_C_COMPILER=clang-23
+ -DCMAKE_CXX_COMPILER=clang++-23
+ -DCMAKE_BUILD_TYPE=Debug
+ -DCMAKE_C_COMPILER_LAUNCHER=sccache
+ -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
+ -DLLVM_USE_LINKER=lld-23
+ -DLLVM_ENABLE_RUNTIMES=libc
+ -DLLVM_LIBC_FULL_BUILD=ON
+ -DLIBC_ENABLE_COVERAGE=ON
+ -DLIBC_TEST_UNIT_TEST_ONLY=ON
+ -DLIBC_TEST_SKIP_DEATH_TESTS=ON
+ -DLIBC_TEST_SKIP_SHARED_TESTS=ON
+ "
+ cmake $CMAKE_FLAGS
+
+ END_TIME=$(date +%s)
+ echo "=== [STAGE 1/4] CMake Configuration Completed in $((END_TIME - START_TIME))s ==="
+
+ - name: Run Full Codebase Unit Tests
+ run: |
+ echo "=== [STAGE 2/4] Running Full Unit Test Suite at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ START_TIME=$(date +%s)
+
+ export LLVM_PROFILE_FILE="libc_cov_%p.profraw"
+ ninja -k 0 -C build-cov libc-unit-tests || true
+
+ END_TIME=$(date +%s)
+ echo "=== [STAGE 2/4] libc-unit-tests Completed in $((END_TIME - START_TIME))s ==="
+
+ - name: Merge Profiles
+ run: |
+ echo "=== [STAGE 3/4] Merging Raw Profiles at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ find . build-cov -name "libc_cov_*.profraw" > profraw_list.txt
+ NUM_PROFS=$(wc -l < profraw_list.txt || echo 0)
+ echo "[LOG] Discovered $NUM_PROFS raw profile data files."
+ llvm-profdata-23 merge -sparse --input-files=profraw_list.txt -o libc_full.profdata
+ echo "[LOG] Merged full codebase profile into libc_full.profdata."
+
+ - name: Generate Reports and Summary
+ env:
+ COMMIT_SHA: ${{ github.sha }}
+ BRANCH_REF: ${{ github.ref_name }}
+ run: |
+ echo "=== [STAGE 4/4] Generating Full Codebase Coverage Reports at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ EXECUTABLES=($(find build-cov -type f -executable -name "*__build__"))
+ OBJECTS=("${EXECUTABLES[@]:1}")
+ OBJECTS=("${OBJECTS[@]/#/-object=}")
+
+ echo "[LOG] Exporting coverage data across ${#EXECUTABLES[@]} test binaries."
+
+ # 1. Generate JSON export for Full Coverage Analyzer
+ llvm-cov-23 export -format=text -instr-profile=libc_full.profdata "${EXECUTABLES[0]}" "${OBJECTS[@]}" > coverage.json
+
+ # 2. Generate HTML Coverage Report with directory hierarchy and branch tracking
+ llvm-cov-23 show -format=html \
+ -output-dir=coverage_html \
+ -instr-profile=libc_full.profdata \
+ "${EXECUTABLES[0]}" "${OBJECTS[@]}" \
+ --show-directory-coverage \
+ --show-branches=count \
+ --compilation-dir=. \
+ --path-equivalence="$GITHUB_WORKSPACE,." \
+ -ignore-filename-regex=".*(test|utils).*"
+ touch coverage_html/.nojekyll
+
+ # 3. Run Codebase Coverage Analyzer
+ PYTHONPATH="libc/utils/coverage" python3 libc/utils/coverage/codebase_coverage.py coverage.json "$COMMIT_SHA" "$BRANCH_REF" >> $GITHUB_STEP_SUMMARY
+ echo "[LOG] Post-commit summary report written to GITHUB_STEP_SUMMARY."
+
+ - name: Deploy Coverage Dashboard to GitHub Pages
+ if: github.repository != 'llvm/llvm-project'
+ continue-on-error: true
+ uses: peaceiris/actions-gh-pages@v4
+ with:
+ github_token: ${{ secrets.GITHUB_TOKEN }}
+ publish_dir: ./coverage_html
+ keep_files: true
+ enable_jekyll: false
+ force_orphan: false
diff --git a/.github/workflows/libc-full-mcdc.yml b/.github/workflows/libc-full-mcdc.yml
new file mode 100644
index 0000000000000..fdebc1c66f399
--- /dev/null
+++ b/.github/workflows/libc-full-mcdc.yml
@@ -0,0 +1,149 @@
+name: Libc Full Codebase MC/DC Coverage
+
+permissions:
+ contents: write
+
+env:
+ FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: 'true'
+
+on:
+ # Daily overnight run at 02:30 UTC (staggered by 30 mins from baseline sweep)
+ schedule:
+ - cron: '30 2 * * *'
+
+ # Allow manual on-demand execution from GitHub Actions UI ("Run workflow" button)
+ workflow_dispatch:
+
+ # Trigger on pushes to main / libc-mcdc-coverage / post-commit-bot
+ push:
+ branches:
+ - main
+ - libc-mcdc-coverage
+ - post-commit-bot
+ paths:
+ - 'libc/**'
+ - '.github/workflows/libc-full-mcdc.yml'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ full-mcdc-coverage:
+ timeout-minutes: 60
+ name: libc-full-mcdc-coverage
+ runs-on: ubuntu-24.04
+ container:
+ image: ghcr.io/llvm/libc-ubuntu-24.04:latest@sha256:8fee4c9ce5a1fd095686593cd36e032c48a31b6ae575378c82accd1d86a08d59
+ options: --privileged
+ defaults:
+ run:
+ shell: bash
+ steps:
+ - name: Checkout Code
+ uses: actions/checkout@v4
+ with:
+ fetch-depth: 1
+ persist-credentials: false
+
+ - name: Setup Compiler Cache (sccache)
+ uses: hendrikmuhs/ccache-action@v1.2.23
+ with:
+ max-size: 1G
+ key: libc_mcdc_coverage_unified_x86_64
+ variant: sccache
+
+ - name: Configure CMake with MC/DC
+ run: |
+ echo "=== [STAGE 1/4] Starting CMake Configuration (MC/DC) at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ START_TIME=$(date +%s)
+
+ export CMAKE_FLAGS="
+ -G Ninja
+ -S runtimes
+ -B build-cov
+ -DCMAKE_C_COMPILER=clang-23
+ -DCMAKE_CXX_COMPILER=clang++-23
+ -DCMAKE_BUILD_TYPE=Debug
+ -DCMAKE_C_COMPILER_LAUNCHER=sccache
+ -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
+ -DLLVM_USE_LINKER=lld-23
+ -DLLVM_ENABLE_RUNTIMES=libc
+ -DLLVM_LIBC_FULL_BUILD=ON
+ -DLIBC_ENABLE_COVERAGE=ON
+ -DLIBC_ENABLE_MCDC=ON
+ -DLIBC_TEST_UNIT_TEST_ONLY=ON
+ -DLIBC_TEST_SKIP_DEATH_TESTS=ON
+ -DLIBC_TEST_SKIP_SHARED_TESTS=ON
+ "
+ cmake $CMAKE_FLAGS
+
+ END_TIME=$(date +%s)
+ echo "=== [STAGE 1/4] CMake Configuration Completed in $((END_TIME - START_TIME))s ==="
+
+ - name: Run Full Codebase Unit Tests
+ run: |
+ echo "=== [STAGE 2/4] Running Full Unit Test Suite at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ START_TIME=$(date +%s)
+
+ export LLVM_PROFILE_FILE="libc_cov_%p.profraw"
+ ninja -k 0 -C build-cov libc-unit-tests || true
+
+ END_TIME=$(date +%s)
+ echo "=== [STAGE 2/4] libc-unit-tests Completed in $((END_TIME - START_TIME))s ==="
+
+ - name: Merge Profiles
+ run: |
+ echo "=== [STAGE 3/4] Merging Raw Profiles at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ find build-cov -name "libc_cov_*.profraw" > profraw_list.txt
+ NUM_PROFS=$(wc -l < profraw_list.txt || echo 0)
+ echo "[LOG] Discovered $NUM_PROFS raw profile data files."
+ llvm-profdata-23 merge -sparse --input-files=profraw_list.txt -o libc_full.profdata
+ echo "[LOG] Merged full codebase profile into libc_full.profdata."
+
+ - name: Generate Reports and Summary
+ env:
+ COMMIT_SHA: ${{ github.sha }}
+ BRANCH_REF: ${{ github.ref_name }}
+ run: |
+ echo "=== [STAGE 4/4] Generating Full Codebase MC/DC Coverage Reports at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ EXECUTABLES=($(find build-cov -type f -executable -name "*__build__"))
+ OBJECTS=("${EXECUTABLES[@]:1}")
+ OBJECTS=("${OBJECTS[@]/#/-object=}")
+
+ echo "[LOG] Exporting coverage data across ${#EXECUTABLES[@]} test binaries."
+
+ # 1. Generate JSON export for Full Coverage Analyzer (contains mcdc_records)
+ llvm-cov-23 export -format=text -instr-profile=libc_full.profdata "${EXECUTABLES[0]}" "${OBJECTS[@]}" > coverage.json
+
+ # 2. Generate HTML Coverage Report with directory hierarchy, branch tracking, and MC/DC truth tables
+ llvm-cov-23 show -format=html \
+ -output-dir=coverage_mcdc_html \
+ -instr-profile=libc_full.profdata \
+ "${EXECUTABLES[0]}" "${OBJECTS[@]}" \
+ --show-directory-coverage \
+ --show-branches=count \
+ --show-mcdc \
+ --show-mcdc-summary \
+ --compilation-dir=. \
+ --path-equivalence="$GITHUB_WORKSPACE,." \
+ -ignore-filename-regex=".*(test|utils).*"
+ touch coverage_mcdc_html/.nojekyll
+
+ # 3. Run Codebase Coverage Analyzer
+ PYTHONPATH="libc/utils/coverage" python3 libc/utils/coverage/codebase_coverage.py coverage.json "$COMMIT_SHA" "$BRANCH_REF" >> $GITHUB_STEP_SUMMARY
+ echo "[LOG] Post-commit summary report written to GITHUB_STEP_SUMMARY."
+
+ - name: Deploy MC/DC Coverage Report to GitHub Pages
+ if: github.repository != 'llvm/llvm-project'
+ continue-on-error: true
+ uses: peaceiris/actions-gh-pages@v4
+ with:
+ github_token: ${{ secrets.GITHUB_TOKEN }}
+ publish_dir: ./coverage_mcdc_html
+ destination_dir: mcdc
+ keep_files: true
+ enable_jekyll: false
+ force_orphan: false
+
+
diff --git a/.github/workflows/libc-patch-coverage.yml b/.github/workflows/libc-patch-coverage.yml
new file mode 100644
index 0000000000000..5cd2b723862e1
--- /dev/null
+++ b/.github/workflows/libc-patch-coverage.yml
@@ -0,0 +1,362 @@
+name: Libc Patch Code Coverage
+
+permissions:
+ contents: read
+ pull-requests: write # Required to post/update patch coverage comments
+
+env:
+ FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: 'true'
+
+on:
+ workflow_dispatch:
+ push:
+ branches:
+ - pre-commit-bot
+ - libc-coverage-ci-bots
+ - main
+ paths:
+ - 'libc/src/**'
+ - 'libc/include/**'
+ - 'libc/test/**'
+ - 'libc/CMakeLists.txt'
+ - '.github/workflows/libc-patch-coverage.yml'
+ - '!libc/docs/**'
+ - '!libc/benchmarks/**'
+ - '!libc/fuzzing/**'
+ - '!libc/utils/**'
+ - '!**.md'
+ pull_request:
+ paths:
+ - 'libc/src/**'
+ - 'libc/include/**'
+ - 'libc/test/**'
+ - 'libc/CMakeLists.txt'
+ - '.github/workflows/libc-patch-coverage.yml'
+ - '!libc/docs/**'
+ - '!libc/benchmarks/**'
+ - '!libc/fuzzing/**'
+ - '!libc/utils/**'
+ - '!**.md'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ pre-commit-coverage:
+ timeout-minutes: 60
+ name: libc-pre-commit-coverage
+ runs-on: ubuntu-24.04
+ container:
+ image: ghcr.io/llvm/libc-ubuntu-24.04:latest@sha256:8fee4c9ce5a1fd095686593cd36e032c48a31b6ae575378c82accd1d86a08d59
+ options: >-
+ --privileged
+ defaults:
+ run:
+ shell: bash
+ steps:
+ - name: Checkout Code
+ uses: actions/checkout@v4
+ with:
+ ref: ${{ github.event.pull_request.head.sha || github.sha }}
+ fetch-depth: 2
+ persist-credentials: false
+
+ - name: Setup Compiler Cache (sccache)
+ uses: hendrikmuhs/ccache-action@v1.2.23
+ with:
+ max-size: 1G
+ key: libc_coverage_unified_v2_x86_64
+ variant: sccache
+
+ - name: Configure CMake
+ run: |
+ echo "=== [STAGE 1/4] Starting CMake Configuration at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ START_TIME=$(date +%s)
+
+ export CMAKE_FLAGS="
+ -G Ninja
+ -S runtimes
+ -B build-cov
+ -DCMAKE_C_COMPILER=clang-23
+ -DCMAKE_CXX_COMPILER=clang++-23
+ -DCMAKE_BUILD_TYPE=Debug
+ -DCMAKE_C_COMPILER_LAUNCHER=sccache
+ -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
+ -DLLVM_USE_LINKER=lld-23
+ -DLLVM_ENABLE_RUNTIMES=libc
+ -DLLVM_LIBC_FULL_BUILD=ON
+ -DLIBC_ENABLE_COVERAGE=ON
+ -DLIBC_TEST_UNIT_TEST_ONLY=ON
+ -DLIBC_TEST_SKIP_DEATH_TESTS=ON
+ -DLIBC_TEST_SKIP_SHARED_TESTS=ON
+ "
+ cmake $CMAKE_FLAGS
+
+ END_TIME=$(date +%s)
+ echo "=== [STAGE 1/4] CMake Configuration Completed in $((END_TIME - START_TIME))s ==="
+
+ - name: Build and Run Targeted Tests
+ env:
+ EVENT_NAME: ${{ github.event_name }}
+ PR_BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+ PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
+ PR_HEAD_REF: ${{ github.event.pull_request.head.ref }}
+ PR_BASE_REPO: ${{ github.event.pull_request.base.repo.full_name }}
+ PR_HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }}
+ COMMIT_SHA: ${{ github.sha }}
+ BRANCH_REF: ${{ github.ref_name }}
+ REPO_NAME: ${{ github.repository }}
+ run: |
+ echo "=== [STAGE 2/4] Starting Targeted Test Discovery at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ START_TIME=$(date +%s)
+
+ # Ensure upstream llvm-project remote is configured
+ git remote add upstream https://github.com/llvm/llvm-project.git 2>/dev/null || true
+
+ # 1. Deterministically resolve Base and Head references against upstream llvm-project
+ if [ "$EVENT_NAME" == "pull_request" ]; then
+ BASE_SHA="$PR_BASE_SHA"
+ HEAD_SHA="$PR_HEAD_SHA"
+ BASE_REF="$PR_BASE_REF"
+ HEAD_REF="$PR_HEAD_REF"
+ BASE_REPO="${PR_BASE_REPO:-llvm/llvm-project}"
+ HEAD_REPO="${PR_HEAD_REPO:-$REPO_NAME}"
+ git fetch upstream "$BASE_SHA" --depth=1 2>/dev/null || git fetch origin "$BASE_SHA" --depth=1 2>/dev/null || git fetch origin "$BASE_REF" --depth=1 2>/dev/null || true
+ DIFF_BASE="$BASE_SHA"
+ else
+ # For direct pushes, compare the pushed commit against its parent (HEAD~1)
+ BASE_SHA=$(git rev-parse HEAD~1 2>/dev/null || echo "HEAD~1")
+ HEAD_SHA="$COMMIT_SHA"
+ BASE_REF="main"
+ HEAD_REF="$BRANCH_REF"
+ BASE_REPO="llvm/llvm-project"
+ HEAD_REPO="$REPO_NAME"
+ DIFF_BASE="HEAD~1"
+ fi
+
+ # Persist resolved commit references for all downstream steps
+ echo "DIFF_BASE=$DIFF_BASE" >> $GITHUB_ENV
+ echo "BASE_SHA=$BASE_SHA" >> $GITHUB_ENV
+ echo "HEAD_SHA=$HEAD_SHA" >> $GITHUB_ENV
+ echo "BASE_REF=$BASE_REF" >> $GITHUB_ENV
+ echo "HEAD_REF=$HEAD_REF" >> $GITHUB_ENV
+ echo "BASE_REPO=$BASE_REPO" >> $GITHUB_ENV
+ echo "HEAD_REPO=$HEAD_REPO" >> $GITHUB_ENV
+
+ echo "[LOG] Resolved Base: ${BASE_REF} (${BASE_SHA:0:7}) in ${BASE_REPO}"
+ echo "[LOG] Resolved Head: ${HEAD_REF} (${HEAD_SHA:0:7}) in ${HEAD_REPO}"
+
+ MODIFIED_FILES=$(git diff --name-only "$DIFF_BASE" HEAD -- libc/src/ || true)
+ echo "[LOG] Modified files in libc/src/:"
+ echo "$MODIFIED_FILES"
+
+ if [ -z "$MODIFIED_FILES" ]; then
+ echo "[LOG] No source files modified in libc/src/. Exiting successfully."
+ echo "TARGETS=" >> $GITHUB_ENV
+
+ echo "## LLVM-libc Patch Coverage Report" > coverage_report.md
+ echo "" >> coverage_report.md
+ echo "- **Base Branch:** [\`${BASE_REF}\` (${BASE_SHA:0:7})](https://github.com/${BASE_REPO}/commit/${BASE_SHA})" >> coverage_report.md
+ echo "- **Head Commit:** [\`${HEAD_REF}\` (${HEAD_SHA:0:7})](https://github.com/${HEAD_REPO}/commit/${HEAD_SHA})" >> coverage_report.md
+ echo "---" >> coverage_report.md
+ echo "" >> coverage_report.md
+ echo "**Coverage Validated**" >> coverage_report.md
+ echo "No \`.cpp\` source files in \`libc/src/\` were modified in this patch." >> coverage_report.md
+
+ cat coverage_report.md >> $GITHUB_STEP_SUMMARY
+ exit 0
+ fi
+
+ # Get all valid ninja targets
+ ninja -C build-cov -t targets > all_targets.txt
+
+ TARGETS=""
+
+ for FILE in $MODIFIED_FILES; do
+ # Skip core support files from direct target mapping (tested transitively)
+ if [[ "$FILE" == *"__support/"* ]]; then
+ echo "[LOG] -> Support file ($FILE) will be tested transitively."
+ continue
+ fi
+
+ if [[ "$FILE" =~ ^libc/src/([^/]+)/(.*/)?([^/]+)\.cpp$ ]]; then
+ DIR="${BASH_REMATCH[1]}"
+ FUNC="${BASH_REMATCH[3]}"
+ ALL_MATCHES=$(grep -oE "^libc\.test\.src\.${DIR}\.([a-zA-Z0-9_]+\.)*${FUNC}_test\.__unit__" all_targets.txt || true)
+ FOUND_TARGET=$(echo "$ALL_MATCHES" | head -n 1 | tr -d '\r\n ' || true)
+ if [ -n "$FOUND_TARGET" ]; then
+ echo "[LOG] -> Matched $FILE -> $FOUND_TARGET"
+ TARGETS="$TARGETS $FOUND_TARGET"
+ else
+ echo "[LOG] -> Notice: No direct unit test target found for $FILE."
+ fi
+ fi
+ done
+
+ # Remove duplicate targets
+ TARGETS=$(echo "$TARGETS" | xargs -n1 | sort -u | xargs || true)
+ echo "[LOG] Executing Ninja targets: $TARGETS"
+
+ if [ -n "$TARGETS" ]; then
+ export LLVM_PROFILE_FILE="libc_cov_%p.profraw"
+ ninja -C build-cov $TARGETS
+ echo "TARGETS=$TARGETS" >> $GITHUB_ENV
+ else
+ echo "[LOG] No standalone unit test targets to run."
+ echo "TARGETS=" >> $GITHUB_ENV
+
+ echo "## LLVM-libc Patch Coverage Report" > coverage_report.md
+ echo "" >> coverage_report.md
+ echo "- **Base Branch:** [\`${BASE_REF}\` (${BASE_SHA:0:7})](https://github.com/${BASE_REPO}/commit/${BASE_SHA})" >> coverage_report.md
+ echo "- **Head Commit:** [\`${HEAD_REF}\` (${HEAD_SHA:0:7})](https://github.com/${HEAD_REPO}/commit/${HEAD_SHA})" >> coverage_report.md
+ echo "---" >> coverage_report.md
+ echo "" >> coverage_report.md
+ echo "**Coverage Validated**" >> coverage_report.md
+ echo "Modified files do not have standalone unit test targets." >> coverage_report.md
+
+ cat coverage_report.md >> $GITHUB_STEP_SUMMARY
+ fi
+
+ END_TIME=$(date +%s)
+ echo "=== [STAGE 2/4] Targeted Tests Completed in $((END_TIME - START_TIME))s ==="
+
+ - name: Merge Profiles
+ if: env.TARGETS != ''
+ run: |
+ echo "=== [STAGE 3/4] Merging Raw Profiles at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ find build-cov -name "libc_cov_*.profraw" > profraw_list.txt
+ NUM_PROFS=$(wc -l < profraw_li...
[truncated]
|
| echo "[LOG] Post-commit summary report written to GITHUB_STEP_SUMMARY." | ||
|
|
||
| - name: Deploy Coverage Dashboard to GitHub Pages | ||
| if: github.repository != 'llvm/llvm-project' |
There was a problem hiding this comment.
We need a better solution than just using the GH Pages branch of the monorepo. For now we should probably just upload the report as an artifact.
There was a problem hiding this comment.
Hi Aiden, I've removed the GitHub Pages deployment step entirely and replaced it with actions/upload-artifact to upload the report.
Replace the third-party peaceiris/actions-gh-pages deployment action with standard actions/upload-artifact@v4 in both full coverage workflows. This aligns with LLVM CI security best practices by removing unnecessary contents: write repository permissions and untrusted third-party actions. Harden raw profile discovery and report generation with defensive guards. Configure overwrite: true on artifact uploads to ensure robustness across workflow re-runs. Update codebase_coverage.py and associated unit tests to direct users to the workflow Artifacts section. Assisted-by: Automated tooling, human reviewed.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
✅ With the latest revision this PR passed the Python code formatter. |
| runs-on: ubuntu-24.04 | ||
| container: | ||
| image: ghcr.io/llvm/libc-ubuntu-24.04:latest@sha256:8fee4c9ce5a1fd095686593cd36e032c48a31b6ae575378c82accd1d86a08d59 | ||
| options: --privileged |
There was a problem hiding this comment.
Why do you need this --privileged flags?
There was a problem hiding this comment.
The --privileged flag was a leftover from earlier container experiments and wasn't actually needed for these tests. I've removed it.
| END_TIME=$(date +%s) | ||
| echo "=== [STAGE 1/4] CMake Configuration Completed in $((END_TIME - START_TIME))s ===" |
There was a problem hiding this comment.
Why do you need to track how long this step takes? GtiHub already tracks this.
There was a problem hiding this comment.
I originally added that to measure stage timings while benchmarking build performance and caching in local container runs. I've removed it now.
Pin workflow actions to 40-character commit hashes to comply with Zizmor security policy. Format codebase and diff coverage Python utilities and unit test suites to conform to Black/darker standards (88-character limit, standard spacing). Remove unnecessary container --privileged flag and redundant manual step timing logic from coverage workflows. Assisted-by: Automated tooling, human reviewed.
| - libc-mcdc-coverage | ||
| - post-commit-bot |
There was a problem hiding this comment.
What are these branches for?
There was a problem hiding this comment.
Those were leftover branches used for testing the workflows during development on my fork. I've removed them so the push triggers only target main.
|
@tapiwagonga Was this patch originally generated using AI or have you only been using AI for the updates? |
The original patch wasn't generated by AI, I wrote the initial implementation myself. I've mainly been using AI as an assistant to help turn around review comments and formatting updates faster, but I review, test, and verify all the changes myself before committing and pushing. |
This is not necessary. There was already an RFC. Please stop writing low-quality comments on PRs and only post when you're confident what you're saying is correct/helpful. |
Remove development and testing branch filters from coverage workflows so they only trigger on upstream main and pull requests targeting main. Assisted-by: Automated tooling, human reviewed.
tstellar
left a comment
tstellar
left a comment
There was a problem hiding this comment.
Can you explain why this is split into 4 separate workflows instead of just 1 or 2?
The overall idea was to track both standard statement/branch coverage and MC/DC (Modified Condition / Decision Coverage), and to do so across two distinct levels of scope: incrementally on every pull request commit so developers and contributors get immediate visibility into the coverage of their commits directly on their pull requests, and globally across the entire codebase as a recurring baseline. To support this, I set up a dedicated bot for each of the four configurations:
|
…plify CI summaries Streamline test_diff_coverage.py by 41% (reducing 360+ lines of fixture boilerplate) while preserving 100% statement and branch coverage. Harden diff and codebase coverage utilities against C++ syntax edge cases, including closing braces with namespace comments, C++ access specifiers, and compile-time declarations (static_assert, friend). Filter non-source files from patch diff evaluations and normalize repository URLs for GitHub links. Remove the redundant sticky PR comment step from patch workflows in favor of standard GITHUB_STEP_SUMMARY and artifact uploads, restricting workflow permissions strictly to contents: read to avoid fork token permission failures. Add downloadable artifact packaging with if-no-files-found guards, merge-base diff resolution, and executable verification guards across CI. Assisted-by: Automated tooling, human reviewed.
|
|
||
|
|
||
| - name: Upload Patch Coverage Artifacts | ||
| if: always() && hashFiles('coverage_report.md') |
There was a problem hiding this comment.
This should be !cancelled() instead of always(). https://docs.github.com/en/actions/reference/workflows-and-actions/expressions#always
|
@tapiwagonga libc-full-* files look almost identical. Would it be possible to merged those into one workflow and use a matrix? Same thing for the libc-patch-* files. It looks like there is a lot of cut and paste duplication. |
Merge full and patch coverage workflows into matrix jobs across line and MC/DC coverage, eliminating duplicate workflow definitions. Rename unit test files to patch_coverage_test.py and full_coverage_test.py, and remove synthetic commit SHAs, dummy branch names, and mock repository fixtures from test cases. Remove redundant Node.js environment settings and ensure strict compliance with darker, Black, and git-clang-format. Assisted-by: Automated tooling, human reviewed.
@tstellar I have consolidated both sets of workflows into single matrix pipelines. The libc-full-* workflows are now merged into |
Assisted-by: Automated tooling, human reviewed.
Temporarily enable patch coverage workflow runs on libc-coverage-bots for testing purposes. Assisted-by: Automated tooling, human reviewed.
Assisted-by: Automated tooling, human reviewed.
…e on isalpha Assisted-by: Automated tooling, human reviewed.
…files Assisted-by: Automated tooling, human reviewed.
…ild directory Assisted-by: Automated tooling, human reviewed.
…date isdigit for testing patch coverage This commit updates commit reference resolution, adds diagnostic workflow logging, and modifies isdigit for testing purposes to validate the patch coverage pipeline. Assisted-by: Automated tooling, human reviewed.
…sdigit This commit switches the patch coverage bot to overlay build mode to resolve profile runtime linker errors, adds LIBC_ENABLE_MCDC support, and updates isdigit for testing purposes to validate the patch coverage pipeline. Assisted-by: Automated tooling, human reviewed.
…porting Replace cryptic MC/DC summaries with structured condition diagnostics and actionable test guidance. Ensure clean unified diff formatting and add GitHub Flavored Markdown summary callouts. Assisted-by: Automated tooling, human reviewed.
…testing Update local bounds checking variables in isdigit to validate the coverage artifact formatting pipeline on GitHub Actions. Assisted-by: Automated tooling, human reviewed.
Remove alert callouts in diff_coverage.py and package interactive HTML. Synchronize patch_coverage_test.py unit tests with updated report structure. Update isalpha bounds checking to trigger and exercise the coverage bot. Assisted-by: Automated tooling, human reviewed.
Align full codebase and patch coverage workflows with Clang continuous profiling mode. Add repository owner guards, isolate sccache keys, persist modified file lists across workflow steps, and correct trigger path filters. Re-add Clang compiler checks and remove compiler-rt profile path-guessing in CMakeLists. Revert temporary testing modifications in ctype source files. Assisted-by: Automated tooling, human reviewed.
Configure libc-full-coverage and libc-patch-coverage workflows to strictly use full-build mode (LLVM_LIBC_FULL_BUILD=ON) and target libc-hermetic-tests. Align CMake continuous profiling options with LIBC_ENABLE_COVERAGE_MCDC and use -fprofile-instr-generate=libc_cov_%p.profraw. Assisted-by: Automated tooling, human reviewed.
|
@tstellar PTAL |
tstellar
left a comment
tstellar
left a comment
There was a problem hiding this comment.
I've tried to review the workflow structure as best as I can. I don't really understand what this is doing and the code is hard for me to follow. You should have someone more familiar with libc take a look at this too.
| export CMAKE_FLAGS=" | ||
| -G Ninja | ||
| -S runtimes | ||
| -B build-cov | ||
| -DCMAKE_C_COMPILER=clang-23 | ||
| -DCMAKE_CXX_COMPILER=clang++-23 | ||
| -DCMAKE_BUILD_TYPE=Debug | ||
| -DCMAKE_C_COMPILER_LAUNCHER=sccache | ||
| -DCMAKE_CXX_COMPILER_LAUNCHER=sccache | ||
| -DLLVM_USE_LINKER=lld-23 | ||
| -DLLVM_ENABLE_RUNTIMES=libc | ||
| -DLLVM_LIBC_FULL_BUILD=ON | ||
| -DLIBC_ENABLE_COVERAGE=ON | ||
| -DLIBC_ENABLE_COVERAGE_MCDC=${{ matrix.enable_mcdc }} | ||
| -DLIBC_TEST_SKIP_DEATH_TESTS=ON | ||
| -DLIBC_TEST_SKIP_SHARED_TESTS=ON | ||
| " |
There was a problem hiding this comment.
Why use an environment variable here? Can't you just pass this directly to cmake?
| -DLIBC_TEST_SKIP_DEATH_TESTS=ON | ||
| -DLIBC_TEST_SKIP_SHARED_TESTS=ON | ||
| " | ||
| cmake $CMAKE_FLAGS |
| REPO_NAME: ${{ github.repository }} | ||
| run: | | ||
| # Ensure upstream llvm-project remote is configured | ||
| git remote add upstream https://github.com/llvm/llvm-project.git 2>/dev/null || true |
This branch has not been deployed
5 participants
I previously wrote an RFC regarding Enabling Code Coverage in LLVM-libc to address the circular dependencies and link-time failures caused by standard compiler-rt profiling in hermetic (-nostdlib) testing environments. After incorporating community feedback, the core freestanding instrumentation was merged in Code Coverage via Linux Syscalls.
Following that integration, I evaluated how best to surface this coverage data automatically within the monorepo. This resulted in a second RFC proposing an Automated Code Coverage Infrastructure utilising GitHub Actions to track coverage on commit. This pull request implements that infrastructure.
Implementation Details
The architecture of this infrastructure is explicitly decoupled into execution workflows and zero-dependency analytical scripts. The pipeline is entirely self-contained within the monorepo. The GitHub Actions workflows compile the test suite, generate the raw coverage payloads using
llvm-cov, and pipe that data into the Python analysers to render native Markdown summaries.This submission consists of 8 cohesive files:
GitHub Actions Workflows (
.github/workflows/):Execution Model:These workflows leverage the official
ghcr.io/llvm/libc-ubuntu-24.04container. By using a pre-installed toolchain and building exclusively at the runtimes layer (-DLLVM_ENABLE_RUNTIMES=libc), they eliminate the overhead of compiling Clang from source.Pre-commit bots (
libc-patch-coverage.yml, libc-patch-mcdc.yml)that evaluate patch diffs and provide actionable diagnostics regarding unexecuted lines and unverified boolean conditions on pull requests.Post-commit bots (
libc-full-coverage.yml, libc-full-mcdc.yml) that aggregate repository-wide metrics across all directories on a daily schedule.In-Tree Reporting Analysers (
libc/utils/coverage/):diff_coverage.py: A Python utility that intercepts llvm-cov JSON exports, correlates them against unified Git diffs, filters out non-executable code lines (comments, macros, braces), and strictly evaluates MC/DC bitmasks to determine patch completeness.codebase_coverage.py: Aggregates full repository execution segments into top-level directories (e.g.,src/math, src/string), It implements explicit path filtering to drop internal mock dependencies and test infrastructure (libc/test/), ensuring the final metrics accurately reflect the core library implementation.Unit Tests
The Python reporting scripts are supported by fast-executing unit test suites that verify the underlying analysis algorithms using synthetic, in-memory data structures:
test_diff_coverage.py: Verifies the integrity of the patch evaluation pipeline, including Git diff parsing, C++ heuristic filtering, segment intersection math, and MC/DC boolean extraction.test_codebase_coverage.py: Verifies the global repository tracking logic, specifically asserting directory-level metric aggregation and infrastructure path exclusion.