Skip to content

ci: check benchmark timings only without coverage - #816

Open
haakon-e wants to merge 1 commit into
mainfrom
he/ci-coverage-timing
Open

haakon-e wants to merge 1 commit into
mainfrom
he/ci-coverage-timing

Conversation

@haakon-e

Copy link
Copy Markdown
Member

Purpose

Make the timing checks in test/performance_tests.jl measure the code instead of the coverage instrumentation. They fail on every Julia 1.12 job since #811, although the code runs well within its budgets.

Content

The ci jobs ran the tests with coverage. On Julia 1.12, coverage slows the 1M bulk tendencies about 20-fold, so the timing budgets failed:

LinearizedAverage, Float64 Julia 1.10 Julia 1.12
without coverage 0.43 μs 0.41 μs
with coverage 3.1 μs 8.9 μs (budget 5.5 μs)
with coverage, 3 substeps 9.3 μs 26.4 μs (budget 18 μs)

Measured locally on main (Apple M-series); the coverage numbers on 1.12 match those in the CI logs of #811, #797 and #801.

  • The ci jobs now run without coverage, so their benchmarks time the code itself. Coverage comes from the CodeCov workflow alone, which already runs the test suite with coverage on every pull request and uploads it.
  • bench_press skips the timing check when coverage is on and keeps the allocation checks. Without this, a timing failure in the CodeCov job would fail the job and skip its coverage upload.

Locally on Julia 1.12, all 536 performance checks pass without coverage. With coverage, the 134 timing checks are reported as broken and the 402 others pass.

Codecov now receives one upload per push, from Julia 1.11 on Ubuntu, instead of seven, so lines that run only on another Julia version or operating system no longer count as covered.

This supersedes #760.

Breaking changes

None.

Coverage instrumentation slows the 1M bulk tendencies about 20-fold on
Julia 1.12 (LinearizedAverage: 0.41 us plain, 8.9 us with coverage), so
the timing budgets in test/performance_tests.jl measured the
instrumentation and failed on the 1.12 jobs.

The ci jobs now run without coverage, so their benchmarks time the code
itself. Coverage comes from the CodeCov workflow alone, which already
uploaded it on every pull request. Under coverage, bench_press skips the
timing check and keeps the allocation checks.
@codecov

codecov Bot commented Oct 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.89%. Comparing base (542dd84) to head (06b2c39).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #816      +/-   ##
==========================================
+ Coverage   93.17%   93.89%   +0.71%     
==========================================
  Files          59       55       -4     
  Lines        3166     3078      -88     
==========================================
- Hits         2950     2890      -60     
+ Misses        216      188      -28     
Components Coverage Δ
src 93.89% <ø> (-0.02%) ⬇️
ext ∅ <ø> (∅)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant