(WIP) Optimize test workflows #6091
Draft
+243
−26
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Great job! No new security vulnerabilities introduced in this pull request |
Contributor
|
Warning @SaintPatrck Uploading code coverage report failed. Please check the "Upload to codecov.io" step of [https://github.com/bitwarden/android/actions/runs/18896909331/attempts/1] for more details. |
Contributor
Author
|
For comparison: https://github.com/bitwarden/android/actions/runs/18896909331 contains changes from #6087 Build without optimization (build cache present):Build with optimization (build cache not present):Initial ~13% decrease in build time. Optimized workflow did not have an established build cache to pull from so gains are expected to be higher. |
SaintPatrck
force-pushed
the
optimize-test-workflows
branch
from
2c18341 to
1d9f786
Compare
Contributor
|
Warning @SaintPatrck Uploading code coverage report failed. Please check the "Upload to codecov.io" step of [https://github.com/bitwarden/android/actions/runs/18919692963/attempts/1] for more details. |
Contributor
Author
|
For comparison:
Follow-up run exhibits ~23% decrease from the baseline. Compared to the initial measure, there was a further ~11% decrease in build time. 🥳 Seeing improvements over the previous measurement. Improvements are potentially due to partial cache hits from recent workflow executions on |
Contributor
|
Warning @SaintPatrck Uploading code coverage report failed. Please check the "Upload to codecov.io" step of [https://github.com/bitwarden/android/actions/runs/18926405325/attempts/1] for more details. |
- Add org.gradle.configuration-cache=true - Set problems to warn level to identify compatibility issues - Expected 5-15% improvement in Gradle configuration phase - Part of test workflow optimization (Tier 1)
Major changes: - Split single test job into 5 parallel jobs: 1. lint-and-static-analysis: Run detekt and lint checks 2. test-libraries: Test core, data, network, ui, authenticatorbridge, cxf 3. test-app: Test app module (5,447 tests) 4. test-authenticator: Test authenticator module (281 tests) 5. aggregate-coverage: Merge coverage reports and upload to codecov Benefits: - 40-55% expected reduction in CI runtime - Early failure detection (lint and library tests complete quickly) - Better resource utilization across 5 concurrent jobs - Maintains existing coverage reporting behavior Technical details: - Each test job uploads test reports and coverage data as artifacts - Coverage aggregation job downloads all coverage artifacts - Uses Fastlane's koverXmlReportMergedCoverage for merging - Preserves all existing codecov.io integration
Problem: The aggregate-coverage job was failing because koverXmlReportMergedCoverage requires all tests to have run in the same Gradle session. When tests are split across separate jobs, Kover has no test execution data to merge. Root Cause: - Kover collects binary coverage data during test execution - koverXmlReportMergedCoverage merges this data from the current build - In parallel jobs, the aggregation job never ran any tests - Result: No coverage data to merge, causing exit code 1 Solution: 1. Each test job now generates its own XML coverage report immediately after running tests: - test-libraries: koverXmlReportDebug for each library module - test-app: koverXmlReportStandardDebug - test-authenticator: koverXmlReportDebug 2. Coverage artifacts now contain XML reports, not just binary data 3. Aggregation job simplified: - Remove Fastlane/Ruby setup (no longer needed) - Download all coverage-* artifacts - Upload entire coverage-reports/ directory to codecov - Codecov automatically merges multiple XML files Benefits: - Each module's coverage is captured independently - No dependency on cross-job Gradle state - Codecov handles merging (tested and reliable) - Simpler, more maintainable workflow Technical Details: - Kover generates XML reports at: module/build/reports/kover/report*.xml - Codecov action with directory parameter finds all XML files recursively - disable_search=false allows automatic file discovery
Problem:
Running tests and coverage report generation as separate Gradle invocations
caused a ConcurrentModificationException when Kover tried to access test
execution data:
Could not determine the dependencies of task ':app:koverGenerateArtifactStandardDebug'
> java.util.ConcurrentModificationException (no error message)
Root Cause:
- Configuration cache is enabled (improves build performance)
- Kover's coverage tasks need test execution data from the same Gradle session
- Separate invocations = separate Gradle daemon sessions
- Coverage report task cannot access test data from previous invocation
- Configuration cache + separate invocations = state corruption
Solution:
Combine test execution and coverage generation into single Gradle command:
BEFORE: ./gradlew :app:testStandardDebug
./gradlew :app:koverXmlReportStandardDebug # Fails!
AFTER: ./gradlew :app:testStandardDebug :app:koverXmlReportStandardDebug
Benefits:
- Both tasks run in same Gradle daemon session
- Coverage task has access to test execution data
- Configuration cache works correctly
- No state corruption between invocations
Applied to all test jobs:
- test-libraries: Combined 6 test + 6 coverage tasks
- test-app: Combined test + coverage
- test-authenticator: Combined test + coverage
This commit adds a new step to the `test.yml` GitHub Actions workflow to extract coverage reports from downloaded artifacts. Previously, the workflow would download coverage report artifacts but did not extract them before the upload step. This change adds a new step named "Extract coverage reports from archives" that finds all `.tar` and `.zip` files within the `coverage-reports/` directory and extracts their contents. This ensures that the individual XML coverage files are available for the subsequent "Upload to codecov.io" step.
This commit updates the `test.yml` GitHub workflow to correctly process and locate code coverage reports before uploading them to Codecov. The previous steps extracted coverage reports from `.tar` and `.zip` archives into subdirectories within the `coverage-reports` directory. This commit adds a step to move all extracted `.xml` files from these subdirectories into the root of the `coverage-reports` directory. Additionally, it cleans up the original `.tar` and `.zip` archives after extraction to save space. This ensures that the Codecov action can find all the necessary XML reports in the expected location.
SaintPatrck
force-pushed
the
optimize-test-workflows
branch
from
b2f1cf1 to
11d9537
Compare
Contributor
|
Warning @SaintPatrck Uploading code coverage report failed. Please check the "Upload to codecov.io" step of [https://github.com/bitwarden/android/actions/runs/18942244831/attempts/1] for more details. |
Contributor
|
Warning @SaintPatrck Uploading code coverage report failed. Please check the "Upload to codecov.io" step of [https://github.com/bitwarden/android/actions/runs/18942334928/attempts/1] for more details. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🎟️ Tracking
📔 Objective
This PR restructures the test workflow to execute tests in parallel across multiple jobs, with the goal of significantly reducing CI runtime. The workflow now splits testing into 5 concurrent jobs instead of a single sequential job.
Workflow Structure
Before: Single job running all tasks sequentially
After: 5 parallel jobs with specialized responsibilities
Jobs Breakdown
1.
lint-and-static-analysisdetektfor code quality checkslintStandardDebugandlintDebug2.
test-libraries:core,:data,:network,:ui,:authenticatorbridge,:cxf3.
test-app:appmodule (5,447 tests - 88.5% of total)4.
test-authenticator:authenticatormodule (281 tests - 4.6% of total)5.
aggregate-coveragetest-libraries,test-app,test-authenticatorTechnical Changes
1. Enable Gradle Configuration Cache
File:
gradle.propertiesBenefits:
2. Restructure Test Workflow
File:
.github/workflows/test.ymlKey Changes:
testjob into 5 specialized jobsCoverage Strategy:
coverage-libraries,coverage-app,coverage-authenticator)3. Combined Test + Coverage Commands
Tests and coverage generation run in a single Gradle invocation to ensure Kover has access to test execution data:
Running both tasks together in the same Gradle daemon session ensures the coverage report generation can access the test execution data produced by the test task.
Performance Impact
Expected Improvements
Test Distribution
:appmodule (still bottleneck):authenticatormoduleTesting & Validation
Local Testing
CI Validation Checklist
Future Optimization Opportunities
This PR implements Tier 1 (Quick Wins) optimizations. Additional improvements are possible:
Tier 2: App Module Sharding
:appmodule's 5,447 tests across 3-4 parallel jobsTier 3: Strategic Investments
:app)Risks & Mitigations
Risk: Coverage Reporting Changes
Mitigation: Codecov's XML merging is battle-tested and widely used. Coverage % should match previous implementation.
Risk: Workflow Complexity Increase
Mitigation: Clear job naming, comprehensive documentation, each job remains simple and focused.
Risk: More CI Runner Usage
Impact: Uses 5 concurrent runners instead of 1. Total runner-minutes may be similar or slightly higher, but wall-clock time is significantly reduced.
Risk: Configuration Cache Issues
Mitigation: Enabled with
problems=warnto identify compatibility issues without failing builds. Can be disabled if problems arise.Rollback Plan
If issues arise:
gradle.propertieschanges📸 Screenshots
Coming soon!
⏰ Reminders before review
🦮 Reviewer guidelines
:+1:) or similar for great changes:memo:) or ℹ️ (:information_source:) for notes or general info:question:) for questions:thinking:) or 💭 (:thought_balloon:) for more open inquiry that's not quite a confirmed issue and could potentially benefit from discussion:art:) for suggestions / improvements:x:) or:warning:) for more significant problems or concerns needing attention:seedling:) or ♻️ (:recycle:) for future improvements or indications of technical debt:pick:) for minor or nitpick changes