diff --git a/.github/PUBLISHING_WORKFLOWS.md b/.github/PUBLISHING_WORKFLOWS.md new file mode 100644 index 0000000..dc9e804 --- /dev/null +++ b/.github/PUBLISHING_WORKFLOWS.md @@ -0,0 +1,218 @@ +# GitHub Actions Publishing Workflows - Documentation + +## Overview + +This document describes the publishing workflows for the benpy package to PyPI and TestPyPI. + +## Workflows + +### 1. test_publish.yml - TestPyPI Publishing + +**Purpose**: Test the publication process using TestPyPI before publishing to production PyPI. + +**Triggers**: +- Automatically when a GitHub release is published +- Manual workflow dispatch with confirmation input + +**Usage (Automatic)**: +1. Create a new release in GitHub +2. Tag the release (e.g., `v2.1.0`) +3. Publish the release +4. The workflow will automatically trigger and publish to TestPyPI + +**Usage (Manual)**: +1. Go to Actions tab in GitHub +2. Select "Publish to TestPyPI" workflow +3. Click "Run workflow" +4. Type `test-publish` in the confirmation field +5. Click "Run workflow" + +**What it does**: +- Builds the source distribution (sdist) +- Publishes to TestPyPI for testing +- Attempts to verify installation from TestPyPI + +**Requirements**: +- Configure TestPyPI trusted publishing OR set `TESTPYPI_API_TOKEN` secret +- Create `testpypi` environment in repository settings + +### 2. publish_pypi.yml - Production PyPI Publishing + +**Purpose**: Publish the package to production PyPI after successful testing. + +**Trigger**: Manual workflow dispatch only (after confirming TestPyPI works) + +**Usage**: +1. First, verify the package works on TestPyPI (published automatically on release) +2. Go to Actions tab in GitHub +3. Select "Publish to PyPI" workflow +4. Click "Run workflow" +5. Type `publish` in the confirmation field +6. Click "Run workflow" + +**What it does**: +- Verifies manual confirmation +- Builds the source distribution (sdist) +- Optionally downloads pre-built wheels (if available from build-wheels.yml) +- Publishes to production PyPI +- Verifies the publication by installing from PyPI + +**Requirements**: +- Configure PyPI trusted publishing OR set `PYPI_API_TOKEN` secret +- Create `pypi` environment in repository settings + +## Changes Made + +### Fixed Issues + +1. **Build Artifacts Job**: + - Previous: Only printed placeholder messages + - Now: Actually builds and uploads the source distribution + - Added GLPK system dependencies installation + - Added Python build dependencies installation + +2. **Workflow Separation**: + - Previous: Single workflow tried to handle both TestPyPI and PyPI + - Now: Separate workflows for testing and production + - `test_publish.yml` for TestPyPI testing + - `publish_pypi.yml` for production PyPI + +3. **Artifact Handling**: + - Fixed artifact download to handle missing wheels gracefully + - Source distribution is always built and uploaded + - Wheels are optional (can be built separately via build-wheels.yml) + +### Workflow Structure + +Both workflows follow similar patterns: + +``` +verify → build_artifacts → [download_artifacts] → publish → verify_publication +``` + +**verify**: Checks trigger conditions and confirmations +**build_artifacts**: Builds the source distribution +**download_artifacts**: (PyPI only) Consolidates artifacts +**publish**: Uploads to PyPI/TestPyPI +**verify_publication**: Tests installation from the repository + +## PyPI Trusted Publishing + +Trusted publishing is the recommended authentication method for GitHub Actions. + +### Setup for PyPI: + +1. Go to https://pypi.org/manage/account/publishing/ +2. Add a new publisher: + - PyPI Project Name: `benpy` + - Owner: `markobud` + - Repository: `benpy` + - Workflow: `publish_pypi.yml` + - Environment: `pypi` + +### Setup for TestPyPI: + +1. Go to https://test.pypi.org/manage/account/publishing/ +2. Add a new publisher: + - PyPI Project Name: `benpy` + - Owner: `markobud` + - Repository: `benpy` + - Workflow: `test_publish.yml` + - Environment: `testpypi` + +## Alternative: API Token Authentication + +If you prefer API tokens instead of trusted publishing: + +1. Generate an API token from PyPI/TestPyPI +2. Add it as a repository secret: + - `PYPI_API_TOKEN` for PyPI + - `TESTPYPI_API_TOKEN` for TestPyPI +3. Uncomment the `password:` line in the workflow files + +## Environments + +Both workflows use GitHub Environments for additional protection: + +- `pypi` environment for production publishing +- `testpypi` environment for test publishing + +### Creating Environments: + +1. Go to repository Settings → Environments +2. Create two environments: `pypi` and `testpypi` +3. Optionally add protection rules (e.g., required reviewers) + +## Testing the Workflows + +### Recommended Release Process: + +1. **Create a GitHub release** (e.g., tag `v2.1.0`) + - This automatically triggers `test_publish.yml` + - Package is published to TestPyPI + +2. **Test the TestPyPI package**: + ```bash + pip install --index-url https://test.pypi.org/simple/ benpy + ``` + Note: Dependencies might not be available on TestPyPI, which is expected. + +3. **If testing is successful, publish to PyPI**: + - Go to Actions → "Publish to PyPI" + - Click "Run workflow" + - Type `publish` and run + +### Manual TestPyPI Publishing: + +If you need to test without creating a release: + +```bash +# Trigger the test workflow manually +gh workflow run test_publish.yml -f confirm=test-publish + +# Check the workflow run +gh run list --workflow=test_publish.yml +``` + +## Building with Wheels + +For a complete release including wheels: + +1. First run the `build-wheels.yml` workflow to build platform-specific wheels +2. Then run the `publish_pypi.yml` workflow +3. The publish workflow will download wheels if available from the build-wheels run + +Alternatively, publish just the source distribution and let users build from source. + +## Troubleshooting + +### "Artifact not found" Error + +This was the original issue. It occurred because: +- The build_artifacts job didn't actually build anything +- Solution: We now build and upload the sdist in the build_artifacts job + +### Verification Failures + +If installation verification fails: +- Check if the package was actually uploaded to PyPI +- Wait longer (PyPI can take time to index new packages) +- Check for dependency issues + +### Permission Denied + +- Ensure trusted publishing is configured correctly +- Or ensure API tokens are set as secrets +- Check environment protection rules + +## File Locations + +- Test workflow: `.github/workflows/test_publish.yml` +- Production workflow: `.github/workflows/publish_pypi.yml` +- Wheel building: `.github/workflows/build-wheels.yml` + +## Related Documentation + +- [GitHub Actions Documentation](https://docs.github.com/en/actions) +- [PyPI Publishing Guide](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/) +- [Trusted Publishing](https://docs.pypi.org/trusted-publishers/) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish_pypi.yml similarity index 62% rename from .github/workflows/publish.yml rename to .github/workflows/publish_pypi.yml index a6765a4..cd47a40 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish_pypi.yml @@ -1,15 +1,18 @@ name: Publish to PyPI -# This is a DRAFT workflow for publishing to PyPI -# Before using in production: -# 1. Configure PyPI trusted publishing in your PyPI project settings -# 2. Set up the PYPI_API_TOKEN secret if not using trusted publishing -# 3. Remove the manual approval requirement if desired -# 4. Test with TestPyPI first +# Workflow for publishing to production PyPI +# This workflow: +# 1. Manually triggered only (after testing with TestPyPI) +# 2. Builds the source distribution (sdist) +# 3. Optionally downloads wheels from a previous build-wheels workflow run +# 4. Publishes to production PyPI +# +# Note: For trusted publishing, configure it in PyPI project settings +# Otherwise, set the PYPI_API_TOKEN secret +# +# For testing, use test_publish.yml which publishes to TestPyPI automatically on release on: - release: - types: [published] workflow_dispatch: inputs: confirm: @@ -32,11 +35,9 @@ jobs: - name: Check release conditions id: check run: | - if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then - if [[ "${{ github.event.inputs.confirm }}" != "publish" ]]; then - echo "Manual trigger requires 'publish' confirmation" - exit 1 - fi + if [[ "${{ github.event.inputs.confirm }}" != "publish" ]]; then + echo "Manual trigger requires 'publish' confirmation" + exit 1 fi echo "should_publish=true" >> $GITHUB_OUTPUT @@ -49,10 +50,30 @@ jobs: - name: Checkout code uses: actions/checkout@v4 - - name: Trigger wheel build workflow + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Install system dependencies + run: | + sudo apt-get update -qq + sudo apt-get install -y libglpk-dev glpk-utils + + - name: Install build dependencies run: | - echo "In production, this would trigger the build-wheels.yml workflow" - echo "For now, wheels should be built separately before publishing" + python -m pip install --upgrade pip + pip install build wheel setuptools Cython numpy + + - name: Build source distribution + run: python -m build --sdist + + - name: Upload source distribution + uses: actions/upload-artifact@v4 + with: + name: sdist + path: dist/*.tar.gz + retention-days: 90 download_artifacts: name: Download build artifacts @@ -60,12 +81,13 @@ jobs: runs-on: ubuntu-latest steps: - - name: Download wheels + - name: Download wheels (if available) uses: actions/download-artifact@v4 with: pattern: wheels-* path: dist merge-multiple: true + continue-on-error: true # Wheels may not exist in this run - name: Download source distribution uses: actions/download-artifact@v4 @@ -75,8 +97,13 @@ jobs: - name: List artifacts run: | - ls -lh dist/ - echo "Total artifacts: $(ls dist/ | wc -l)" + ls -lh dist/ || echo "No artifacts found" + if [ -d "dist" ]; then + echo "Total artifacts: $(ls dist/ | wc -l)" + else + echo "No dist directory found" + exit 1 + fi - name: Upload combined artifacts uses: actions/upload-artifact@v4 @@ -85,32 +112,9 @@ jobs: path: dist/* retention-days: 90 - publish_testpypi: - name: Publish to TestPyPI (Optional) - needs: [download_artifacts] - runs-on: ubuntu-latest - environment: - name: testpypi - url: https://test.pypi.org/p/benpy - - steps: - - name: Download all artifacts - uses: actions/download-artifact@v4 - with: - name: all-dist-artifacts - path: dist - - - name: Publish to TestPyPI - uses: pypa/gh-action-pypi-publish@release/v1 - with: - repository-url: https://test.pypi.org/legacy/ - skip-existing: true - verbose: true - continue-on-error: true # Don't fail if TestPyPI upload fails - publish_pypi: name: Publish to PyPI - needs: [download_artifacts, publish_testpypi] + needs: [download_artifacts] runs-on: ubuntu-latest environment: name: pypi @@ -128,13 +132,14 @@ jobs: echo "Verifying artifacts before publication..." ls -lh dist/ - # Check we have wheels and sdist - if [ ! -f dist/*.tar.gz ]; then + # Check we have source distribution + if ! ls dist/*.tar.gz >/dev/null 2>&1; then echo "ERROR: No source distribution found" exit 1 fi - if [ -z "$(ls dist/*.whl 2>/dev/null)" ]; then + # Check for wheels (optional) + if ! ls dist/*.whl >/dev/null 2>&1; then echo "WARNING: No wheels found, publishing source only" fi diff --git a/.github/workflows/test_publish.yml b/.github/workflows/test_publish.yml new file mode 100644 index 0000000..59a4a7d --- /dev/null +++ b/.github/workflows/test_publish.yml @@ -0,0 +1,128 @@ +name: Publish to TestPyPI + +# Workflow for testing publication to TestPyPI +# This workflow: +# 1. Automatically triggers when a release is published +# 2. Builds the source distribution (sdist) +# 3. Publishes to TestPyPI for testing before production +# +# Note: Configure TestPyPI trusted publishing or set TESTPYPI_API_TOKEN secret + +on: + release: + types: [published] + workflow_dispatch: + inputs: + confirm: + description: 'Type "test-publish" to confirm TestPyPI publication' + required: true + default: '' + +permissions: + contents: read + id-token: write # Required for trusted publishing + +jobs: + verify_test: + name: Verify test conditions + runs-on: ubuntu-latest + outputs: + should_publish: ${{ steps.check.outputs.should_publish }} + + steps: + - name: Check test conditions + id: check + run: | + if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then + if [[ "${{ github.event.inputs.confirm }}" != "test-publish" ]]; then + echo "Manual trigger requires 'test-publish' confirmation" + exit 1 + fi + fi + echo "should_publish=true" >> $GITHUB_OUTPUT + + build_artifacts: + name: Build test artifacts + needs: [verify_test] + runs-on: ubuntu-latest + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Install system dependencies + run: | + sudo apt-get update -qq + sudo apt-get install -y libglpk-dev glpk-utils + + - name: Install build dependencies + run: | + python -m pip install --upgrade pip + pip install build wheel setuptools Cython numpy + + - name: Build source distribution + run: python -m build --sdist + + - name: Upload source distribution + uses: actions/upload-artifact@v4 + with: + name: sdist + path: dist/*.tar.gz + retention-days: 30 + + publish_testpypi: + name: Publish to TestPyPI + needs: [build_artifacts] + runs-on: ubuntu-latest + environment: + name: testpypi + url: https://test.pypi.org/p/benpy + + steps: + - name: Download artifacts + uses: actions/download-artifact@v4 + with: + name: sdist + path: dist + + - name: List artifacts + run: | + ls -lh dist/ + echo "Total artifacts: $(ls dist/ | wc -l)" + + - name: Publish to TestPyPI + uses: pypa/gh-action-pypi-publish@release/v1 + with: + repository-url: https://test.pypi.org/legacy/ + skip-existing: true + verbose: true + + verify_testpypi: + name: Verify TestPyPI publication + needs: [publish_testpypi] + runs-on: ubuntu-latest + + steps: + - name: Wait for TestPyPI to update + run: sleep 60 + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Install system dependencies + run: | + sudo apt-get update -qq + sudo apt-get install -y libglpk-dev glpk-utils + + - name: Try to install from TestPyPI + run: | + pip install --index-url https://test.pypi.org/simple/ --no-cache-dir benpy || echo "Installation from TestPyPI failed (may need wheels or dependencies)" + # Note: This may fail if dependencies are not on TestPyPI + continue-on-error: true