Fix publish workflow artifact build and separate TestPyPI publishing

.github/PUBLISHING_WORKFLOWS.md

@@ -0,0 +1,218 @@
+# GitHub Actions Publishing Workflows - Documentation
+
+## Overview
+
+This document describes the publishing workflows for the benpy package to PyPI and TestPyPI.
+
+## Workflows
+
+### 1. test_publish.yml - TestPyPI Publishing
+
+**Purpose**: Test the publication process using TestPyPI before publishing to production PyPI.
+
+**Triggers**:
+- Automatically when a GitHub release is published
+- Manual workflow dispatch with confirmation input
+
+**Usage (Automatic)**:
+1. Create a new release in GitHub
+2. Tag the release (e.g., `v2.1.0`)
+3. Publish the release
+4. The workflow will automatically trigger and publish to TestPyPI
+
+**Usage (Manual)**:
+1. Go to Actions tab in GitHub
+2. Select "Publish to TestPyPI" workflow
+3. Click "Run workflow"
+4. Type `test-publish` in the confirmation field
+5. Click "Run workflow"
+
+**What it does**:
+- Builds the source distribution (sdist)
+- Publishes to TestPyPI for testing
+- Attempts to verify installation from TestPyPI
+
+**Requirements**:
+- Configure TestPyPI trusted publishing OR set `TESTPYPI_API_TOKEN` secret
+- Create `testpypi` environment in repository settings
+
+### 2. publish_pypi.yml - Production PyPI Publishing
+
+**Purpose**: Publish the package to production PyPI after successful testing.
+
+**Trigger**: Manual workflow dispatch only (after confirming TestPyPI works)
+
+**Usage**:
+1. First, verify the package works on TestPyPI (published automatically on release)
+2. Go to Actions tab in GitHub
+3. Select "Publish to PyPI" workflow
+4. Click "Run workflow"
+5. Type `publish` in the confirmation field
+6. Click "Run workflow"
+
+**What it does**:
+- Verifies manual confirmation
+- Builds the source distribution (sdist)
+- Optionally downloads pre-built wheels (if available from build-wheels.yml)
+- Publishes to production PyPI
+- Verifies the publication by installing from PyPI
+
+**Requirements**:
+- Configure PyPI trusted publishing OR set `PYPI_API_TOKEN` secret
+- Create `pypi` environment in repository settings
+
+## Changes Made
+
+### Fixed Issues
+
+1. **Build Artifacts Job**: 
+   - Previous: Only printed placeholder messages
+   - Now: Actually builds and uploads the source distribution
+   - Added GLPK system dependencies installation
+   - Added Python build dependencies installation
+
+2. **Workflow Separation**:
+   - Previous: Single workflow tried to handle both TestPyPI and PyPI
+   - Now: Separate workflows for testing and production
+   - `test_publish.yml` for TestPyPI testing
+   - `publish_pypi.yml` for production PyPI
+
+3. **Artifact Handling**:
+   - Fixed artifact download to handle missing wheels gracefully
+   - Source distribution is always built and uploaded
+   - Wheels are optional (can be built separately via build-wheels.yml)
+
+### Workflow Structure
+
+Both workflows follow similar patterns:
+
+```
+verify → build_artifacts → [download_artifacts] → publish → verify_publication
+```
+
+**verify**: Checks trigger conditions and confirmations
+**build_artifacts**: Builds the source distribution
+**download_artifacts**: (PyPI only) Consolidates artifacts
+**publish**: Uploads to PyPI/TestPyPI
+**verify_publication**: Tests installation from the repository
+
+## PyPI Trusted Publishing
+
+Trusted publishing is the recommended authentication method for GitHub Actions.
+
+### Setup for PyPI:
+
+1. Go to https://pypi.org/manage/account/publishing/
+2. Add a new publisher:
+   - PyPI Project Name: `benpy`
+   - Owner: `markobud`
+   - Repository: `benpy`
+   - Workflow: `publish_pypi.yml`
+   - Environment: `pypi`
+
+### Setup for TestPyPI:
+
+1. Go to https://test.pypi.org/manage/account/publishing/
+2. Add a new publisher:
+   - PyPI Project Name: `benpy`
+   - Owner: `markobud`
+   - Repository: `benpy`
+   - Workflow: `test_publish.yml`
+   - Environment: `testpypi`
+
+## Alternative: API Token Authentication
+
+If you prefer API tokens instead of trusted publishing:
+
+1. Generate an API token from PyPI/TestPyPI
+2. Add it as a repository secret:
+   - `PYPI_API_TOKEN` for PyPI
+   - `TESTPYPI_API_TOKEN` for TestPyPI
+3. Uncomment the `password:` line in the workflow files
+
+## Environments
+
+Both workflows use GitHub Environments for additional protection:
+
+- `pypi` environment for production publishing
+- `testpypi` environment for test publishing
+
+### Creating Environments:
+
+1. Go to repository Settings → Environments
+2. Create two environments: `pypi` and `testpypi`
+3. Optionally add protection rules (e.g., required reviewers)
+
+## Testing the Workflows
+
+### Recommended Release Process:
+
+1. **Create a GitHub release** (e.g., tag `v2.1.0`)
+   - This automatically triggers `test_publish.yml`
+   - Package is published to TestPyPI
+
+2. **Test the TestPyPI package**:
+   ```bash
+   pip install --index-url https://test.pypi.org/simple/ benpy
+   ```
+   Note: Dependencies might not be available on TestPyPI, which is expected.
+
+3. **If testing is successful, publish to PyPI**:
+   - Go to Actions → "Publish to PyPI"
+   - Click "Run workflow"
+   - Type `publish` and run
+
+### Manual TestPyPI Publishing:
+
+If you need to test without creating a release:
+
+```bash
+# Trigger the test workflow manually
+gh workflow run test_publish.yml -f confirm=test-publish
+
+# Check the workflow run
+gh run list --workflow=test_publish.yml
+```
+
+## Building with Wheels
+
+For a complete release including wheels:
+
+1. First run the `build-wheels.yml` workflow to build platform-specific wheels
+2. Then run the `publish_pypi.yml` workflow
+3. The publish workflow will download wheels if available from the build-wheels run
+
+Alternatively, publish just the source distribution and let users build from source.
+
+## Troubleshooting
+
+### "Artifact not found" Error
+
+This was the original issue. It occurred because:
+- The build_artifacts job didn't actually build anything
+- Solution: We now build and upload the sdist in the build_artifacts job
+
+### Verification Failures
+
+If installation verification fails:
+- Check if the package was actually uploaded to PyPI
+- Wait longer (PyPI can take time to index new packages)
+- Check for dependency issues
+
+### Permission Denied
+
+- Ensure trusted publishing is configured correctly
+- Or ensure API tokens are set as secrets
+- Check environment protection rules
+
+## File Locations
+
+- Test workflow: `.github/workflows/test_publish.yml`
+- Production workflow: `.github/workflows/publish_pypi.yml`
+- Wheel building: `.github/workflows/build-wheels.yml`
+
+## Related Documentation
+
+- [GitHub Actions Documentation](https://docs.github.com/en/actions)
+- [PyPI Publishing Guide](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/)
+- [Trusted Publishing](https://docs.pypi.org/trusted-publishers/)

.github/workflows/publish.yml → .github/workflows/publish_pypi.yml

@@ -1,15 +1,18 @@
 name: Publish to PyPI
 
-# This is a DRAFT workflow for publishing to PyPI
-# Before using in production:
-# 1. Configure PyPI trusted publishing in your PyPI project settings
-# 2. Set up the PYPI_API_TOKEN secret if not using trusted publishing
-# 3. Remove the manual approval requirement if desired
-# 4. Test with TestPyPI first
+# Workflow for publishing to production PyPI
+# This workflow:
+# 1. Manually triggered only (after testing with TestPyPI)
+# 2. Builds the source distribution (sdist)
+# 3. Optionally downloads wheels from a previous build-wheels workflow run
+# 4. Publishes to production PyPI
+#
+# Note: For trusted publishing, configure it in PyPI project settings
+# Otherwise, set the PYPI_API_TOKEN secret
+# 
+# For testing, use test_publish.yml which publishes to TestPyPI automatically on release
 
 on:
-  release:
-    types: [published]
   workflow_dispatch:
     inputs:
       confirm:
@@ -32,11 +35,9 @@ jobs:
       - name: Check release conditions
         id: check
         run: |
-          if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
-            if [[ "${{ github.event.inputs.confirm }}" != "publish" ]]; then
-              echo "Manual trigger requires 'publish' confirmation"
-              exit 1
-            fi
+          if [[ "${{ github.event.inputs.confirm }}" != "publish" ]]; then
+            echo "Manual trigger requires 'publish' confirmation"
+            exit 1
           fi
           echo "should_publish=true" >> $GITHUB_OUTPUT
 
@@ -49,23 +50,44 @@ jobs:
       - name: Checkout code
         uses: actions/checkout@v4
 
-      - name: Trigger wheel build workflow
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update -qq
+          sudo apt-get install -y libglpk-dev glpk-utils
+
+      - name: Install build dependencies
         run: |
-          echo "In production, this would trigger the build-wheels.yml workflow"
-          echo "For now, wheels should be built separately before publishing"
+          python -m pip install --upgrade pip
+          pip install build wheel setuptools Cython numpy
+
+      - name: Build source distribution
+        run: python -m build --sdist
+
+      - name: Upload source distribution
+        uses: actions/upload-artifact@v4
+        with:
+          name: sdist
+          path: dist/*.tar.gz
+          retention-days: 90
 
   download_artifacts:
     name: Download build artifacts
     needs: [build_artifacts]
     runs-on: ubuntu-latest
 
     steps:
-      - name: Download wheels
+      - name: Download wheels (if available)
         uses: actions/download-artifact@v4
         with:
           pattern: wheels-*
           path: dist
           merge-multiple: true
+        continue-on-error: true  # Wheels may not exist in this run
 
       - name: Download source distribution
         uses: actions/download-artifact@v4
@@ -75,8 +97,13 @@ jobs:
 
       - name: List artifacts
         run: |
-          ls -lh dist/
-          echo "Total artifacts: $(ls dist/ | wc -l)"
+          ls -lh dist/ || echo "No artifacts found"
+          if [ -d "dist" ]; then
+            echo "Total artifacts: $(ls dist/ | wc -l)"
+          else
+            echo "No dist directory found"
+            exit 1
+          fi
 
       - name: Upload combined artifacts
         uses: actions/upload-artifact@v4
@@ -85,32 +112,9 @@ jobs:
           path: dist/*
           retention-days: 90
 
-  publish_testpypi:
-    name: Publish to TestPyPI (Optional)
-    needs: [download_artifacts]
-    runs-on: ubuntu-latest
-    environment:
-      name: testpypi
-      url: https://test.pypi.org/p/benpy
-
-    steps:
-      - name: Download all artifacts
-        uses: actions/download-artifact@v4
-        with:
-          name: all-dist-artifacts
-          path: dist
-
-      - name: Publish to TestPyPI
-        uses: pypa/gh-action-pypi-publish@release/v1
-        with:
-          repository-url: https://test.pypi.org/legacy/
-          skip-existing: true
-          verbose: true
-        continue-on-error: true  # Don't fail if TestPyPI upload fails
-
   publish_pypi:
     name: Publish to PyPI
-    needs: [download_artifacts, publish_testpypi]
+    needs: [download_artifacts]
     runs-on: ubuntu-latest
     environment:
       name: pypi
@@ -128,13 +132,14 @@ jobs:
           echo "Verifying artifacts before publication..."
           ls -lh dist/
 
-          # Check we have wheels and sdist
-          if [ ! -f dist/*.tar.gz ]; then
+          # Check we have source distribution
+          if ! ls dist/*.tar.gz >/dev/null 2>&1; then
             echo "ERROR: No source distribution found"
             exit 1
           fi
 
-          if [ -z "$(ls dist/*.whl 2>/dev/null)" ]; then
+          # Check for wheels (optional)
+          if ! ls dist/*.whl >/dev/null 2>&1; then
             echo "WARNING: No wheels found, publishing source only"
           fi