diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 832a913a..77186986 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -2,12 +2,12 @@ ## Project Overview -**go-pugleaf** is a modern NNTP server and web gateway for Usenet/NetNews built in Go. It provides a complete newsgroup platform with full RFC 3977 compliant NNTP server implementation, modern web interface for browsing, efficient article fetching and threading, SQLite-based storage with per-group databases, and spam flagging/moderation tools. +**go-pugleaf** is a modern NNTP server and web gateway for Usenet/NetNews built in Go. It provides a complete newsgroup platform with full RFC 3977 compliant NNTP server implementation, modern web interface for browsing, efficient article fetching and threading, SQLite-based storage with per-group databases, spam flagging/moderation tools, and bridge integrations for Fediverse (ActivityPub) and Matrix protocols. ### High-Level Repository Information - **Type**: Production newsgroup server system (medium-to-large codebase) -- **Size**: 150 Go files across 13 internal packages, 21 command applications -- **Languages**: Go 1.24.3+ (primary), HTML/CSS/JavaScript (web frontend), Shell scripts (build system) +- **Size**: 149 Go files across 13 internal packages, 21 command applications +- **Languages**: Go 1.25.0+ (primary), HTML/CSS/JavaScript (web frontend), Shell scripts (build system) - **Frameworks**: Gin web framework, custom NNTP protocol implementation, SQLite3 database - **Target Runtime**: Linux/Unix systems (Windows support not tested) - **Dependencies**: Minimal external dependencies - SQLite3, Gin, Go crypto libraries @@ -16,8 +16,8 @@ ### Essential Prerequisites ```bash -# Always ensure Go 1.24.3+ is available -go version # Must be 1.24.3 or higher +# Always ensure Go 1.25.0+ is available +go version # Must be 1.25.0 or higher # Always run module commands before building go mod tidy && go mod verify @@ -25,14 +25,29 @@ go mod tidy && go mod verify ### Build Commands (Always run in repository root) ```bash -# Build all applications (14+ binaries) - takes ~15-30 seconds +# Build all applications (13 binaries) - takes ~15-30 seconds ./build_ALL.sh # Build individual applications - takes ~1-2 seconds each +# 15 individual build scripts available: ./build_webserver.sh # Main web interface ./build_fetcher.sh # Article fetcher ./build_nntp-server.sh # NNTP server ./build_expire-news.sh # Article expiration tool +./build_merge-active.sh # Merge active files utility +./build_merge-descriptions.sh # Merge newsgroup descriptions +./build_nntpmgr.sh # NNTP management tool +./build_analyze.sh # NNTP analysis tool +./build_history-rebuild.sh # History rebuild utility +./build_fix-references.sh # Fix article references +./build_fix-thread-activity.sh # Fix thread activity +./build_import_flat-files.sh # Import flat files +./build_recover-db.sh # Database recovery +./build_rslight_importer.sh # RSLight importer +./build_TestMsgIdItemCache.sh # Message ID cache tester + +# Note: Some cmd applications don't have build scripts: +# benchmark_hash, extract_hierarchies, history-demo, parsedates, test-nntp, usermgr # Build output goes to build/ directory ls -la build/ # List all built binaries @@ -96,7 +111,19 @@ golangci-lint run ├── nntp-fetcher/ # Article fetching from NNTP providers ├── nntp-server/ # NNTP server implementation ├── expire-news/ # Article expiration and cleanup - └── [17 other tools] # Import, analysis, migration utilities + ├── merge-active/ # Merge active files utility + ├── merge-descriptions/ # Merge newsgroup descriptions utility + ├── nntpmgr/ # NNTP management tool + ├── nntp-analyze/ # NNTP analysis tool + ├── history-rebuild/ # History rebuild utility + ├── fix-references/ # Fix article references + ├── fix-thread-activity/ # Fix thread activity + ├── import-flat-files/ # Import flat files + ├── recover-db/ # Database recovery + ├── rslight-importer/ # RSLight importer + ├── test-MsgIdItemCache/ # Message ID cache tester + └── [6 other tools] # benchmark_hash, extract_hierarchies, history-demo, + # parsedates, test-nntp, usermgr (no build scripts) /internal/ # 13 internal packages (core business logic) ├── nntp/ # NNTP protocol implementation (60+ files) @@ -104,7 +131,14 @@ golangci-lint run ├── database/ # SQLite abstraction and per-group databases ├── config/ # Configuration management ├── models/ # Data models and structures - └── [8 other packages] # cache, embedded, history, processor, etc. + ├── cache/ # Caching layer + ├── processor/ # Article processing + ├── history/ # History tracking + ├── utils/ # Utility functions + ├── preloader/ # Data preloading + ├── spam/ # Spam detection and filtering + ├── fediverse/ # Fediverse integration + └── matrix/ # Matrix protocol support /web/ # Frontend assets ├── templates/ # HTML templates (Gin templating) @@ -141,6 +175,11 @@ golangci-lint run - Overview data caching for performance - Migration system for schema updates +**Bridge Integrations** (New in v0.4.8.6+): +- **Fediverse Bridge** (`internal/fediverse/`): ActivityPub protocol support for bridging NNTP to Mastodon/Pleroma +- **Matrix Bridge** (`internal/matrix/`): Matrix protocol support for bridging newsgroups to Matrix rooms +- **Spam Filtering** (`internal/spam/`): SpamAssassin integration with fast rule-based filtering + ### Dependencies Not Obvious from Layout - **Race Detection**: All builds use `-race` flag for concurrency safety - **Version Injection**: Build scripts inject version from `appVersion.txt` @@ -154,7 +193,9 @@ golangci-lint run **Database Schema**: Add migrations to `internal/database/migrations/` **Configuration**: Update `internal/config/` and `configs/sample.yaml` **Build Process**: Modify individual `build_*.sh` scripts -**New Applications**: Add to `cmd/` directory with corresponding build script +**New Applications**: Add to `cmd/` directory with corresponding build script (build_.sh) +**Bridge Configuration**: Update `internal/fediverse/`, `internal/matrix/`, or `internal/spam/` for integration features +**Spam Rules**: Update SpamAssassin configuration or quick rules in `internal/spam/` ### Key Files by Importance 1. `cmd/web/main.go` - Main web application entry point @@ -163,6 +204,9 @@ golangci-lint run 4. `internal/database/database.go` - Database abstraction layer 5. `build_ALL.sh` - Master build script for all applications 6. `go.mod` - Dependencies and Go version requirements +7. `internal/fediverse/bridge.go` - Fediverse integration (ActivityPub) +8. `internal/matrix/bridge.go` - Matrix room bridging +9. `internal/spam/filter.go` - Spam filtering system ### Validation and CI/CD Currently **no GitHub Actions or automated CI/CD**. Validation is manual: @@ -179,7 +223,7 @@ Currently **no GitHub Actions or automated CI/CD**. Validation is manual: - **Per-Group Databases**: Articles stored in separate SQLite files per newsgroup ### Troubleshooting Common Issues -- **Build Failures**: Check Go version (must be 1.24.3+), run `go mod tidy` +- **Build Failures**: Check Go version (must be 1.25.0+), run `go mod tidy` - **Missing Templates**: Ensure `web/` directory present for web server - **Memory Issues**: Known issue with long-running fetcher and webserver processes - **NNTP Testing**: Use telnet to localhost:1119 for manual NNTP server testing diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 6a312c6c..f631797b 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -34,14 +34,6 @@ jobs: with: go-version: '1.25.x' - - name: Cache Go modules - uses: actions/cache@v4 - with: - path: ~/go/pkg/mod - key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }} - restore-keys: | - ${{ runner.os }}-go- - - name: Install dependencies run: go mod download @@ -117,12 +109,33 @@ jobs: fi done + - name: Generate SHA256 checksums + run: | + OS="${{ matrix.os }}" + ARCH="${{ matrix.arch }}" + CHECKSUMS_FILE="checksums-${OS}-${ARCH}.sha256" + + echo "Generating SHA256 checksums for ${OS}-${ARCH} binaries..." + cd build + sha256sum * > "../${CHECKSUMS_FILE}" + cd .. + + echo "CHECKSUMS_FILE=${CHECKSUMS_FILE}" >> $GITHUB_ENV + echo "" + echo "Generated checksums for ${OS}-${ARCH}:" + echo "=================================" + cat "${CHECKSUMS_FILE}" + - name: Create release archive run: | VERSION="${{ steps.version.outputs.version }}" OS="${{ matrix.os }}" ARCH="${{ matrix.arch }}" ARCHIVE_NAME="go-pugleaf-v${VERSION}-${OS}-${ARCH}" + + # Include checksums in the archive + cp "${{ env.CHECKSUMS_FILE }}" build/checksums.sha256 + if [ "${{ matrix.os }}" = "windows" ]; then zip -r "${ARCHIVE_NAME}.zip" build/ README.md LICENSE echo "ARCHIVE_FILE=${ARCHIVE_NAME}.zip" >> $GITHUB_ENV @@ -135,7 +148,9 @@ jobs: uses: actions/upload-artifact@v4 with: name: go-pugleaf-${{ matrix.os }}-${{ matrix.arch }} - path: ${{ env.ARCHIVE_FILE }} + path: | + ${{ env.ARCHIVE_FILE }} + ${{ env.CHECKSUMS_FILE }} release: name: Create release @@ -163,6 +178,38 @@ jobs: with: path: artifacts + - name: Create comprehensive checksums file + run: | + echo "Creating comprehensive SHA256 checksums file..." + echo "# Go-Pugleaf v${{ steps.version.outputs.version }} - SHA256 Checksums" > SHA256SUMS.txt + echo "# Generated on $(date -u '+%Y-%m-%d %H:%M:%S UTC')" >> SHA256SUMS.txt + echo "" >> SHA256SUMS.txt + + # Process each platform's checksums + for artifact_dir in artifacts/go-pugleaf-*; do + if [ -d "$artifact_dir" ]; then + platform=$(basename "$artifact_dir" | sed 's/go-pugleaf-//') + echo "## Platform: $platform" >> SHA256SUMS.txt + + # Find and process checksums file for this platform + checksums_file=$(find "$artifact_dir" -name "checksums-*.sha256" | head -1) + if [ -f "$checksums_file" ]; then + echo "Processing checksums for $platform..." + while IFS= read -r line; do + # Extract hash and filename + hash=$(echo "$line" | cut -d' ' -f1) + filename=$(echo "$line" | cut -d' ' -f2-) + echo "$hash $filename ($platform)" >> SHA256SUMS.txt + done < "$checksums_file" + fi + echo "" >> SHA256SUMS.txt + fi + done + + echo "Comprehensive checksums file created:" + echo "=====================================" + cat SHA256SUMS.txt + - name: Create release uses: softprops/action-gh-release@v2 with: @@ -193,8 +240,27 @@ jobs: - macOS (amd64, arm64) - Windows (amd64) + ### Binary Verification: + All binaries include SHA256 checksums for integrity verification: + - **SHA256SUMS.txt** - Comprehensive checksums for all platforms + - **checksums-{os}-{arch}.sha256** - Individual platform checksums + - Each archive also contains a `checksums.sha256` file for verification after extraction + + **Verification Examples:** + ```bash + # Verify individual platform binaries + sha256sum -c checksums-linux-amd64.sha256 + + # Verify after archive extraction + tar -xzf go-pugleaf-v${{ steps.version.outputs.version }}-linux-amd64.tar.gz + cd build/ + sha256sum -c checksums.sha256 + ``` + Extract the archive for your platform and run the binaries. See README.md for usage instructions. - files: artifacts/*/* + files: | + artifacts/*/* + SHA256SUMS.txt draft: false prerelease: false \ No newline at end of file diff --git a/.gitignore b/.gitignore index ac547f57..9336fbf5 100644 --- a/.gitignore +++ b/.gitignore @@ -29,6 +29,8 @@ go.work /demo_history internal/nntp/analysis_* update.tar.gz +checksums.sha256 +checksums.sha256.archive active_files/local-mode.active.filtered active_files/local-mode.active.filtered.sorted active_files/local-mode.active.new diff --git a/.update b/.update index 898c6c04..77dbdb7a 100644 --- a/.update +++ b/.update @@ -1 +1 @@ -2cc6dbd00dc59836970e79b6ed101020c952164e222a52c6a0694294d8da5bfb update.tar.gz +3e20a56cf38c0b7be78fb3791dacccf4a9b59f6127c55bc6fb1ca9999d266958 update.tar.gz diff --git a/README.md b/README.md index 1ea4aaeb..55a455c9 100644 --- a/README.md +++ b/README.md @@ -180,6 +180,43 @@ Or build a single tool manually, e.g.: go build -o build/usermgr ./cmd/usermgr ``` +### Building and Release + +**Build all binaries:** +```bash +# Build all binaries (automatically generates checksums) +./build_ALL.sh +``` + +**Generate checksums manually:** +```bash +# Generate SHA256 checksums for all executables in build/ +./createChecksums.sh +``` + +**Build and create release package:** +```bash +# Build all binaries and create release package with checksums +./build_ALL.sh update +``` + +This creates: +- `checksums.sha256` - SHA256 hashes for all individual executables (with build/ paths) +- `checksums.sha256.archive` - SHA256 hashes with relative paths for archive inclusion +- `update.tar.gz` - Compressed archive of all binaries including checksums.sha256 +- `.update` - SHA256 hash of the tar.gz file + +**Verify checksums:** +```bash +# Verify all executable checksums (from repository root) +sha256sum -c checksums.sha256 + +# Verify checksums after extracting release archive +tar -xzf update.tar.gz +cd extracted-directory/ +sha256sum -c checksums.sha256 # Verify all executables in release +``` + ## 📚 Binary Documentation go-pugleaf includes command-line applications for various newsgroup management tasks. diff --git a/build_ALL.sh b/build_ALL.sh index 3f3e6a66..c7a2e911 100755 --- a/build_ALL.sh +++ b/build_ALL.sh @@ -17,4 +17,8 @@ rm -v build/* ./build_recover-db.sh ./build_expire-news.sh +# Always generate checksums after building +echo "Generating checksums for built executables..." +./createChecksums.sh + test "$1" = "update" && ./createUpdate.sh diff --git a/createChecksums.sh b/createChecksums.sh new file mode 100755 index 00000000..559e5057 --- /dev/null +++ b/createChecksums.sh @@ -0,0 +1,45 @@ +#!/bin/sh -e + +# Generate SHA256 checksums for all executables in build directory + +BUILD_DIR="build" +CHECKSUMS_FILE="checksums.sha256" + +echo "Generating SHA256 checksums for executables in $BUILD_DIR/" + +# Check if build directory exists +if [ ! -d "$BUILD_DIR" ]; then + echo "Error: Build directory '$BUILD_DIR' not found" + echo "Please run ./build_ALL.sh first to build all executables" + exit 1 +fi + +# Check if there are any files in build directory +if [ ! "$(ls -A $BUILD_DIR)" ]; then + echo "Error: Build directory '$BUILD_DIR' is empty" + echo "Please run ./build_ALL.sh first to build all executables" + exit 1 +fi + +# Remove existing checksums file +rm -f "$CHECKSUMS_FILE" + +# Generate checksums for all files in build directory +echo "Creating $CHECKSUMS_FILE with SHA256 hashes..." +sha256sum "$BUILD_DIR"/* > "$CHECKSUMS_FILE" + +# Also create a version with relative paths for inclusion in the release archive +cd "$BUILD_DIR" +sha256sum * > "../${CHECKSUMS_FILE}.archive" +cd .. + +# Display the checksums file +echo "" +echo "Checksums generated successfully:" +echo "=================================" +cat "$CHECKSUMS_FILE" + +echo "" +echo "Checksums file created: $CHECKSUMS_FILE" +echo "Archive checksums file created: ${CHECKSUMS_FILE}.archive (for inclusion in release)" +echo "Number of executables: $(wc -l < $CHECKSUMS_FILE)" \ No newline at end of file diff --git a/createUpdate.sh b/createUpdate.sh index 89ed0b85..b973c415 100755 --- a/createUpdate.sh +++ b/createUpdate.sh @@ -1,8 +1,20 @@ #!/bin/sh -e +# Generate checksums for individual executables +echo "Generating checksums for individual executables..." +./createChecksums.sh + +# Copy archive checksums file into build directory for inclusion in release archive +echo "Including checksums in release archive..." +cp checksums.sha256.archive build/checksums.sha256 + rm -f update.tar.gz tar -f update.tar.gz -C build -c -v -z . sha256sum update.tar.gz > .update HASH=$(cat .update|cut -d" " -f1) du -b update.tar.gz cat .update +echo "" +echo "Release archive created with individual executable checksums included" +echo "Individual executable checksums available in: checksums.sha256" +echo "Checksums are also included in the release archive as checksums.sha256" exit 0