Skip to content

Add file-io 0.2.0 to git-annex build environment - #295

Merged
yarikoptic merged 2 commits into
masterfrom
claude/kind-curie-a9wb4m
Sep 26, 2026
Merged

yarikoptic merged 2 commits into
masterfrom
claude/kind-curie-a9wb4m

Conversation

@yarikoptic-gitmate

Copy link
Copy Markdown
Collaborator

Summary

This change adds the file-io Haskell package (version 0.2.0) to the git-annex build environment Docker image to enable the OsPath build flag in git-annex.

Key Changes

  • Added installation of file-io 0.2.0 from Hackage into the global GHC package database
  • Included SHA256 verification of the downloaded tarball for security
  • Configured and built file-io using Cabal's Setup.hs mechanism with global installation

Implementation Details

The file-io package is required because:

  • git-annex's OsPath build flag (enabled by default) depends on file-io >= 0.2.0 since version 10.20260213
  • Debian's libghc-file-io-dev package only provides version 0.1.5, which is insufficient
  • OsPath enables important fixes for file handling on systems like BeeGFS

The installation:

  • Downloads the package from Hackage with checksum verification
  • Builds and installs it globally so it's available to subsequent builds
  • Uses only GHC boot packages as dependencies, avoiding additional package requirements
  • Includes a note that this workaround can be removed once Debian provides libghc-file-io-dev >= 0.2.0

https://claude.ai/code/session_013n9Q2igij49kxdtiakUzYv

Our debianstandalone builds report no OsPath in "build flags", e.g.
10.20260901+git60-g7c2e8019c9-1~ndall+1:

  build flags: Assistant Webapp Inotify DBus DesktopNotify TorrentParser
    MagicMime Benchmark Feeds Testsuite S3 WebDAV Servant

The OsPath flag is Default: True but not Manual, so `./Setup configure`
quietly turns it off when a dependency is missing.  In this image
(GHC 9.10.3) filepath 1.5.4.0, os-string 2.0.7 and directory 1.3.8.5
already satisfy its bounds; only file-io is absent.  Since 10.20260213
git-annex requires file-io >= 0.2.0, while Debian ships only
libghc-file-io-dev 0.1.5, so `apt-get build-dep` cannot provide it.

Build file-io 0.2.0 from Hackage (checksum-pinned) into the global GHC
package db.  It only depends on GHC boot packages.  Verified in the
current image: `./Setup configure -v2` of git-annex 10.20260901 goes
from "ospath=False" to "Dependency file-io >=0.2.0: using file-io-0.2.0".

Without OsPath, git-annex keeps known issues fixed only in OsPath
builds, e.g. https://git-annex.branchable.com/bugs/35_failed_tests_on_beegfs/
and fd leaks to child processes (per upstream CHANGELOG).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013n9Q2igij49kxdtiakUzYv

Copy link
Copy Markdown
Member

The build check (Build Linux buildenv image, run 36139747634) was cancelled at the 6h job timeout. That is not caused by this PR's change: the job never reached the new file-io step. It hung in the first, pre-existing RUN layer, at nd_freeze 20260425 → INFO: Refreshing apt cache. That is apt update against snapshot.debian.org, and the log has no output after 13:15:50 until the cancel at 19:15:18. Previous runs of this workflow took about 2.5h, most of it downloading from snapshot.debian.org.

No fix exists for that yet. Hardening it (e.g. Acquire::http::Timeout next to the existing Dl-Limit/Retries settings, or timeout-minutes on the job) would be a separate change, so I have not widened this PR with it. I'm re-running the job once.

The new step itself was verified locally, layered on the current datalad/buildenv-git-annex:latest:

  • ghc-pkg field file-io version gives version: 0.2.0.
  • A full make debianstandalone-dsc plus dpkg-buildpackage -b of upstream 10.20260901+git60 reports build flags: ... Servant OsPath.
  • git annex test on that build: all 26 test groups pass.

Generated by Claude Code

The buildenv image build for #295 hung for the full 6h job limit
right after nd_freeze printed "INFO: Refreshing apt cache", with no
further output.  After that first `apt-get update`, nd_freeze scrapes
snapshot.debian.org via get_www_content(): a raw-socket HTTP/1.1 GET
without "Connection: close" and with no timeout, reading until EOF,
and it logs nothing there unless --debug.  A stalled connection there
hangs forever silently.

Write deb822 sources for snapshot.debian.org ourselves instead,
as of DEBIAN_SNAPSHOT (an ARG, now 20260924T000000Z).  snapshot.debian.org
serves the latest snapshot at or before the given timestamp.  This also
drops what only existed to work around nd_freeze: the neurodebian-freeze
install from deb.debian.org, disabling debian.sources, and the sed adding
deb-src lines (the deb822 stanzas carry "Types: deb deb-src").  Add
Acquire::http::Timeout so a stalled transfer fails and gets retried
(Acquire::Retries 10) instead of hanging.

build-linux-buildenv.yaml derived the image tag's date by grepping for
nd_freeze; it now reads DEBIAN_SNAPSHOT (YYYYMMDD part), so tags keep
the forky-YYYYMMDD-<commit> shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013n9Q2igij49kxdtiakUzYv

Copy link
Copy Markdown
Member

The re-run hung at the same point: nd_freeze 20260425 → INFO: Refreshing apt cache, then no output from 19:17:20 until the 6h cancel at 01:16:35. Two identical 6h hangs means this is not a flake. So I've pushed 7942b06, which replaces nd_freeze with apt sources pinned to snapshot.debian.org directly.

Most likely cause. After its first apt-get update, nd_freeze's write_snapshot_sources scrapes snapshot.debian.org with get_www_content. That is a raw-socket HTTP/1.1 GET with no Connection: close and no timeout, which reads until EOF. It logs nothing unless --debug is given, so a stalled connection there looks exactly like these logs.

Change:

  • ARG DEBIAN_SNAPSHOT=20260924T000000Z feeds a deb822 debian-snapshot.sources for forky / forky-updates / forky-security, with Types: deb deb-src (needed by build-dep).
  • Acquire::Check-Valid-Until "false", since snapshot Release files are past their Valid-Until.
  • Acquire::http::Timeout "120", so a stalled transfer fails into the existing Retries 10 instead of hanging.
  • Dropped the neurodebian-freeze install, the debian.sources → .disabled dance and the deb-src sed, which only existed for nd_freeze.
  • build-linux-buildenv.yaml derived the tag date by grepping for nd_freeze. It now reads DEBIAN_SNAPSHOT, so tags stay forky-YYYYMMDD-<commit> (here forky-20260924-…).

Note: the snapshot date moves from 2026-04-25 to 2026-09-24, so the GHC / libghc-* set may change. Once this image builds, OsPath needs re-checking in a standalone build. The file-io step becomes redundant, but harmless, if Debian has meanwhile shipped file-io ≥ 0.2.0.

Tested locally only up to apt: the ARG expands, apt accepts the sources, and apt-get update --print-uris lists the expected InRelease, Packages.xz and Sources.xz URIs. snapshot.debian.org itself is not reachable from where I tested.


Generated by Claude Code

@yarikoptic
yarikoptic merged commit a323fe6 into master Sep 26, 2026
3 checks passed
@yarikoptic
yarikoptic deleted the claude/kind-curie-a9wb4m branch September 26, 2026 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants