Resolve package spec directories dynamically instead of hardcoded maps

[Odilhao]

Summary

find_package.py used hand-maintained package_mappings/reverse_mappings/lowercase_packages
tables to guess a PyPI project name's on-disk spec directory. These tables silently break
whenever an entry is missing or goes stale:

• aiohttp-socks reverse-mapped to a nonexistent aiohttp_socks (underscore) directory, so it
  was silently dropped from every automated update run. This let python-socks get bumped to
  3.0.0 while aiohttp-socks stayed on 0.10.1 (which pins python-socks<3.0.0), breaking
  repoclosure on the python-socks 3.0.0 bump PR.
• galaxy-importer and importlib-resources reverse-mapped to stale underscored directory
  names after those directories were renamed to hyphenated form; same silent failure.
• The forward-transformed name written to packages-to-update.txt didn't always match the
  on-disk directory suffix used by update_packages.sh's own (unmapped)
  packages/python-$pkg/python-$pkg.spec template — e.g. a poetry-core bump would fail at the
  actual spec-bump step even though discovery succeeded.

Change

Replaced all of this with a single directory index built from the real packages/python-*/ tree,
matched via PEP 503-style canonicalization (lowercase, collapse -/_/. runs into a single
separator). This resolves correctly for all 272 packages currently in the repo with no
hand-maintained table, and self-adapts as packages are added, renamed, or removed — no more
manual mapping entries needed going forward.

Rewrote the test suite to validate resolution against the real packages/ tree (which catches
the three bugs above) instead of only testing the old cosmetic name transform in isolation.

Test plan

• pytest test/ — 21 tests pass
• Verified all 272 current packages/python-* directories resolve correctly via a full sweep
• Verified regression cases: aiohttp-socks, galaxy-importer, importlib-resources,
  poetry-core, psycopg-c, opentelemetry-*, ruamel.yaml*, jaraco.*, et_xmlfile,
  pyasn1_modules, python-socks/python-dateutil/python-debian/python-gnupg (own
  python- prefix), mixed-case names (PyYAML, GitPython)
• CI (Test find_package.py workflow) — pending

[zjhuntin]

Excellent refactoring that replaces fragile hardcoded mappings with robust dynamic directory resolution.

Key improvements:

• Fixes real bugs (aiohttp-socks, galaxy-importer, importlib-resources)
• Eliminates maintenance burden of hardcoded mapping tables
• Self-adapting as packages are added/renamed/removed
• Comprehensive test coverage (21 tests, validates against real packages/ tree)
• Correct use of PEP 503 canonicalization

Minor optimization suggestion:
Consider building the directory index once in main() rather than rebuilding on every resolve_package_dir() call when processing large requirements files. Not a blocker, but would eliminate redundant filesystem globs.

Code quality is excellent - clean, well-tested, well-documented. Ready to merge.

[Odilhao]

Thanks for the review! Addressed the index-rebuild suggestion in fa2928a — build_directory_index() is now called once in main()/build_package_list() and threaded through, instead of re-globbing on every resolve_package_dir() call. CI is green.

[ogajduse]

Nice refactor — I verified the core claim independently: all 272 current packages/python-* dirs resolve correctly, the dir-suffix output matches what update_packages.sh consumes, and the old tables really did carry 9 broken entries this fixes. Also confirmed fa2928a resolves the per-line index rebuild. A few things I'd still flag, roughly in priority order:

Two prefix-handling gaps I'd fix before merge (both one-liners):

1. strip_python_prefix() runs before canonicalization and only matches the literal python- spelling, but pip freeze emits the dist METADATA name, which can be underscore-spelled (this very input stream has carried aiohttp_socks and et_xmlfile). If python-socks ever freezes as python_socks, the prefix isn't stripped, canonicalization yields python-socks, the index key is socks, resolution returns None, and the package is silently dropped — the exact bug class this PR exists to kill. Fix: strip after canonicalizing, i.e. strip_python_prefix(canonicalize(pkg)).
2. resolve_package_dir() strips python- unconditionally with no exact-match-first fallback, so distinct PyPI projects foo and python-foo collapse onto one directory — e.g. a freeze line for PyPI's gnupg would resolve to packages/python-gnupg (which packages the different project python-gnupg) and write a wrong-project bump. Trying directory_index.get(canonicalize(pkg)) before the stripped form fixes it. (Carried over from the old prefix_removals semantics, not a regression — but it's a one-line guard now.)

Silent-failure hardening (the failure mode in all of these is a green no-op run):

3. An empty index (script run from the wrong cwd, since PACKAGES_DIR is cwd-relative) is indistinguishable from "nothing packaged": every package prints "Spec file not found" and the script exits 0. A cheap if not directory_index: sys.exit(...) guard makes it loud.
4. The dir-suffix contract with update_packages.sh lives only in a comment. If the suffix mismatches at bump time, rpmspec fails, rpm_version is empty, rpmdev-vercmp matches none of its branches, and the set +e script still exits 0 — bump job green, nothing bumped. Emitting the spec path (or exposing resolve_package_dir as a CLI entry the shell script calls) would enforce it.
5. build_directory_index() silently takes last-glob-wins on key collisions. No collisions exist today, but this repo has renamed dirs between separator styles before (galaxy_importer, importlib_resources); a transition leaving a stale duplicate makes the winner filesystem-order-dependent. A duplicate-key raise/warn would catch it.
6. name, version = line.split("==") aborts the whole run with a traceback on any non-name==version freeze line (URL/VCS/editable deps emit name @ url / -e git+…). Pre-existing, but this rewrite was the moment to skip-and-warn instead.

Cleanups (take or leave):

7. build_package_list() has no caller anywhere in the repo (verified repo-wide, before and after this PR) — and fa2928a even threaded the new index through it. I'd delete it rather than maintain it.
8. parse_package_list() is now character-identical to the one in build_matrix.py — a format change to the freeze stream has to be fixed in two-going-on-three places. test/conftest.py already puts the repo root on sys.path, so sharing one is cheap.
9. The integration test hardcodes on-disk suffixes for 40 packages, but neither test workflow triggers on packages/** — so a rename-only PR skips the test and breaks CI for the next unrelated PR instead. Add packages/** to the path filters, or assert resolvability rather than exact spelling.

[Odilhao]

Thanks for the thorough review — genuinely caught two real bugs. Addressed in 46072fb:

Fixed:

1. Prefix-order bug: canonicalize now runs before python- stripping, so python_socks (underscore) resolves correctly, not just the hyphen spelling.
2. Exact-match-first: resolve_package_dir() now tries an exact canonical match before falling back to stripping python-, so distinct projects like gnupg/python-gnupg don't collapse onto each other.
3. build_directory_index() now raises on canonical-key collisions instead of silent last-glob-wins.
4. main() exits loudly on an empty directory index instead of silently no-op'ing every package.
5. parse_package_list() skips and warns on unparseable lines (VCS/URL/editable deps) instead of crashing the whole run.
6. Removed the dead build_package_list().
7. build_matrix.py now imports parse_package_list from find_package.py instead of duplicating it.

Deferred (flagged, not fixed here):

• Item 4 (enforcing the dir-suffix contract at the update_packages.sh bump step) — that's a different script/failure surface; would rather scope it as its own PR.
• Item 9 (adding packages/** to the test workflow path filters) — agreed this is a real gap, but I don't have the workflow OAuth scope to push .github/workflows/*.yml changes from this session. Could someone with that scope pick it up, or should I open a tracking issue?
• Also worth flagging: update-pulp-packages.yml (the dependabot-merge bump path) calls build_matrix.py directly on a raw requirements.txt diff, bypassing find_package.py's resolution entirely — same bug class could exist there under a different trigger. Separate issue, happy to file it if useful.

CI is green (27 tests now, up from 21).

[ogajduse]

ACK — thanks for the quick turnaround.

Verified 46072fb against each point of the review: all seven fixes check out in the code, each one carries a named regression test, and the suite passes locally for me as well (27 tests). The exact-match-first ordering and the canonicalize-before-strip change both resolve the cases from the review (python_socks, gnupg vs python-gnupg), and I confirmed the new skip-warning on stdout is harmless since build_matrix.py writes its matrix to $GITHUB_OUTPUT, not stdout.

The deferrals make sense to me. On your question: yes, please open a tracking issue — one issue covering all three follow-ups is fine (the update_packages.sh suffix-contract enforcement, the packages/** workflow path filters, and the update-pulp-packages.yml raw-diff bypass), so they don't get lost after this merges. The path-filter change needs someone with the workflow scope anyway, so having it written down is the main thing.

[Odilhao]

Opened #2988 tracking all three follow-ups (update_packages.sh suffix-contract enforcement, packages/** workflow path filters, update-pulp-packages.yml raw-diff bypass).