fix(hts): resolve versionless canonicals to the owning package, not a copy (#200)

crates/hts/docs/ig-publisher-compatibility.md

@@ -123,19 +123,95 @@ with `version: "current"` and `content: not-present` (no concepts).
 
 When `$validate-code` (or backend `validate_code`) receives `version=current`:
 
-1. **`fetch_versions` / Postgres resolve query** orders candidates so that:
+1. **`cs_precedence_order_by`** (`src/backends/mod.rs`) is the single definition of
+   same-URL precedence. Both backends embed its text verbatim, so they cannot drift.
+   It orders candidates so that:
    - `content = complete` or `supplement` ranks before `fragment`, `example`, `not-present`
    - rows **with concepts** rank before empty stubs
+   - rows from the package that **owns** the canonical URL rank before re-published
+     copies (`authority_rank` — see §2a)
    - then highest `version` string, then stable `id`
 2. **`resolve_code_system*`** treats `current` as “pick the first candidate after
    ordering” instead of literal string equality.
 
+The first two tiers are load-bearing for this behaviour and must stay ahead of the
+version tier: the stub's version string is literally `"current"`, and `'c' > '2'`,
+so `"current"` text-sorts **above** a real edition like `"20260501"`. Only the
+content and has-concepts tiers keep the real import winning.
+
 ### Code changes
 
-- `src/backends/mod.rs` — `code_system_version_is_current()`
+- `src/backends/mod.rs` — `code_system_version_is_current()`, `cs_precedence_order_by()`
 - `src/backends/sqlite/code_system.rs` — `fetch_versions`, `resolve_code_system_uncached`
 - `src/backends/postgres/code_system.rs` — `resolve_code_system` ORDER BY + `current` arm
 
+---
+
+## 2a. Provenance precedence: original vs re-published copy (issue #200)
+
+`hl7.fhir.r4.core` declares canonical `http://hl7.org/fhir` but re-ships **746
+CodeSystems and 603 ValueSets** owned by `http://terminology.hl7.org`. Those copies
+are stamped with the *FHIR release* version (`4.0.1`) rather than the code system's
+own version (`1.0.0`), and many are truncated — `audit-event-type` carries 1 of its
+5 concepts.
+
+Both rows are `content: complete` and both have concepts, so no tier above could
+separate them, and `4.0.1` outsorts `1.0.0` under **any** version ordering
+(lexicographic or semver). Before this fix, 208 CodeSystems resolved to the stale
+core copy and 1,297 concepts were unreachable via a bare `$validate-code`.
+
+### The rule
+
+Nothing intrinsic to the rows distinguishes them — only where they came from. Every
+FHIR NPM package declares a `canonical` base in its `package.json`; a package
+publishing a URL **outside** its own base is re-publishing someone else's resource.
+That is recorded at import time in `code_systems.authority_rank` /
+`value_sets.authority_rank` (see `import::bundle_parser::authority_rank_for`):
+
+| rank | meaning |
+|------|---------|
+| `0` `AUTHORITY_OWNER` | Positive evidence: the package declares a `canonical` base covering this URL, or a native importer loaded the publisher's own distribution (SNOMED RF2, LOINC, …). |
+| `1` `AUTHORITY_UNKNOWN` | Everything else — no package context (REST write, `$import-bundle`), or the package does not declare a base covering the URL. |
+
+**We only promote on positive evidence; we never demote on the absence of it.** That
+asymmetry is load-bearing, and getting it wrong breaks the fix in two ways:
+
+* **A package manifest may not describe its content's authority.** `us.nlm.vsac`
+  declares the fhir.org *registry* base `http://fhir.org/packages/us.nlm.vsac` yet
+  ships 14,850 ValueSets under `http://cts.nlm.nih.gov/...`. Those are genuine
+  originals. Branding them "foreign copies" would be a claim we have not established
+  — and since `vs_precedence_order_by` has no content/has-concepts tier beneath the
+  provenance tier, it would let any REST upload outrank the real VSAC definition
+  outright. Marking them merely *unknown* costs them nothing: rank is only ever a
+  tiebreak among rows sharing a URL, and no one else ships those URLs.
+* **An unattributed write must not claim ownership.** If `$import-bundle` defaulted to
+  `OWNER`, replaying `hl7.fhir.r4.core`'s truncated copy would stamp it rank 0 — and
+  because the upsert's `MIN` only ever *lowers* a rank, that copy would outrank THO's
+  original permanently, reinstating this bug with no way back.
+
+This needs no hardcoded URL list and no hand-maintained package-priority table: it
+self-maintains as packages are added.
+
+The tier sits **below** content and has-concepts deliberately: a populated copy still
+beats an empty stub from the owning package — being authoritative is worthless if the
+row has no concepts to validate against.
+
+### Upgrading an existing database
+
+`authority_rank` cannot be backfilled — nothing already stored on a row says which
+package supplied it. The column is therefore **nullable**, `NULL` meaning "never
+claimed", and readers `COALESCE` it to `0`, so an un-re-imported database behaves
+exactly as it did before. Adding the column is the signal that the database predates
+provenance, and the migration clears the `.tgz` rows from the `bootstrap_imports`
+ledger so the next startup re-imports those packages and records the truth.
+Without that, the ledger (which skips files whose size and mtime are unchanged) would
+leave every row at its default and the fix would be a silent no-op in production.
+Only `.tgz` entries are cleared — the multi-GB SNOMED/LOINC/RxNorm archives are not
+re-imported.
+
+The copies are **ranked, not discarded**: an explicit `version=4.0.1` still resolves
+to the core row, and `resource_json` stays byte-faithful to what the package shipped.
+
 ### Tests
 
 - `validate_code_version_current_resolves_loaded_edition` (SQLite integration test)

crates/hts/scripts/check-canonical-precedence.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+# Guard for issue #200.
+#
+# Same-URL precedence (which of several rows sharing a canonical URL wins) must
+# be expressed ONCE, in `backends::cs_precedence_order_by` / `vs_precedence_order_by`.
+# The bug this guards against was caused by that rule being re-implemented ad hoc
+# in ~40 backend queries that drifted apart, so that `$validate-code` resolved one
+# row while `$lookup` read another.
+#
+# Fails if any query orders `code_systems` / `value_sets` by a hand-rolled version
+# sort instead of the shared helper.
+set -euo pipefail
+cd "$(dirname "$0")/.."
+
+# A bare version sort is the signature of the old, divergent rule.
+if hits=$(grep -rn --include=*.rs \
+        -e "ORDER BY COALESCE(version, '') DESC" \
+        -e "ORDER BY COALESCE(s.version, '') DESC" \
+        -e "ORDER BY COALESCE(cs.version, '') DESC" \
+        src/ 2>/dev/null); then
+    echo "ERROR: hand-rolled same-URL ordering found." >&2
+    echo "Use crate::backends::cs_precedence_order_by() / vs_precedence_order_by() instead." >&2
+    echo "See crates/hts/docs/ig-publisher-compatibility.md §2a (issue #200)." >&2
+    echo "$hits" >&2
+    exit 1
+fi
+
+echo "OK: same-URL precedence is centralized."

crates/hts/src/backends/mod.rs

@@ -22,6 +22,80 @@ pub mod postgres;
 #[cfg(feature = "postgres")]
 pub use postgres::PostgresTerminologyBackend;
 
+/// Precedence among `code_systems` rows sharing one canonical URL.
+///
+/// This is the single definition of "which row wins when the caller did not pin
+/// a version". Both backends embed it verbatim, so SQLite and Postgres cannot
+/// drift apart. It assumes the candidate table is named/aliased `code_systems`.
+///
+/// Tiers, in order:
+///
+/// 1. **`content`** — a usable definition beats a `fragment`/`example`, which
+///    beats a `not-present` stub. Load-bearing: VSAC and the FHIR core packages
+///    ship empty `not-present` stubs for SNOMED/LOINC whose `version` string
+///    (`"current"`) text-sorts *above* a real edition (`"20260501"`), so this
+///    tier is the only thing keeping `version=current` resolving to the real
+///    import. See `docs/ig-publisher-compatibility.md`.
+/// 2. **has concepts** — a populated row beats an empty one, for the same reason.
+/// 3. **`authority_rank`** — the original beats a re-published copy. This is the
+///    tier that fixes the `terminology.hl7.org` collisions: `hl7.fhir.r4.core`
+///    re-ships 746 THO CodeSystems stamped with the *FHIR release* version
+///    (`4.0.1`), which outsorts the code system's own version (`1.0.0`) under any
+///    ordering — lexicographic or semver — even though the copy is truncated.
+///    Provenance is the only thing that separates those rows; nothing intrinsic
+///    to them does. `NULL` (a row imported before the column existed) coalesces
+///    to 0, so an un-re-imported database keeps its previous behaviour.
+/// 4. **`version`** — highest version string wins, as before.
+/// 5. **`id`** — a stable, deterministic final tiebreak.
+///
+/// Tier 3 sits *below* 1 and 2 deliberately: if a package that owns a canonical
+/// ships only an empty stub of it while some other package ships a populated
+/// copy, the populated row should still win — being authoritative is worthless
+/// if the row has no concepts to validate against.
+///
+/// `alias` is how the `code_systems` table is named in the enclosing query
+/// (`"code_systems"` when unaliased, or e.g. `"s"` / `"cs"`).
+///
+/// Every query that has to choose one row for a canonical URL must order by
+/// this. When they disagree, the server becomes internally incoherent — e.g.
+/// `$validate-code` resolving against the authoritative row while `$lookup`
+/// reads `resource_json` from the truncated copy. That incoherence is what
+/// let this bug class persist.
+pub(crate) fn cs_precedence_order_by(alias: &str) -> String {
+    format!(
+        "(CASE COALESCE({alias}.content, 'complete') \
+              WHEN 'complete'    THEN 0 \
+              WHEN 'supplement'  THEN 0 \
+              WHEN 'fragment'    THEN 1 \
+              WHEN 'example'     THEN 1 \
+              WHEN 'not-present' THEN 2 \
+              ELSE 1 END), \
+         (CASE WHEN EXISTS \
+             (SELECT 1 FROM concepts hc WHERE hc.system_id = {alias}.id) \
+             THEN 0 ELSE 1 END), \
+         COALESCE({alias}.authority_rank, 0), \
+         COALESCE({alias}.version, '') DESC, \
+         {alias}.id",
+    )
+}
+
+/// Precedence among `value_sets` rows sharing one canonical URL.
+///
+/// The ValueSet analogue of [`cs_precedence_order_by`]. `value_sets` has no
+/// `content` column and no concept rows, so only the provenance and version
+/// tiers apply: original beats re-published copy, then highest version, then a
+/// stable `id`.
+///
+/// This matters for the same reason: `hl7.fhir.r4.core` re-ships 603 THO
+/// ValueSets under canonical URLs it does not own.
+pub(crate) fn vs_precedence_order_by(alias: &str) -> String {
+    format!(
+        "COALESCE({alias}.authority_rank, 0), \
+         COALESCE({alias}.version, '') DESC, \
+         {alias}.id",
+    )
+}
+
 /// True when `ver` is the sentinel `current` (case-insensitive).
 ///
 /// IG Publisher, VSAC stubs, and CQL-generated Library `dataRequirement` entries