Skip to content

feat(search): store SearchParameters in storage; registry becomes a TTL cache#244

Open
angela-helios wants to merge 2 commits into
mainfrom
feat/235-storage-backed-search-params
Open

feat(search): store SearchParameters in storage; registry becomes a TTL cache#244
angela-helios wants to merge 2 commits into
mainfrom
feat/235-storage-backed-search-params

Conversation

@angela-helios

Copy link
Copy Markdown
Contributor

Closes #235. Stacked on #242 (the (base, code) shadowing fix) — per #239, that had to land first: with storage as the source of truth, every client-POSTed SearchParameter flows through register() with an arbitrary URL, and this PR's refresh relies on #242's unregister_source semantics (dropping a stored override re-promotes the spec param it shadowed).

What changes

Seeding (storage becomes the source of truth). On boot each primary backend seeds the default tenant with the spec bundle's raw resource JSON plus the embedded fallbacks (new definition_to_fhir_resource). Idempotent and multi-node safe, as the issue requires:

  • spec-bundle ids (Patient-name, Resource-id, …) make a concurrent second writer's create fail AlreadyExists, treated as already-seeded — no duplication, no clobbering;
  • a complete set short-circuits to a single count per boot;
  • a partial set (e.g. a user-POSTed parameter predating this feature) is completed resource-by-resource without touching what exists.

GET /SearchParameter?base=Patient now returns the parameters the server actually resolves searches with, instead of the empty Bundle the issue demonstrated.

Registry as a TTL cache. New HFS_SEARCH_PARAM_CACHE_TTL (seconds, default 3600, 0 disables) spawns a periodic refresh per serve path. refresh_stored_search_parameters() on SQLite/Postgres/Mongo rebuilds the registry's Stored slice from storage via the new SearchParameterRegistry::unregister_source primitive. A SearchParameter POSTed to one cluster node becomes visible to the others within the TTL — the deliberate eventual consistency the issue specifies.

Open questions from the issue, decided:

  • Re-read failure: stale-serve, explicitly. Rows are fetched and parsed before the sync write lock is taken; on any read error the registry is left untouched and the next tick retries (a node keeps resolving searches rather than losing its registry).
  • /metadata lag: accepted; a node's CapabilityStatement lags cluster-mates by up to the TTL, same as search resolution.
  • Reindexing on a mid-TTL parameter: out of scope as stated; noted in the seeder docs rather than left implicit.
  • params_by_type override hazard: resolved by fix(search): resolve (base, code) shadowing by source precedence #242 underneath this PR.

Design constraints honored (why the refresh looks like this)

  • The registry Arc identity is never swapped — the extractor, the ES shared registry, and both chain builders captured clones at construction and would silently serve a stale registry forever.
  • The write lock is sync (parking_lot) and shared with request paths — the refresh never holds it across an await; /metadata's single-guard build stays consistent.
  • In *-elasticsearch modes, seed/refresh run on the primary, whose registry Arc the ES backend shares. The S3 path is unchanged (no settings/registry write hooks exist there today — pre-existing gap, unchanged by this PR).
  • Seeded resources keep the spec's status: draft verbatim; the registry still loads spec definitions from the bundled file as Embedded (draft→active promotion unchanged), so the stored copies serve discovery without becoming a second registration path. This is SearchParameter viewer & editor UI: Resource Filter rail, faceted browse, CRUD with registry-aware validation #238's open question 1 made explicit.
  • Bonus fix: Mongo never reloaded stored SearchParameters after a restart (write-hooks only); its refresh now also runs at init_schema.

Multi-tenant note (flagging for review)

Seeding targets the default tenant because searches are tenant-scoped — seeding __system__ would leave GET /SearchParameter empty, defeating the issue's first goal. Non-default tenants don't see the seeded resources via the API (the in-memory registry still resolves searches for every tenant, and the refresh reads cross-tenant exactly like today's boot load). If we later want shared-resource search (TenancyModel::Shared exists but nothing consults it), that's a separate design.

Verification

  • New integration tests (SQLite): idempotent seed + API discoverability, partial-set completion without clobbering, and the TTL contract (stored param enters resolution on refresh, deleted param leaves it).
  • New unit test: unregister_source drops only its source and re-promotes a shadowed spec param.
  • End-to-end on a file-backed hfs: first boot seeds 1377 (failed=0), GET /SearchParameter?base=Patient returns spec entries, POST /SearchParameter → 201, second boot takes the fast path (created=0, existing=1384) and reloads the stored custom param.
  • helios-fhir 22/22, seeding suite 3/3, full helios-hfs check clean with postgres,mongodb,elasticsearch.

…TL cache (#235)

Storage is now the source of truth for SearchParameters; the in-memory
registry is a cache over it.

- Seeding: on boot, each primary backend seeds the default tenant with
  the spec bundle (raw resource JSON, verbatim including status) plus
  the embedded fallbacks rendered as FHIR resources. Idempotent and
  multi-node safe: spec bundle ids make a concurrent second writer fail
  AlreadyExists (treated as seeded); a full set short-circuits to one
  count per boot; a partial set is completed without clobbering.
  GET SearchParameter now discovers the ~1380 parameters the server
  actually resolves, instead of an empty Bundle.
- Refresh: refresh_stored_search_parameters() on the SQLite, Postgres,
  and Mongo backends rebuilds the registry Stored slice from storage
  (new SearchParameterRegistry::unregister_source primitive, which also
  re-promotes a spec param shadowed by a since-deleted override). Rows
  are fetched and parsed before the sync write lock is taken and the
  registry Arc identity is preserved (extractor / ES / chain builders
  hold clones); on a failed read the registry is left untouched, i.e.
  stale-serve, retried next tick.
- HFS_SEARCH_PARAM_CACHE_TTL (default 3600, 0 disables) drives a
  periodic refresh task per serve path, so a SearchParameter POSTed to
  one cluster node becomes visible to the others within the TTL. In
  elasticsearch composite modes seed/refresh run on the primary, whose
  registry Arc the ES backend shares.
- Mongo previously never reloaded stored SearchParameters after a
  restart (they only entered the registry via write hooks); its new
  refresh also runs at init_schema, closing that gap.

Verified end to end on a file-backed server: first boot seeds 1377,
GET SearchParameter?base=Patient returns spec entries, POST custom is
201, second boot takes the fast path (created=0) and reloads the
stored custom param.
…ed-search-params

# Conflicts:
#	crates/hfs/src/main.rs
@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Store SearchParameters in storage; make the in-memory registry a TTL cache over it

2 participants