Skip to content

fix(hts): create the concepts_search_fts FTS5 table it uses (#274)#280

Closed
mauripunzueta wants to merge 1 commit into
mainfrom
fix/274-concepts-search-fts
Closed

fix(hts): create the concepts_search_fts FTS5 table it uses (#274)#280
mauripunzueta wants to merge 1 commit into
mainfrom
fix/274-concepts-search-fts

Conversation

@mauripunzueta

Copy link
Copy Markdown
Contributor

Fixes #274.

The bug

crates/hts/src/backends/sqlite/value_set.rs reads from and writes to a concepts_search_fts FTS5 table (the typeahead ranker fts_candidates_ranked_for_system and its populate path populate_concepts_search_fts_for_system), but the table's CREATE VIRTUAL TABLE existed nowhere in the repository:

$ grep -rn "CREATE VIRTUAL TABLE.*concepts_search_fts" .
(no matches)

So the first text-filtered expansion of any system fails hard with no such table: concepts_search_fts, surfaced as an OperationOutcome exception. main currently fails value_set_ops::expand_is_a_filter_with_text_filter_still_nests on exactly this.

Why CI didn't catch it — semantic merge conflict

PR added green because
#194 the code that uses the table (no DDL) nothing in its tests reached the build path
#233 expand_is_a_filter_with_text_filter_still_nests, which does reach it its CI ran against a base predating #194

Both merged green; the combination on main is broken.

The fix

Add the table next to its siblings in schema.rs, and clear it at startup in the same two blocks that reset concepts_fts / concepts_word_fts.

The shape is derived from how value_set.rs uses it, not guessed:

  • term is the only tokenised column — the INSERTs fill it from both the concept's display and every designation value, which is what lets a query match a concept by any synonym and rank with bm25.
  • code is retrieved (SELECT code … GROUP BY code) and system_id is a post-filter (WHERE system_id = ?), so both are UNINDEXED.
  • Same trigram case_sensitive 0 tokenizer as concepts_fts, its empty-index fallback, so substring matching is consistent between the two.
CREATE VIRTUAL TABLE IF NOT EXISTS concepts_search_fts
USING fts5(system_id UNINDEXED, code UNINDEXED, term,
           tokenize='trigram case_sensitive 0');

Verification

Against a real SQLite (3.45): the DDL applies, both populate INSERTs run, the EXISTS populated-probe returns true, and a trigram MATCH on a designation value returns the owning concept's code ranked by bm25.

@sandhums (author of #194) — please sanity-check the column/tokenizer choice against your intent for the search ranker; the behaviour is inferred from the queries, but you'd know if term was meant to carry anything else.

Found while rebasing #261 (issue #200) onto main; #261's CI is blocked on this because GitHub tests the PR merged into main.

`crates/hts/src/backends/sqlite/value_set.rs` reads from and writes to a
`concepts_search_fts` FTS5 table (the typeahead ranker
`fts_candidates_ranked_for_system` and its populate path), but the table's
`CREATE VIRTUAL TABLE` existed nowhere in the repository. The first text-filtered
expansion of any system therefore fails hard:

    no such table: concepts_search_fts

surfaced as an OperationOutcome exception, and it fails the test
`value_set_ops::expand_is_a_filter_with_text_filter_still_nests` on main.

This is a semantic merge conflict: PR #194 added the code that uses the table
(without the DDL) and was green because nothing in its tests reached the build
path; PR #233 added the test that does reach it and was green because its CI ran
against a base predating #194. Both merged, and the combination breaks.

Add the table next to its siblings in `schema.rs`, and clear it at startup in the
same two blocks that reset `concepts_fts` / `concepts_word_fts`:

* `term` is the only tokenised column — it carries the concept's preferred display
  AND every designation value, which is what lets a query match a concept by any
  synonym and rank the hits with bm25.
* `code` is retrieved (SELECT / GROUP BY) and `system_id` is a post-filter, so
  both are UNINDEXED, exactly as the INSERT/SELECT/MATCH statements in
  value_set.rs use them.
* Same `trigram case_sensitive 0` tokenizer as `concepts_fts`, which is the
  fallback when this index is empty, so substring matching is consistent between
  the two.

Verified against a real SQLite (3.45): the DDL applies, both populate INSERTs run,
the `EXISTS` populated-probe returns true, and a trigram MATCH on a designation
value returns the owning concept's code ranked by bm25.
@mauripunzueta

Copy link
Copy Markdown
Contributor Author

Superseded by #273, which merged the same fix (create concepts_search_fts FTS5 table + valid ranked query) into main on 2026-07-15. main is green and no longer needs this. Closing as redundant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

main is broken: concepts_search_fts FTS5 table is used but never created (semantic merge conflict between #194 and #233)

1 participant