fix(ingestion): match navigation "header" by class token, not substring

[detail-app]

Detail bug report: View on Detail

Bug

HtmlContentExtractor.isNavigationElement flagged any element whose class attribute contained the substring "header" as navigation. That over-matched content wrappers whose class token merely ends in header — page-header, card-header, section-header, subheader — so when one was a direct child of the selected content element, appendFormattedChild skipped it and dropped the nested heading. The heading never reached chunking or either encoder, so it was absent from both the dense and lexical/sparse vectors (and from the chunk-text portion of the reranker prompt). The same "remove header" intent is implemented elsewhere in the file as the CSS selector .header, which matches the whole token header and does not match page-header — so the two paths diverged exactly on the *-header content wrappers.

Fix

Narrow only the header check to a whole class-token match (classTokens.contains("header") over element.classNames()), leaving nav/menu/sidebar/footer as substring checks. This aligns isNavigationElement's header semantics with the existing .header selector. Composite nav classes (top-nav, main-nav, header-nav, primary-nav) stay caught by the unchanged className.contains("nav"), so genuine navigation is still removed; *-header content wrappers, which have no navigational reading, now survive and carry their nested headings through to chunking.

Testing

• Committed unit tests in HtmlContentExtractorTest:
  • pageHeaderWrapperRetainsNestedHeading — a direct-child <div class="page-header"><h1>…</h1></div> retains the heading.
  • exactHeaderClassTokenIsStillRemoved — <div class="header"> is still dropped, preserving the intended "remove header" behavior.
  • compositeNavClassesAreStillRemoved — top-nav/main-nav/header-nav/primary-nav are still removed, pinning the asymmetry (only header was narrowed).
• As a non-versioned check, I reverted the one-line fix and confirmed both survival tests then fail (the heading is absent from the extracted text), then restored the fix and confirmed they pass — so the tests genuinely catch the regression rather than asserting a tautology.
• Routine checks pass: compile, the full unit suite (1169 tests), make test (17 shell contract scripts), and the lint lane (Spotless/PMD/SpotBugs, ast-grep rules, ruby SSOT check). The 10 pre-existing extractor tests, including the Javadoc class="header" fixtures, stay green.
• End-to-end: I ran local Qdrant plus a stub OpenAI-compatible /v1/embeddings endpoint, booted the backend in local-embedding mode, and ingested a crafted page whose only occurrence of a distinctive term sat inside a direct-child <div class="card-header"><h2>…</h2></div>. The resulting Qdrant docs chunk's stored text contained the distinctive heading term — the heading survived extraction, was chunked, was passed to both encoders, and was upserted. Under the old code that wrapper would have been dropped as navigation and the chunk would have held only the sibling body prose.
• A dedicated JavaApiPageExtractorTest doesn't exist in the repo, so that targeted run found no such test class. The Javadoc exact-token .header path it would cover is exercised by the existing Java API page fixtures (extractsModernClassMembersWithTheirExactDomAnchorsInSourceOrder, retainsPackagePagesAsUnanchoredJavaApiOverviews, etc.), which stay green.

Closes #228

---

Automatic Fixes PRs can be configured here.