fix(ingestion): report intentionally-excluded docs separately from Qdrant duplicates

[detail-app]

Detail bug report: View on Detail

Closes #220

Bug

DocumentProcessor logged intentionally-excluded documentation pages (Javadoc class-use/*.html index pages, frameset/Redoc navigation shells) as Qdrant duplicates. It bound duplicates = outcome.backlog().skippedFiles() and labeled that count "Skipped {} duplicate files (already in Qdrant)", but IngestionBacklogStatus.skippedFiles was documented as "already indexed or intentionally excluded". Excluded pages are not upserted and may have their points deleted, so they hold zero Qdrant points — the opposite of "already in Qdrant" — yet were counted as duplicates. Root cause: LocalDocsFileIngestionProcessor.processExcludedPage (and the unchanged-excluded-page branch) reused the generic skippedFile() outcome for a semantically distinct category, and the CLI labeled the whole skippedFiles bucket as duplicates. Producer (skippedFile(): "unchanged or already indexed") and consumer (skippedFiles: "already indexed or intentionally excluded") contracts disagreed.

Fix

Introduced a dedicated excluded category end-to-end so the operator-facing labels are honest and the contracts reconcile (the bug report's preferred option):

• LocalDocsFileOutcome — new Excluded sealed variant + excludedFile() factory; skippedFile() javadoc narrowed back to "unchanged or already indexed" (genuine duplicates that retain their Qdrant points).
• IngestionBacklogStatus — new excludedFiles component; invariant becomes inspected == processed + skipped + excluded + failed; completeBatch(...,excludedCount,...); notStarted/running/startBatch/finish/abandon/resume carry excludedFiles. abandon() keeps excluded in the terminal-success prefix (processed + skipped + excluded), so interrupted runs don't re-process excluded pages. The record is serialized in the durable run checkpoint; old checkpoints without excludedFiles deserialize as 0 and still satisfy the (now-relaxed-to-old) invariant.
• LocalDocsFileIngestionProcessor — processExcludedPage and the "unchanged excluded Java API page" branch return excludedFile(); genuine unchanged-duplicate branches keep skippedFile().
• LocalDocsDirectoryIngestionService — routes Excluded outcomes to a batchExcludedCount and passes it to completeBatch.
• DocumentProcessor — keeps the existing duplicate labels (now truthful, since skippedFiles no longer contains excluded pages) and adds " Excluded {} files (intentionally not indexed)" per set and "Total excluded files: {}" in the summary; excluded flows through IngestionTotals and ProcessingOutcome.
• GitHubRepoProcessor — its exhaustive switch over LocalDocsFileOutcome combines Skipped/Excluded into one arm (source-code ingestion never produces Excluded; both count as the existing skipped bucket; preserves exhaustiveness without a SpotBugs duplicate-clause finding).

Testing

Unit tests, typecheck, lint, and build all pass: ./gradlew test (full JVM suite), compileJava/compileTestJava, spotlessCheck, spotbugsMain/spotbugsTest, pmdMain/pmdTest, ast-grep Java rules, cd frontend && npm run check (svelte-check, 0 errors), and cd cli && node --test (43/43) + npm run pack:check.

New and updated committed tests:

• DocumentProcessorExcludedCategoryLogTest drives two doc sets through the real DocumentProcessor (only the ingestion use case mocked) with skippedFiles=2/excludedFiles=3 and skippedFiles=1/excludedFiles=0, capturing real Logback output, and asserts the exact Skipped … duplicate files (already in Qdrant) and Excluded … files (intentionally not indexed) lines, the Total duplicates skipped: 3 / Total excluded files: 3 summary, and that the pre-fix conflation (Total duplicates skipped: 5/6) is absent.
• IngestionBacklogStatusTest — new cases pin that completeBatch accumulates excludedFiles separately, and that abandon() retains excludedFiles in the terminal-success prefix after an interruption.
• LocalIngestionRunStoreTest.readsCheckpointWrittenBeforeExcludedFilesFieldWasIntroduced — writes a current checkpoint, strips the excludedFiles JSON field to simulate the old schema, and asserts it reads back with excludedFiles=0 and the invariant holds (backward compatibility for in-flight durable checkpoints).
• LocalDocsFileIngestionProcessorTest — the class-use, frameset, and Redoc-shell tests now assert instanceof LocalDocsFileOutcome.Excluded, pinning that excluded pages no longer reuse Skipped; the genuine unchanged-duplicate path still returns Skipped.
• DocumentProcessorFailureContractTest zero case updated for the new field and now also asserts Total excluded files: 0.

End-to-end verification against real Qdrant 1.18.3 (via docker compose -f infra/docker-compose-qdrant.yml), running the built bootJar with the cli profile against a staged Javadoc class-use/List.html corpus:

• First run (fresh excluded page, processExcludedPage with deleteByUrl + empty marker): CLI logged Excluded 1 files (intentionally not indexed) and Total excluded files: 1 with Total duplicates skipped: 0 (pre-fix this was Total duplicates skipped: 1). Qdrant REST points/count filtered by the class-use URL returned 0 — the page holds zero points, the opposite of "already in Qdrant".
• Second run (unchanged excluded page via the empty-chunkHashes marker branch): again logged Excluded not a duplicate, confirming the unchanged-excluded path now returns excludedFile().

One sub-case could not be verified live: a genuine already-indexed duplicate that retains its Qdrant points. Reproducing it requires indexing a file first (which calls the embedding gateway), then re-running to hit the skip path; the local embedding server / shared LLM gateway was not reachable in this environment. The preserved duplicate label is nonetheless asserted against real DocumentProcessor log output in DocumentProcessorExcludedCategoryLogTest, and the genuine-duplicate code branch (unchanged fingerprint + Qdrant points present → skippedFile()) is exercised through the real processor in LocalDocsFileIngestionProcessorTest.shouldKeepDistinctIngestionStateForJavaPagesThatShareOneCitation.

---

Automatic Fixes PRs can be configured here.