Skip to content

Improve data-collection insights drop-reason clarity for operators #747

Description

@kevinhermawan

Summary

The data-collection insights tab shows a flat breakdown of drop reasons with counts and bars, but gives an operator no way to tell a benign by-design drop (already saved, duplicate, dead URL, noise) from a real quality rejection (not about the ticker, too old, too short). Plans 121 and 122 removed the confusing "Per-query budget" label, so "Relevance" now leads the breakdown — but the underlying gap remains: an operator can still ask "is a high Relevance drop good or bad?" and the dashboard cannot answer that on its own.

This issue tracks the product and design work needed before implementation can begin.

Scope

  • In scope:
    • Clarifying drop-reason presentation in the insights tab so an operator can self-serve without engineering help
    • Deciding between tooltips/descriptions, grouping (by-design vs quality-filtered), a plain-language summary, or a collection funnel — or a combination
    • Scoping any agent-side counter emission work if the funnel option is chosen
  • Out of scope:
    • Changing what gets collected or how it is filtered (presentation only, except for optional funnel emission)
    • Anything already delivered in Plans 121 and 122

Acceptance criteria

  • Open questions below are answered by product and design
  • A revised execution plan (or updated 123 stub) is agreed upon before any code is written
  • After implementation: an operator reading the insights tab can understand each drop reason and distinguish by-design drops from quality rejections without asking an engineer

Notes

Open questions to resolve before execution:

  • What specific questions must an operator be able to answer from this tab? That set defines done.
  • Which UX direction: descriptions/tooltips, grouping, a summary line, the funnel, or a combination?
  • Is the funnel wanted enough to justify agent-side emission work (discovered, afterPrefilter, fetched counters are currently empty in stored run records)?
  • Should drop-reason descriptions live in the API payload (data-driven, reusable across agents) or in the dashboard (faster to iterate)?
  • Does this generalize to other agents' insights breakdowns, or is it data-collection only for now?

Depends on: Plans 121 (#744) and 122 (#746) must be merged first so the drop-reason set is stable. Both are already on main.

Related prior work: Plan 120 built the current insights visual; Plan 115 built the insights provider.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions