Skip to content

Rework article-analysis section criteria and issuer context - #1073

Merged
kevinhermawan merged 1 commit into
mainfrom
fix/article-analysis-section-criteria
Jul 29, 2026
Merged

Rework article-analysis section criteria and issuer context#1073
kevinhermawan merged 1 commit into
mainfrom
fix/article-analysis-section-criteria

Conversation

@kevinhermawan

Copy link
Copy Markdown
Contributor

Summary

article-analysis scored an article against every section by counting matched rules over total, took the argmax, and broke ties on the order sections are printed in. Because every section restated the same rules about issuer relevance, recency, and sourcing, those rules fired for nearly any on-topic article and lifted all sections together, so 34% of placements (1,277 of 3,743) were settled by a tie rather than by evidence. This rewrites the criteria so scores separate on their own, widens the issuer gate to cover the issuer's market, and grounds the prompt placeholders in the curated ticker profile.

Related issues

Closes #1072

Important changes

  • All 30 acceptance criteria rewritten, five per section across all six sections, under three constraints: every rule must be able to be false, no rule may read the same under another section, and a section's rules must be able to hold at the same time. The old set broke all three. ip-material-to-issuer matched 100% of the time and ip-industry-specific 99%, so both were constants. Issuer relevance was one question asked six times, recency appeared in three sections, sourcing in three. And dm-corporate-action and dm-leadership were alternatives, so a Deals article could essentially never clear 2/5 while other sections reached 5/5.
  • The issuer gate now admits the issuer's market, not only the issuer. It previously read "genuinely about, or materially concerns, the named issuer or its stated sector/industry", and the model collapsed that to "is this about the issuer", rejecting a competitor's product launch for FORE with the reason "not directly related to FORE". A competitor's news is the raw material of Competitive Landscape, so the gate text and the system-prompt line that reinforced the narrow reading both changed.
  • Prompt placeholders now resolve from the curated TickerProfile, falling back to the ticker's IDX columns only when no profile exists. The IDX columns are stored in Indonesian and are frequently too coarse to identify the real market, so FORE, a coffee shop chain, was described to the model as Minuman Ringan, meaning soft drinks, and the gate was asked whether a tea launch concerned a soft drinks company.
Placeholder Was sent Now
FORE {{INDUSTRY}} Minuman Coffee Shop Chains
FORE {{SUB_INDUSTRY}} Minuman Ringan App and Delivery-Led Coffee Chains
DSSA {{INDUSTRY}} Batu Bara Coal, Power and Telecom Holdings

Other changes

  • Quick Hits keeps five rules but they are now hard. qh-not-elsewhere matched only 13% of the time in rows Quick Hits already won, meaning 87% of its placements were articles the model itself said fit a main section, while four rules matching 78% to 94% carried it over the line.
  • dealsAndMovements folds appointments into dm-corporate-action so the section's rules stack instead of branching.

Key files to review

  • apps/mediapulse/agents/article-analysis/src/config-schema.ts - the full rule set. Worth reading as six blocks and checking each one against the three constraints above.
  • apps/mediapulse/agents/article-analysis/src/llm-classify-section.ts - gate text and the system-prompt line. Both had to change together, since the prompt line restated the narrow reading.
  • apps/mediapulse/agent-data-api/src/services/analysis.ts - profile preferred over IDX columns in mapTickerContext, plus the Prisma selects that feed it. The profile fields are optional so a ticker without a profile still resolves.

How to test

  1. pnpm code-quality from the repo root. Expect 108 tasks green and a clean format check.
  2. pnpm --filter '@mediapulse/article-analysis' test and pnpm --filter '@mediapulse/agent-data-api' test. Expect 70 and 304 passing.
  3. Re-save the article-analysis config in Hermes, then run the agent for FORE. The stored orchestration.agent_config row is a materialised copy of the old 32 rules and wins over the code defaults, so nothing changes in production until it is re-saved.
  4. Check "Lokale Luncurkan Signature Tea Series di Balikpapan" for FORE. It should now pass the gate and land in Competitive Landscape instead of being rejected as issuer-irrelevant.
  5. Check "Pengamat: Tata Kelola Batu Bara Jaga Keandalan Listrik Nasional" for DSSA. It should leave Competitive Landscape, which it only won on a four-way 0.40 tie. Industry Pulse is the likely destination, since the article reports no rule change, names no instrument, and states no obligation.

Notes for reviewers

ISSUER_RELEVANCE_RULE_IDS and ISSUER_UNMATCHED_SCORE_CAP go inert with this change, because every rule id they name is gone. They were already stale: the comment justifies the 0.4 cap by pointing at a content-generation Quick Hits demotion floor of 0.7 that does not exist anywhere in the codebase. Removing them touches scoreFromEvaluations and its tests, so it is left as a follow-up rather than widening this diff. The three ACCEPTANCE_CRITERIA*.md files at the repo root also go stale, and no code reads them.

Every newsletter section restated the same rules about issuer relevance,
recency, and sourcing, so those rules fired for nearly any on-topic article
and lifted all sections together. A third of placements were decided by a
tie broken on print order rather than on editorial signal. The criteria are
rewritten to five rules per section where every rule can be false, no rule
repeats across sections, and a section's rules can hold at the same time.

The issuer gate now admits the issuer's market rather than only the issuer,
since a competitor's news is what Competitive Landscape is built from, and
the placeholder family resolves from the curated ticker profile instead of
the raw IDX columns, which described a coffee chain as soft drinks.
@kevinhermawan
kevinhermawan merged commit 29adfca into main Jul 29, 2026
10 checks passed
@kevinhermawan
kevinhermawan deleted the fix/article-analysis-section-criteria branch July 29, 2026 07:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rework article-analysis section criteria and issuer context

2 participants