Rework article-analysis section criteria and issuer context - #1073
Merged
Conversation
Every newsletter section restated the same rules about issuer relevance, recency, and sourcing, so those rules fired for nearly any on-topic article and lifted all sections together. A third of placements were decided by a tie broken on print order rather than on editorial signal. The criteria are rewritten to five rules per section where every rule can be false, no rule repeats across sections, and a section's rules can hold at the same time. The issuer gate now admits the issuer's market rather than only the issuer, since a competitor's news is what Competitive Landscape is built from, and the placeholder family resolves from the curated ticker profile instead of the raw IDX columns, which described a coffee chain as soft drinks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
article-analysisscored an article against every section by counting matched rules over total, took the argmax, and broke ties on the order sections are printed in. Because every section restated the same rules about issuer relevance, recency, and sourcing, those rules fired for nearly any on-topic article and lifted all sections together, so 34% of placements (1,277 of 3,743) were settled by a tie rather than by evidence. This rewrites the criteria so scores separate on their own, widens the issuer gate to cover the issuer's market, and grounds the prompt placeholders in the curated ticker profile.Related issues
Closes #1072
Important changes
ip-material-to-issuermatched 100% of the time andip-industry-specific99%, so both were constants. Issuer relevance was one question asked six times, recency appeared in three sections, sourcing in three. Anddm-corporate-actionanddm-leadershipwere alternatives, so a Deals article could essentially never clear 2/5 while other sections reached 5/5.TickerProfile, falling back to the ticker's IDX columns only when no profile exists. The IDX columns are stored in Indonesian and are frequently too coarse to identify the real market, so FORE, a coffee shop chain, was described to the model asMinuman Ringan, meaning soft drinks, and the gate was asked whether a tea launch concerned a soft drinks company.{{INDUSTRY}}{{SUB_INDUSTRY}}{{INDUSTRY}}Other changes
qh-not-elsewherematched only 13% of the time in rows Quick Hits already won, meaning 87% of its placements were articles the model itself said fit a main section, while four rules matching 78% to 94% carried it over the line.dealsAndMovementsfolds appointments intodm-corporate-actionso the section's rules stack instead of branching.Key files to review
apps/mediapulse/agents/article-analysis/src/config-schema.ts- the full rule set. Worth reading as six blocks and checking each one against the three constraints above.apps/mediapulse/agents/article-analysis/src/llm-classify-section.ts- gate text and the system-prompt line. Both had to change together, since the prompt line restated the narrow reading.apps/mediapulse/agent-data-api/src/services/analysis.ts- profile preferred over IDX columns inmapTickerContext, plus the Prisma selects that feed it. The profile fields are optional so a ticker without a profile still resolves.How to test
pnpm code-qualityfrom the repo root. Expect 108 tasks green and a clean format check.pnpm --filter '@mediapulse/article-analysis' testandpnpm --filter '@mediapulse/agent-data-api' test. Expect 70 and 304 passing.article-analysisconfig in Hermes, then run the agent for FORE. The storedorchestration.agent_configrow is a materialised copy of the old 32 rules and wins over the code defaults, so nothing changes in production until it is re-saved.Notes for reviewers
ISSUER_RELEVANCE_RULE_IDSandISSUER_UNMATCHED_SCORE_CAPgo inert with this change, because every rule id they name is gone. They were already stale: the comment justifies the 0.4 cap by pointing at a content-generation Quick Hits demotion floor of 0.7 that does not exist anywhere in the codebase. Removing them touchesscoreFromEvaluationsand its tests, so it is left as a follow-up rather than widening this diff. The threeACCEPTANCE_CRITERIA*.mdfiles at the repo root also go stale, and no code reads them.