Resolves LAT-TUA1: skip user-centric flaggers on signal-details dogfood traces - #4378
Resolves LAT-TUA1: skip user-centric flaggers on signal-details dogfood traces#4378geclos wants to merge 1 commit into
Conversation
…races The signal-details generation (issue:details) embeds occurrence feedback quoting end users in its user-role prompt, so the frustration flagger read the quoted complaints as real user frustration and raised a false signal in the Signal Discovery dogfood project (LAT-TUA1). Add the tag to the nested-conversation-sample list so user/input-centric strategies skip those sessions, matching the existing flagger and taxonomy handling. Co-authored-by: Gerard <gerard@latitude.so>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughSignal-details telemetry tags are now treated as nested conversation samples. User-centric reflagging is skipped without occurrence feedback and remains applicable when occurrence feedback is enabled. Tests and flagger documentation cover this behavior. ChangesFlagger reflagging
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
📄 Generated a rich HTML walkthrough of this PR's diff against Artifact: https://artifact-pr-4378-diff-review.nostalgic-keyboard.workers.dev Claim within 60 minutes (permanent link): https://dash.cloudflare.com/claim-preview?claimToken=wHKGSa7HOTDve69WxvteteJ8ucJHB1gM1QumM4cpDtI Self-contained artifact, no external requests. Generated by an automated agent. |
What does this PR do?
Resolves LAT-TUA1
Fixes a false-positive signal in the internal Signal Discovery dogfood project: Contradictory platform guidance causes user confusion.
The flagged trace (
d485c729427aae2d8af55e389a697aa5, tagissue:details) is Latitude's own signal-details generation (generateSignalDetailsUseCase). Its user-role prompt embeds occurrence feedback that quotes end users of another organization ("kazales uzywac shopify jako miejsce edycji a teraz mowisz ze w mc", "I am just going to pay someone to do this", etc.). The user-centricfrustrationflagger ran on that dogfood session and read the quoted complaints as real user frustration, raising a signal about a "contradictory platform guidance" problem that does not exist in the trace being scored — it belongs to the customer conversations quoted inside the prompt.The repo already has a guard for exactly this failure mode:
isUserCentricReflagInapplicabledrops user/input-centric strategies (frustration,jailbreaking,nsfw) on sessions whose tags mark the user-role text as nested conversation samples. The list covered flagger classify/draft and taxonomy tags but notissue:details. This PR addsAI_GENERATE_TELEMETRY_TAGS.signalDetailstoNESTED_CONVERSATION_SAMPLE_TAGS, so signal-details generations keep assistant-response-centric flagging (self-observability of the naming output) while no longer being misread by user-centric flaggers.Also updates
dev-docs/flaggers.mdto list the new tag in the self-observability section.Related issue (if applicable)
Latitude signal LAT-TUA1 (ID
pdyowwdmxpdqc4lrgzyjukhd).How was this tested?
packages/domain/flaggers/src/reflag.test.ts:issue:detailstags skip user-centric strategies while assistant-response-centric strategies still run.pnpm --filter @domain/flaggers test— 346 passed, 2 skipped.pnpm --filter @domain/flaggers typecheck— clean.biome checkon the touched files — clean.Checklist
Summary by CodeRabbit
Bug Fixes
Tests