Skip to content

fix(slack): Stop an empty Seer mention before it runs an agent turn - #122941

Merged
sehr-m merged 1 commit into
masterfrom
sehrmoosabhoy/slack-empty-prompt-after-mention-strip
Aug 28, 2026
Merged

fix(slack): Stop an empty Seer mention before it runs an agent turn#122941
sehr-m merged 1 commit into
masterfrom
sehrmoosabhoy/slack-empty-prompt-after-mention-strip

Conversation

@sehr-m

@sehr-m sehr-m commented Aug 27, 2026

Copy link
Copy Markdown
Member

Problem

_handle_seer_prompt already rejects empty text, but it checks before the bot mention is stripped — that happens later, in process_mention_for_slack via extract_prompt. A bare @Seer has truthy text "<@U0BOT>", so it passes the guard and reaches Seer as query="".

Seer then runs a full turn on nothing — an ExplorerIndex query, four context engine stages, three Vertex embedding calls and an LLM call — only to answer "It looks like your message came through empty!".

The embedding calls 400 with The text content is empty.:

path error events / 30d
direct google-genai ClientError: Empty instances. SEER-8GR — 393 (296 prod, 97 de)
llm-proxy /embeddings Vertex 400: "The text content is empty." LLM-PROXY-S

All 393 are explorer.category_key:slack_thread and explorer_main_task. Slack is the only entrypoint affected — the Explorer chat endpoint already enforces allow_blank=False, and every other caller templates its prompt, so query is structurally non-blank there.

Fix

Guard at the webhook, before set_thread_status("is thinking...") so no spinner is left running.

The check is a new has_prompt_content — is there anything besides mentions and whitespace — rather than reusing extract_prompt. extract_prompt needs the bot's user id, which comes from the event's authorizations; Slack omits that in some cases, and test_app_mention_dispatches_task_no_authorizations already covers it. With no id the regex degrades to <@> and strips nothing, so a bare mention would have slipped through as the non-empty prompt "<@U0BOT>".

Another user's mention still counts as context whenever real text accompanies it; only a mention-only message is empty.

Halting silently would have been a regression — the user used to get an answer, however wasteful. send_empty_prompt_message sends a fixed ephemeral nudge, mirroring send_not_org_member_message: no link button, and send failures are logged rather than raised so a Slack outage can't turn the webhook into a 500. No model, no agent run, no Vertex call.

Uses a distinct EMPTY_PROMPT halt reason rather than MISSING_EVENT_DATA, which is documented as meaning the event was malformed — a bare mention is a normal user action, and conflating them would make that metric misleading.

Test

tests/sentry/integrations/slack/webhooks/events/ + tests/sentry/seer/entrypoints/slack/ — 201 passed. mypy and pre-commit clean.

New coverage: bare mention halts and sends the nudge; halts with no authorizations; mention-only-with-other-users halts; a mention plus real text still dispatches; a failing nudge still returns 200 without dispatching.

Notes for review

  • This guard is the only empty check on the Slack path. process_mention_for_slack re-derives the prompt and trigger_agent forwards it, neither re-checks — so invoking the task directly still bypasses it.
  • Not fixed here: when authorizations is missing, the <@U0BOT> token still rides along into the query as prompt noise. Storing the bot user id on the integration would close that; it's a prompt-quality issue, not correctness, now that this guard doesn't depend on the id.

🤖 Generated with Claude Code

`_handle_seer_prompt` rejects empty `text`, but it checks before the bot
mention is stripped — that happens later, in `process_mention_for_slack`
via `extract_prompt`. A bare "@seer" has truthy text `"<@U0BOT>"`, passes
the guard, and reaches Seer as `query=""`.

Seer then runs a full turn on nothing: an ExplorerIndex query, four
context engine stages, three Vertex embedding calls and an LLM call, only
to answer "It looks like your message came through empty!". The embedding
calls 400 with `The text content is empty.` — 393 events in 30 days
(SEER-8GR, plus LLM-PROXY-S on the proxy path), all of them
`explorer.category_key:slack_thread`.

Guard at the webhook instead, before `set_thread_status("is thinking...")`
so no spinner is left running.

The check is `has_prompt_content`, which asks whether anything remains
besides mentions and whitespace, rather than reusing `extract_prompt`.
`extract_prompt` needs the bot's user id, which comes from the event's
`authorizations` — Slack omits that in some cases, and the existing
`test_app_mention_dispatches_task_no_authorizations` covers exactly that.
With no id the regex degrades to `<@>` and strips nothing, so a bare
mention would have slipped through as the non-empty prompt `"<@U0BOT>"`.
Another user's mention still counts as context whenever real text
accompanies it; only a mention-only message is empty.

Halting silently would have been a regression — the user used to get an
answer, however wasteful. `send_empty_prompt_message` sends a fixed
ephemeral nudge, mirroring `send_not_org_member_message`: no link button,
send failures logged rather than raised so a Slack outage cannot turn the
webhook into a 500. No model, no agent run, no Vertex call.

Uses a distinct `EMPTY_PROMPT` halt reason rather than
MISSING_EVENT_DATA, which is documented as meaning the event was
malformed — a bare mention is a normal user action, and conflating them
would make that metric misleading.
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Aug 27, 2026

@alexsohn1126 alexsohn1126 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good! thank you 🙏

@sehr-m
sehr-m marked this pull request as ready for review August 28, 2026 16:33
@sehr-m
sehr-m requested review from a team as code owners August 28, 2026 16:33
@sehr-m
sehr-m merged commit 65ceb6b into master Aug 28, 2026
68 of 69 checks passed
@sehr-m
sehr-m deleted the sehrmoosabhoy/slack-empty-prompt-after-mention-strip branch August 28, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants