[Teams Chatbot] Refine intention classifier for human-assistance and add License-CLA-check doc - #16340
Merged
Conversation
|
Azure Pipelines: Successfully started running 3 pipeline(s). 62 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Azure SDK Q&A bot ecosystem to (1) reduce unwanted bot replies to “human escalation / thread bump” messages in Teams, and (2) expand/refresh knowledge coverage by adding an internal wiki doc and automatically triggering Azure AI Search reindexing after knowledge sync.
Changes:
- Refine the intention-classification prompt (and add tests) to suppress bot replies for pure cc/fyi routing addenda and generic “please assist” pleas after the bot already answered.
- Add an internal wiki knowledge doc entry (“My License-CLA check hung”) and enable that internal-wiki source for a tenant.
- Trigger Azure AI Search indexer runs after knowledge sync completes, using AAD-based auth and conflict-tolerant retry semantics.
Reviewed changes
Copilot reviewed 8 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/sdk-ai-bots/azure-sdk-qa-bot-knowledge-sync/src/services/SearchService.ts | Adds indexer client + runIndexer() and switches search auth to chained AAD credentials. |
| tools/sdk-ai-bots/azure-sdk-qa-bot-knowledge-sync/src/DailySyncKnowledge.ts | Invokes searchService.runIndexer() after blob cleanup to kick off reindexing. |
| tools/sdk-ai-bots/azure-sdk-qa-bot-knowledge-sync/package.json | Adds http-status-codes dependency for conflict handling. |
| tools/sdk-ai-bots/azure-sdk-qa-bot-knowledge-sync/package-lock.json | Locks the new http-status-codes dependency. |
| tools/sdk-ai-bots/azure-sdk-qa-bot-knowledge-sync/config/knowledge-config.json | Registers the “My License-CLA check hung” internal wiki document path. |
| tools/sdk-ai-bots/azure-sdk-qa-bot-function/package.json | Adds an overrides entry for brace-expansion. |
| tools/sdk-ai-bots/azure-sdk-qa-bot-function/package-lock.json | Updates brace-expansion resolution to 5.0.7. |
| tools/sdk-ai-bots/azure-sdk-qa-bot-agent/tests/intention_service_test.py | Adds new intention classifier tests for human-assistance pleas and cc routing messages. |
| tools/sdk-ai-bots/azure-sdk-qa-bot-agent/prompts/intention_classify.md | Tightens classifier guidance to treat cc/fyi and generic human-pleas as non-response-worthy. |
| tools/sdk-ai-bots/azure-sdk-qa-bot-agent/config/tenant_config.py | URL-encodes internal-wiki links and enables internal-wiki source for a tenant. |
Files not reviewed (2)
- tools/sdk-ai-bots/azure-sdk-qa-bot-function/package-lock.json: Generated file
- tools/sdk-ai-bots/azure-sdk-qa-bot-knowledge-sync/package-lock.json: Generated file
Co-authored-by: JiaqiZhang-Dev <194873822+JiaqiZhang-Dev@users.noreply.github.com>
JiaqiZhang-Dev
changed the title
lirenhe
reviewed
Jul 15, 2026
lirenhe
approved these changes
Jul 15, 2026
Member
Author
|
/check-enforcer override |
Copilot AI
pushed a commit
that referenced
this pull request
Jul 15, 2026
…add License-CLA-check doc (#16340) Co-authored-by: helen229 <19517014+helen229@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two changes to the Azure SDK Q&A bot:
1. Intention classifier Issues
After the bot already answered a question, a follow-up like "Can someone please assist on this request? Thanks!" or just "cc @xxx" was being re-classified as a "follow-up help request" and the bot answered again. That message is an escalation directed at humans, not a new question for the bot.
2. License-CLA-check knowledge doc
License-CLA-checkdoc inknowledge-config.jsonand addSRC_AZURE_SDK_INTERNAL_WIKIto the tenant's sources inconfig/tenant_config.py.3. Auto trigger AI Search reindex after knowledge sync to storage
Tests
1. Intention Test
Added two tests in
tests/intention_service_test.py(both exercise the real LLM classifier):test_human_assistance_plea_after_bot_reply_should_not_respond— bot answered, then a plea for a human →should_respond=false.test_cc_routing_addendum_should_not_respond— acc @Alice fyirouting addendum after the bot answered →should_respond=false.Live test:
2. License Doc Test
3. Knowledge Sync Pipeline Test
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6558418&view=logs&j=eb5be400-3fc3-547d-15aa-3c791611746f&t=92da56e7-f878-5692-bd8d-2243966cfd5c
