Skip to content

In some agent applications, descriptive text may mislead the classification of tools #736

Description

@AnalogKnight

This issue actually stems entirely from the Agent application, not from the project itself, but I believe some improvements can be made to avoid it.

I'm using this MCP in Copilot, and it initially worked perfectly. However, when using custom endpoints, the model consistently fails to find the required tools.

After further investigation, I discovered that for custom endpoints, Copilot uses virtual tools to collapse and categorize tools based on their names and descriptions when the number of tools exceeds 64. This process is completely automated and cannot be manually intervened. Therefore, if Copilot's categorization is inaccurate, the model will struggle to find the correct tools.

This is indeed the case. For example, Copilot categorizes the list_issues tool under a virtual tool named activate_group_and_repository_management_tools, while simultaneously generating a virtual tool named activate_issue_management_tools, but list_issues is not included. So, unless the user points it out, the model will always assume it cannot list issues.

Perhaps adding explicit categorization suggestion text to the tool descriptions could help? While this may not completely solve the problem, it should significantly improve the accuracy of agent classification.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions