Assistant: Improve managing language models (#9799)

This PR makes a few changes in an attempt to improve managing LLM
provider models.

The manage models quickpick menu has been improved:
* Copilot's BYOK providers, particularly ones that Assistant does not
currently support (e.g. Grok) are no longer shown.
 * Only providers that the user has signed in with are shown
* The `managementCommand` property has been added to Assistant
providers, so that the cog icon works.
 * These changes address #9283.

A new configuration setting `positron.assistant.filterModels` has been
added. With this users (and in the future admins) can set glob filters.
Models will only be shown if they match one of the given globs.
 * This addresses #9788.

A `positron.assistant.preferredModel` setting has been added. With this,
when the Positron window first loads it should default to loading the
model given in that setting. Here we use partial matching on the ID or
model names.
* This partially addresses #9388, in that IIUC it matches the behaviour
requested in the [final
comment](#9388 (comment))
of that thread.

A `positron.assistant.defaultModels` settings has been added. With this,
users can specify the default model for a particular provider, which we
will pre-select in the model selector if the user hasn't specified a
preferred model instead.
* This addresses the remaining parts of
#9388 based on
#9799 (comment).
* We have set the default model for Anthropic to `Claude Sonnet 4` and
Bedrock to `Claude 4 Sonnet Bedrock`

---

### Release Notes

#### New Features

- Assistant: added setting `positron.assistant.filterModels` to
configure available models (#9788)
- Assistant: added setting `positron.assistant.preferredModel` to set
the preferred model to pre-select in the model selector (#9388)
- Assistant: added setting `positron.assistant.defaultModels` to set the
default model per provider (#9388)

#### Bug Fixes

- Assistant: non-functional providers removed from "Manage models..."
quickpick (#9283)


### QA Notes

#### Manage models quickpick
1. First, ensure you have multiple providers configured and signed in. I
used Anthropic, AWS Bedrock, and GitHub Copilot Chat.
2. In the Assistant panel, invoke the model picker: <img width="403"
height="173" alt="Screenshot 2025-10-06 at 15 42 22"
src="https://github.com/user-attachments/assets/2cc33831-37c3-4f46-a1fc-887f32d93311"
/>
3. Ensure only useful entries are shown: <img width="620" height="196"
alt="Screenshot 2025-10-06 at 15 42 27"
src="https://github.com/user-attachments/assets/d644d1ea-f198-4bdb-bc3c-752c0fe8e20a"
/>
4. Select a _signed in_ model, ensure a model list appears:
<img width="617" height="297" alt="Screenshot 2025-10-06 at 15 42 39"
src="https://github.com/user-attachments/assets/9fcef6a6-f623-4349-8529-27f1e6713bb3"
/>

5. Do the same but select a signed out model, ensure the model
configuration modal appears.
6. Do the same thing with a _signed in_ model but click the cog on the
far right, the model configuration modal should again appear.

#### Model filtering

1. In your user settings, set the following two filters:
```
"positron.assistant.filterModels": ["Claude*", "GPT-5"],
```
2. Restart Positron or Reload the window.
3. Ensure that only Claude family models or GPT-5 can be used: 
<img width="366" height="238" alt="Screenshot 2025-10-06 at 15 49 04"
src="https://github.com/user-attachments/assets/3e7ac3ed-b9bd-41a8-b8a4-ca7b2bf0aca3"
/>

4. This should also work with IDs, useful for e.g. OpenRouter:
```
"positron.assistant.filterModels": ["**/google/gemini-*"],
```
<img width="431" height="389" alt="Screenshot 2025-10-06 at 16 07 43"
src="https://github.com/user-attachments/assets/c54e9720-ab3b-429b-a33d-4485dbc454fd"
/>

5. Finally, this should also restrict the manage models quickpick:
<img width="604" height="476" alt="Screenshot 2025-10-06 at 16 08 20"
src="https://github.com/user-attachments/assets/e1c45c9e-ca16-46c6-9f51-0ca3140dd954"
/>

#### Setting preferred model

1. In your user settings, remove the `positron.assistant.filterModels`
setting and set the following:
```
"positron.assistant.preferredModel": "GPT-5",
```
2. In the Assistant chat panel, switch to another provider and model,
e.g. AWS Bedrock and Claude 4 Sonnet.
3. Create a new chat, ensure selected model does not change.
4. Reload the window, ensure that on reload the preferred model set in
the configuration is first selected. In this case, it should switch to
GPT-5 from the Copilot provider on reload.

#### Setting default model per provider

1. Sign into multiple providers, at least Anthropic and AWS Bedrock.
2. Try switching between them, confirm that the defaults we want by
default are selected.
3. Set the following settings,
```
  "positron.assistant.defaultModels": { "anthropic-api": "Opus", "amazon-bedrock": "Opus" },
```
4. Reload the window, confirm that switching between the models now
selects Opus models by default.

Note 1: Upstream has code to remember your last selection, so after the
reload window you have to change providers to see the default kick in.

Note 2: Sometimes Assistant gets stuck in a state showing "Pick
Model..." in the selector. Switching providers does nothing to clear
that, but selecting a model first, and then switching providers, gets
things working again. I don't know if this is related to this commit, or
the changes in #9799.

---------

Co-authored-by: sharon wang <[email protected]>
Co-authored-by: Tim Mok <[email protected]>

Author: 3 people

extensions/positron-assistant/package.json

@@ -287,31 +287,77 @@
             "default": 60,
             "description": "%configuration.providerTimeout.description%",
             "minimum": 1
+          },
+          "positron.assistant.filterModels": {
+            "type": "array",
+            "default": [
+              "**/*"
+            ],
+            "markdownDescription": "%configuration.filterModels.description%",
+            "items": {
+              "type": "string"
+            },
+            "examples": [
+              [
+                "Claude*",
+                "GPT-5"
+              ],
+              [
+                "**/google/gemini*"
+              ]
+            ]
+          },
+          "positron.assistant.preferredModel": {
+            "type": "string",
+            "default": "",
+            "markdownDescription": "%configuration.preferredModel.description%",
+            "examples": [
+              "Claude Sonnet 4.5",
+              "GPT-5"
+            ]
+          },
+          "positron.assistant.defaultModels": {
+            "type": "object",
+            "default": {},
+            "additionalProperties": {
+              "type": "string"
+            },
+            "markdownDescription": "%configuration.defaultModels.description%",
+            "examples": [
+              {
+                "anthropic-api": "Claude Sonnet 4.5"
+              }
+            ]
           }
         }
       }
     ],
     "languageModels": [
       {
         "vendor": "echo",
-        "displayName": "Echo"
+        "displayName": "Echo",
+        "managementCommand": "positron-assistant.configureModels"
       },
       {
         "vendor": "amazon-bedrock",
-        "displayName": "Amazon Bedrock"
+        "displayName": "Amazon Bedrock",
+        "managementCommand": "positron-assistant.configureModels"
       },
       {
         "vendor": "anthropic-api",
-        "displayName": "Anthropic"
+        "displayName": "Anthropic",
+        "managementCommand": "positron-assistant.configureModels"
       },
       {
         "vendor": "openai",
-        "displayName": "OpenAI"
+        "displayName": "OpenAI",
+        "managementCommand": "positron-assistant.configureModels"
       },
-	  {
-		"vendor": "openai-compatible",
-		"displayName": "Custom Provider"
-	  }
+      {
+        "vendor": "openai-compatible",
+        "displayName": "Custom Provider",
+        "managementCommand": "positron-assistant.configureModels"
+      }
     ],
     "languageModelTools": [
       {

extensions/positron-assistant/package.nls.json

@@ -34,5 +34,8 @@
 	"configuration.followups.enable.description": "Enable suggested followup prompts after chat responses.",
 	"configuration.consoleActions.enable.description": "Enable Positron Assistant console actions, such as Fix and Explain.",
 	"configuration.providerTimeout.description": "Specifies the timeout in seconds when signing in to a provider.",
-	"configuration.alwaysIncludeCopilotTools.description": "Allow any model in Positron Assistant to use tools provided by GitHub Copilot.\n\nThis requires that you are signed into Copilot, and **may send data to Copilot models regardless of the selected provider**.\n\nSee also: `#chat.useCopilotParticipantsWithOtherProviders#`"
+	"configuration.alwaysIncludeCopilotTools.description": "Allow any model in Positron Assistant to use tools provided by GitHub Copilot.\n\nThis requires that you are signed into Copilot, and **may send data to Copilot models regardless of the selected provider**.\n\nSee also: `#chat.useCopilotParticipantsWithOtherProviders#`",
+	"configuration.filterModels.description": "A list of [glob patterns](https://aka.ms/vscode-glob-patterns) to filter available LLM models in Positron Assistant.\n\nAllows only language models with ID or name matching at least one of the patterns to be used. Defaults to allowing all models.\n\nRequires a restart to take effect.\n\nExamples:\n- `Claude*` allows all models starting with 'Claude'\n- `GPT-5` allows models named 'GPT-5'\n- `**/google/gemini-*` allows all models with '/google/gemini-' in the ID",
+	"configuration.preferredModel.description": "A preferred model ID or name (partial match supported) to use if available in Positron Assistant for the current provider.\n\nTakes precedence over `#positron.assistant.defaultModels#`.\n\nRequires a restart to take effect.\n\nExamples:\n- `Claude Sonnet 4.5` prefers the model named 'Claude Sonnet 4.5'\n- `GPT-5` prefers the model named 'GPT-5'",
+	"configuration.defaultModels.description": "A mapping of provider IDs to default model IDs or names (partial match supported) to use for that provider in Positron Assistant.\n\n`#positron.assistant.preferredModel#` takes precedence over this setting.\n\nRequires a restart to take effect.\n\nExample: Item `anthropic-api` and Value `Claude Sonnet 4.5` sets the default model for Anthropic to 'Claude Sonnet 4.5'"
 }

extensions/positron-assistant/src/anthropic.ts

@@ -12,6 +12,10 @@ import { DEFAULT_MAX_TOKEN_INPUT, DEFAULT_MAX_TOKEN_OUTPUT } from './constants.j
 import { log, recordTokenUsage, recordRequestTokenUsage } from './extension.js';
 import { TokenUsage } from './tokens.js';
 import { availableModels } from './models.js';
+import { LanguageModelDataPartMimeType } from './types.js';
+
+export const DEFAULT_ANTHROPIC_MODEL_NAME = 'Claude Sonnet 4';
+export const DEFAULT_ANTHROPIC_MODEL_MATCH = 'claude-sonnet-4';
 
 /**
  * Options for controlling cache behavior in the Anthropic language model.
@@ -59,8 +63,8 @@ export class AnthropicLanguageModel implements positron.ai.LanguageModelChatProv
 		},
 		supportedOptions: ['apiKey', 'apiKeyEnvVar'],
 		defaults: {
-			name: 'Claude 3.5 Sonnet v2',
-			model: 'claude-3-5-sonnet-latest',
+			name: DEFAULT_ANTHROPIC_MODEL_NAME,
+			model: DEFAULT_ANTHROPIC_MODEL_MATCH + '-latest',
 			toolCalls: true,
 			apiKeyEnvVar: { key: 'ANTHROPIC_API_KEY', signedIn: false },
 		},
@@ -239,6 +243,17 @@ export class AnthropicLanguageModel implements positron.ai.LanguageModelChatProv
 		return AnthropicLanguageModel.source.provider.displayName;
 	}
 
+	private isDefaultUserModel(id: string, name?: string): boolean {
+		const config = vscode.workspace.getConfiguration('positron.assistant');
+		const defaultModels = config.get<Record<string, string>>('defaultModels') || {};
+		if ('anthropic-api' in defaultModels) {
+			if (id.includes(defaultModels['anthropic-api']) || name?.includes(defaultModels['anthropic-api'])) {
+				return true;
+			}
+		}
+		return id.includes(DEFAULT_ANTHROPIC_MODEL_MATCH);
+	}
+
 	private onContentBlock(block: Anthropic.ContentBlock, progress: vscode.Progress<vscode.ChatResponseFragment2>): void {
 		switch (block.type) {
 			case 'tool_use':
@@ -308,7 +323,6 @@ export class AnthropicLanguageModel implements positron.ai.LanguageModelChatProv
 		const userSetMaxOutputTokens: Record<string, number> = vscode.workspace.getConfiguration('positron.assistant').get('maxOutputTokens', {});
 		let hasMore = true;
 		let nextPageToken: string | undefined;
-		let isFirst = true;
 
 		log.trace(`Fetching models from Anthropic API for provider ${this.provider}`);
 
@@ -334,10 +348,9 @@ export class AnthropicLanguageModel implements positron.ai.LanguageModelChatProv
 					maxInputTokens: maxInputTokens,
 					maxOutputTokens: maxOutputTokens,
 					capabilities: this.capabilities,
-					isDefault: isFirst,
+					isDefault: this.isDefaultUserModel(model.id, model.display_name),
 					isUserSelectable: true,
 				});
-				isFirst = false;
 			});
 
 			hasMore = modelsPage.has_more;
@@ -346,6 +359,26 @@ export class AnthropicLanguageModel implements positron.ai.LanguageModelChatProv
 			}
 		}
 
+		// Mark models as default, ensuring only one default per provider
+		let hasDefault = false;
+		for (let i = 0; i < modelListing.length; i++) {
+			const model = modelListing[i];
+			if (!hasDefault && this.isDefaultUserModel(model.id, model.name)) {
+				modelListing[i] = { ...model, isDefault: true };
+				hasDefault = true;
+			} else {
+				modelListing[i] = { ...model, isDefault: false };
+			}
+		}
+
+		// If no models match the default ID, make the first model the default.
+		if (modelListing.length > 0 && !hasDefault) {
+			modelListing[0] = {
+				...modelListing[0],
+				isDefault: true,
+			};
+		}
+
 		this.modelListing = modelListing;
 
 		return modelListing;
@@ -429,10 +462,12 @@ function toAnthropicUserMessage(message: vscode.LanguageModelChatMessage2, sourc
 				content.push(chatImagePartToAnthropicImageBlock(part, source, dataPart));
 			} else {
 				// Skip other data parts.
-				log.debug(`[anthropic] Skipping unsupported part in user message: ${JSON.stringify(part, null, 2)}`);
+				if (part.mimeType !== LanguageModelDataPartMimeType.CacheControl) {
+					log.debug(`[anthropic] Skipping unsupported part in user message: ${JSON.stringify(part, null, 2)}`);
+				}
 			}
 		} else {
-			throw new Error('Unsupported part type on user message');
+			throw new Error(`Unsupported part type on user message: ${JSON.stringify(part, null, 2)}`);
 		}
 	}
 	return {

extensions/positron-assistant/src/completion.ts

@@ -531,7 +531,7 @@ class AWSCompletion extends FimPromptCompletion {
 		type: positron.PositronLanguageModelType.Completion,
 		provider: {
 			id: 'amazon-bedrock',
-			displayName: 'AWS Bedrock'
+			displayName: 'Amazon Bedrock'
 		},
 		supportedOptions: [],
 		defaults: {

extensions/positron-assistant/src/models.ts

@@ -18,7 +18,7 @@ import { createOpenRouter, OpenRouterProvider } from '@openrouter/ai-sdk-provide
 import { markBedrockCacheBreakpoint, processMessages, toAIMessage } from './utils';
 import { AmazonBedrockProvider, createAmazonBedrock } from '@ai-sdk/amazon-bedrock';
 import { fromNodeProviderChain } from '@aws-sdk/credential-providers';
-import { AnthropicLanguageModel } from './anthropic';
+import { AnthropicLanguageModel, DEFAULT_ANTHROPIC_MODEL_MATCH, DEFAULT_ANTHROPIC_MODEL_NAME } from './anthropic';
 import { DEFAULT_MAX_TOKEN_INPUT, DEFAULT_MAX_TOKEN_OUTPUT } from './constants.js';
 import { log, recordRequestTokenUsage, recordTokenUsage } from './extension.js';
 import { TokenUsage } from './tokens.js';
@@ -347,7 +347,6 @@ abstract class AILanguageModel implements positron.ai.LanguageModelChatProvider2
 		}
 
 		// Return the available models for this provider
-		// The first model is the default model
 		const languageModels: vscode.LanguageModelChatInformation[] = models.map((m, index) => {
 			const modelId = 'identifier' in m ? m.identifier : m.id;
 			const aiModel = this.aiProvider(modelId);
@@ -359,12 +358,19 @@ abstract class AILanguageModel implements positron.ai.LanguageModelChatProvider2
 				maxInputTokens: this.getMaxTokens(aiModel.modelId, 'input'),
 				maxOutputTokens: this.getMaxTokens(aiModel.modelId, 'output'),
 				capabilities: this.capabilities,
-				// is default if it's the first model out of models
-				isDefault: index === 0,
+				isDefault: this.isDefaultUserModel(modelId, m.name),
 				isUserSelectable: true,
 			};
 		});
 
+		// If no models match the default ID, make the first model the default.
+		if (languageModels.length > 0 && !languageModels.some(m => m.isDefault)) {
+			languageModels[0] = {
+				...languageModels[0],
+				isDefault: true,
+			};
+		}
+
 		return languageModels;
 	}
 
@@ -429,8 +435,8 @@ abstract class AILanguageModel implements positron.ai.LanguageModelChatProvider2
 		const modelTools = this._config.toolCalls ? tools : undefined;
 		const requestId = (options.modelOptions as any)?.requestId;
 
-		log.info(`[vercel] Start request ${requestId} to ${this._config.name} [${aiModel.modelId}]: ${aiMessages.length} messages`);
-		log.debug(`[${this._config.name}] SEND ${aiMessages.length} messages, ${modelTools ? Object.keys(modelTools).length : 0} tools`);
+		log.info(`[vercel] Start request ${requestId} to ${model.name} [${aiModel.modelId}]: ${aiMessages.length} messages`);
+		log.debug(`[${model.name}] SEND ${aiMessages.length} messages, ${modelTools ? Object.keys(modelTools).length : 0} tools`);
 		if (modelTools) {
 			log.trace(`tools: ${modelTools ? Object.keys(modelTools).join(', ') : '(none)'}`);
 		}
@@ -456,7 +462,7 @@ abstract class AILanguageModel implements positron.ai.LanguageModelChatProvider2
 		const flushAccumulatedTextDeltas = () => {
 			if (accumulatedTextDeltas.length > 0) {
 				const combinedText = accumulatedTextDeltas.join('');
-				log.trace(`[${this._config.name}] RECV text-delta (${accumulatedTextDeltas.length} parts): ${combinedText}`);
+				log.trace(`[${model.name}] RECV text-delta (${accumulatedTextDeltas.length} parts): ${combinedText}`);
 				accumulatedTextDeltas = [];
 			}
 		};
@@ -468,7 +474,7 @@ abstract class AILanguageModel implements positron.ai.LanguageModelChatProvider2
 
 			if (part.type === 'reasoning') {
 				flushAccumulatedTextDeltas();
-				log.trace(`[${this._config.name}] RECV reasoning: ${part.textDelta}`);
+				log.trace(`[${model.name}] RECV reasoning: ${part.textDelta}`);
 				progress.report({
 					index: 0,
 					part: new vscode.LanguageModelTextPart(part.textDelta)
@@ -485,7 +491,7 @@ abstract class AILanguageModel implements positron.ai.LanguageModelChatProvider2
 
 			if (part.type === 'tool-call') {
 				flushAccumulatedTextDeltas();
-				log.trace(`[${this._config.name}] RECV tool-call: ${part.toolCallId} (${part.toolName}) with args: ${JSON.stringify(part.args)}`);
+				log.trace(`[${model.name}] RECV tool-call: ${part.toolCallId} (${part.toolName}) with args: ${JSON.stringify(part.args)}`);
 				progress.report({
 					index: 0,
 					part: new vscode.LanguageModelToolCallPart(part.toolCallId, part.toolName, part.args)
@@ -494,7 +500,7 @@ abstract class AILanguageModel implements positron.ai.LanguageModelChatProvider2
 
 			if (part.type === 'error') {
 				flushAccumulatedTextDeltas();
-				log.warn(`[${this._config.name}] RECV error: ${JSON.stringify(part.error)}`);
+				log.warn(`[${model.name}] RECV error: ${JSON.stringify(part.error)}`);
 
 				const providerErrorMessage = this.parseProviderError(part.error);
 				if (providerErrorMessage) {
@@ -515,7 +521,7 @@ abstract class AILanguageModel implements positron.ai.LanguageModelChatProvider2
 		result.warnings.then((warnings) => {
 			if (warnings) {
 				for (const warning of warnings) {
-					log.warn(`[${aiModel.modelId}] (${this.id}) warn: ${warning}`);
+					log.warn(`[${aiModel.modelId}] (${this.provider}) warn: ${warning}`);
 				}
 			}
 		});
@@ -545,7 +551,7 @@ abstract class AILanguageModel implements positron.ai.LanguageModelChatProvider2
 			progress.report({ index: 0, part: part });
 
 			// Log the Bedrock usage
-			log.debug(`[${this._config.name}]: Bedrock usage: ${JSON.stringify(usage, null, 2)}`);
+			log.debug(`[${model.name}]: Bedrock usage: ${JSON.stringify(usage, null, 2)}`);
 		}
 
 		if (requestId) {
@@ -605,14 +611,25 @@ abstract class AILanguageModel implements positron.ai.LanguageModelChatProvider2
 					maxInputTokens: 0,
 					maxOutputTokens: model.maxOutputTokens ?? DEFAULT_MAX_TOKEN_OUTPUT,
 					capabilities: this.capabilities,
-					isDefault: false,
+					isDefault: this.isDefaultUserModel(model.identifier, model.name),
 					isUserSelectable: true,
 				} satisfies vscode.LanguageModelChatInformation)));
 			} else {
 				resolve(undefined);
 			}
 		});
 	}
+
+	protected isDefaultUserModel(id: string, name?: string): boolean {
+		const config = vscode.workspace.getConfiguration('positron.assistant');
+		const defaultModels = config.get<Record<string, string>>('defaultModels') || {};
+		if (this.provider in defaultModels) {
+			if (id.includes(defaultModels[this.provider]) || name?.includes(defaultModels[this.provider])) {
+				return true;
+			}
+		}
+		return this._config.model === id;
+	}
 }
 
 class AnthropicAILanguageModel extends AILanguageModel implements positron.ai.LanguageModelChatProvider2 {
@@ -629,8 +646,8 @@ class AnthropicAILanguageModel extends AILanguageModel implements positron.ai.La
 		},
 		supportedOptions: ['apiKey', 'apiKeyEnvVar'],
 		defaults: {
-			name: 'Claude 3.5 Sonnet v2',
-			model: 'claude-3-5-sonnet-latest',
+			name: DEFAULT_ANTHROPIC_MODEL_NAME,
+			model: DEFAULT_ANTHROPIC_MODEL_MATCH + '-latest',
 			toolCalls: true,
 			apiKeyEnvVar: { key: 'ANTHROPIC_API_KEY', signedIn: false },
 		},
@@ -960,17 +977,23 @@ export class AWSLanguageModel extends AILanguageModel implements positron.ai.Lan
 		type: positron.PositronLanguageModelType.Chat,
 		provider: {
 			id: 'amazon-bedrock',
-			displayName: 'AWS Bedrock'
+			displayName: 'Amazon Bedrock'
 		},
 		supportedOptions: ['toolCalls'],
 		defaults: {
-			name: 'Claude 3.5 Sonnet v2 Bedrock',
-			model: 'us.anthropic.claude-3-5-sonnet-20241022-v2:0',
+			name: 'Claude 4 Sonnet Bedrock',
+			model: 'us.anthropic.claude-sonnet-4-20250514-v1:0',
 			toolCalls: true,
 		},
 	};
 
 	constructor(_config: ModelConfig, _context?: vscode.ExtensionContext) {
+		// Update a stale model configuration to the latest defaults
+		const models = availableModels.get('amazon-bedrock')?.map(m => m.identifier) || [];
+		if (!(_config.model in models)) {
+			_config.name = AWSLanguageModel.source.defaults.name;
+			_config.model = AWSLanguageModel.source.defaults.model;
+		}
 		super(_config, _context);
 
 		this.aiProvider = createAmazonBedrock({
@@ -1011,7 +1034,7 @@ export class AWSLanguageModel extends AILanguageModel implements positron.ai.Lan
 			return vscode.l10n.t(`Invalid AWS credentials. {0}`, message);
 		}
 
-		return vscode.l10n.t(`AWS Bedrock error: {0}`, message);
+		return vscode.l10n.t(`Amazon Bedrock error: {0}`, message);
 	}
 }