Right now the fallback code will always fallback to a provider even if the user hasn't set an API key for it.
We should update it to instead check that the user has an API key set for that provider if it's trying to fallback (account for allowed users)
If they don't have an API key set for that provider, check the next provider.
If there are no other API keys set, simply throw the error and end the run instead of trying to fallback to a provider which there isn't an API key set for.
Agent Context
{
"tasks": [
{
"id": "9e10014c-0b2f-4d34-a39b-77fd32b3c158",
"taskIndex": 0,
"request": "[original issue]\n**Don't fallback to providers with no API key set**\nRight now the fallback code will always fallback to a provider even if the user hasn't set an API key for it.\n\nWe should update it to instead check that the user has an API key set for that provider if it's trying to fallback (account for allowed users)\nIf they don't have an API key set for that provider, check the next provider.\nIf there are no other API keys set, simply throw the error and end the run instead of trying to fallback to a provider which there isn't an API key set for.",
"title": "Fix fallback mechanism to skip providers without API keys",
"createdAt": 1755628908008,
"completed": true,
"planRevisions": [
{
"revisionIndex": 0,
"plans": [
{
"index": 0,
"plan": "**Add API key availability check method to ModelManager** - In `/home/daytona/open-swe/apps/open-swe/src/utils/llms/model-manager.ts`, add a new public method `hasApiKeyForProvider(graphConfig: GraphConfig, provider: Provider): boolean` that checks if a user has a valid API key for a provider without throwing errors. This method should: check if the user is allowed (return true), check if apiKeys exists in configurable, attempt to decrypt the API key using the existing `providerToApiKey` helper and `decryptSecret` function, and return false if any step fails instead of throwing.",
"completed": true,
"summary": "Successfully added the `hasApiKeyForProvider` public method to the ModelManager class in `/home/daytona/open-swe/apps/open-swe/src/utils/llms/model-manager.ts`. \n\nThe method:\n- Takes `graphConfig` and `provider` as parameters\n- Returns a boolean indicating API key availability without throwing errors\n- Checks if the user is in the allowed users list (returns true if allowed)\n- Validates the presence of the SECRETS_ENCRYPTION_KEY and user login\n- Checks if apiKeys exists in the configurable object\n- Uses the existing `providerToApiKey` helper to get the encrypted API key\n- Attempts to decrypt the API key using `decryptSecret`\n- Returns true only if the user is allowed or has a valid, decryptable API key\n- Wraps all logic in a try-catch block to ensure no errors are thrown\n\nThis method provides a safe way to check API key availability for fallback provider filtering without disrupting the application flow with exceptions."
},
{
"index": 1,
"plan": "**Filter fallback providers based on API key availability** - In `/home/daytona/open-swe/apps/open-swe/src/utils/llms/model-manager.ts`, modify the `getModelConfigs()` method (around lines 251-286) to filter out providers without API keys before adding them to the fallback list. Use the new `hasApiKeyForProvider` method to check each provider in the fallback order and only add those with valid API keys to the configs array. Add logging when providers are skipped due to missing API keys.",
"completed": true,
"summary": "Successfully modified the `getModelConfigs()` method in `/home/daytona/open-swe/apps/open-swe/src/utils/llms/model-manager.ts` to filter fallback providers based on API key availability.\n\nKey changes made:\n- Added an API key check at the beginning of the fallback provider loop (line 297)\n- Used the newly created `hasApiKeyForProvider()` method to validate each provider\n- Added a `continue` statement to skip providers without valid API keys\n- Included debug logging to indicate when providers are skipped (lines 298-300)\n- The message clearly states which provider is being skipped and why: \"Skipping fallback provider ${provider} - no API key configured\"\n\nThis ensures that only providers with valid API keys (or allowed users) are included in the fallback configuration list, preventing unnecessary attempts to use providers without proper authentication."
},
{
"index": 2,
"plan": "**Improve error handling in FallbackRunnable** - In `/home/daytona/open-swe/apps/open-swe/src/utils/runtime-fallback.ts`, modify the `invoke()` method (around lines 92-188) to better handle cases where no fallbacks are available. After getting modelConfigs, check if there are valid fallback options (configs.length > 1). If the primary model fails and no fallbacks are available, throw the original error immediately with a clear message indicating that no fallback providers have API keys configured. Add logging to indicate when fallback is limited due to missing API keys.",
"completed": true,
"summary": "Successfully improved error handling in the `FallbackRunnable` class's `invoke()` method in `/home/daytona/open-swe/apps/open-swe/src/utils/runtime-fallback.ts`.\n\nKey improvements made:\n1. **Early validation** (lines 103-107): Added check for empty modelConfigs array, throwing an error if no providers have valid API keys\n2. **Fallback availability logging** (lines 109-117): Added informative logging to indicate when no fallback providers are available vs when multiple models are available\n3. **Primary model failure tracking** (lines 120, 202-204): Added `primaryModelFailed` flag to track when the primary model fails\n4. **Immediate failure on no fallbacks** (lines 207-211): When the primary model fails and no fallbacks are available, the method now throws immediately with a clear message\n5. **Context-specific error messages** (lines 216-220): Improved final error messages to distinguish between single model failure (no fallbacks) and all models exhausted scenarios\n\nThese changes ensure that:\n- Users get clear feedback about why fallback isn't available\n- The system doesn't waste time attempting fallbacks that don't exist\n- Error messages explicitly mention when issues are due to missing API keys\n- The logging provides visibility into the fallback decision process"
},
{
"index": 3,
"plan": "**Build and lint the codebase** - Run `yarn build` from the repository root to build all packages including the shared package, then run `yarn lint:fix` and `yarn format` to ensure code quality and formatting standards are met.",
"completed": true,
"summary": "Successfully built and linted the codebase to ensure all changes meet quality standards.\n\nActions completed:\n1. **Dependency Installation**: Ran `yarn install` to ensure all dependencies were properly installed\n2. **Build Process**: Executed `yarn build` which successfully built all packages including:\n - @open-swe/shared (cache hit)\n - @open-swe/cli \n - @open-swe/web\n - @open-swe/agent (cache hit)\n The build completed with only expected warnings about Edge Runtime compatibility\n\n3. **Linting**: Ran `yarn lint:fix` which completed successfully with only warnings (no errors):\n - React refresh warnings in CLI and web packages\n - Some React hooks dependency warnings\n - All are non-critical warnings that don't affect functionality\n\n4. **Formatting**: Executed `yarn format` which verified all files are properly formatted:\n - All files showed \"(unchanged)\" indicating they already meet formatting standards\n - Prettier and ESLint formatting rules are satisfied\n\nThe codebase is now fully built, linted, and formatted, ensuring the API key fallback mechanism changes are production-ready and meet all code quality standards."
}
],
"createdAt": 1755628908008,
"createdBy": "agent"
}
],
"activeRevisionIndex": 0,
"pullRequestNumber": 779,
"completedAt": 1755629533475,
"summary": "I've successfully implemented the fallback mechanism API key handling by:\n1. Adding a `hasApiKeyForProvider()` method to safely check API key availability\n2. Modifying `getModelConfigs()` to filter providers without valid API keys\n3. Improving error handling in `FallbackRunnable` to provide clear messages about missing API keys\n4. Ensuring the changes pass build, lint, and formatting checks\n\nThe implementation prevents unnecessary fallback attempts and provides better user feedback about provider availability."
}
],
"activeTaskIndex": 0
}
