feat(ts-sdk): expose llmProvider for bring-your-own-provider injection#115
Open
adnanrhussain wants to merge 3 commits into
Open
feat(ts-sdk): expose llmProvider for bring-your-own-provider injection#115adnanrhussain wants to merge 3 commits into
adnanrhussain wants to merge 3 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
adnanrhussain
force-pushed
the
ahussain/ts-llm-provider
branch
from
19fbe25 to
a668fc8
Compare
There was a problem hiding this comment.
Pull request overview
Adds a bring-your-own-provider injection point to the TypeScript SDK so evaluators (including batch) can route all LLM calls through a caller-supplied LLMProvider, bypassing the built-in OpenAI/Google/Anthropic API-key adapters.
Changes:
- Exposed
llmProvider?: LLMProvideronBaseEvaluatorConfigand threaded it into provider creation logic (with amodelOverrideconflict guard). - Exposed
llmProvider?: LLMProvideronBatchConfigand plumbed it throughBatchEvaluatorevaluator instantiation. - Added unit tests covering no-key construction, call routing, ambient-key tolerance, and the
llmProvidervsmodelOverrideconflict.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| sdks/typescript/src/evaluators/base.ts | Adds llmProvider to evaluator config, skips key validation when set, and returns injected provider from createConfiguredProvider(). |
| sdks/typescript/src/batch/types.ts | Adds llmProvider to batch config type so batches can share a custom provider. |
| sdks/typescript/src/batch/evaluator.ts | Passes llmProvider through when constructing per-evaluator instances in a batch. |
| sdks/typescript/tests/unit/evaluators/llm-provider.test.ts | Adds unit coverage for injection routing and configuration interactions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+212
to
+215
| // An injected llmProvider replaces the built-in adapters entirely. | ||
| if (config.llmProvider) { | ||
| this.validateLlmProvider(config.llmProvider); | ||
|
|
| maxRetries: this.config.maxRetries, | ||
| telemetry: this.config.telemetry, | ||
| modelOverride: this.config.modelOverride, | ||
| llmProvider: this.config.llmProvider, |
adnanrhussain
force-pushed
the
ahussain/ts-llm-provider
branch
from
affc642 to
ee40021
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
llmProvidertoBaseEvaluatorConfigandBatchConfig. Inject any object implementing the publicLLMProviderinterface and every evaluator routes its LLM calls through it, skipping the built-in OpenAI/Google/Anthropic API-key adapters. Lets you run the evaluators on a provider you already have wired — Vertex AI, Bedrock, an AI-SDK gateway — with no separate API keys.Addresses #106.
Behavior
llmProvideris set, API-key validation is skipped; the injected provider manages its own model, auth, and retries.llmProviderandmodelOverrideare mutually exclusive — setting both throwsConfigurationError.llmProvideris set.Usage
Tests
tests/unit/evaluators/llm-provider.test.ts: no-key construction, routing through the injected provider, ambient-key tolerance, and themodelOverrideconflict throw. typecheck / lint / 245 unit tests / build all pass.