feat: add LiteLLM engine for 100+ LLM providers

[RheagalFire]

Summary

Adds a LiteLLMEngine that gives kani users access to 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Groq, Ollama, etc.) through the LiteLLM Python SDK.

Changes

• kani/engines/litellm/ -- new engine package
  • engine.py -- LiteLLMEngine extending BaseEngine with predict(), stream(), prompt_len()
  • __init__.py -- exports LiteLLMEngine
• pyproject.toml -- added litellm = ["litellm>=1.80.0,<1.87.0"] optional dependency

Tests

Live E2E against Azure Foundry (Anthropic Claude Sonnet 4.6):

from kani.engines.litellm import LiteLLMEngine
from kani import Kani

engine = LiteLLMEngine(
    model="anthropic/claude-sonnet-4-6",
    max_context_size=200000,
    api_base="https://azure-endpoint.services.ai.azure.com/anthropic",
    api_key="...",
)
ai = Kani(engine)
resp = await ai.chat_round_str("What is 2+2? Reply with just the number.")
# Response: 4

Lint: black and isort pass clean.

Example usage

from kani import Kani
from kani.engines.litellm import LiteLLMEngine

# pip install kani[litellm]

# Use any provider -- model format is provider/model-name
engine = LiteLLMEngine(model="anthropic/claude-haiku-4-5", max_context_size=200000)
# Provider API keys read from environment (e.g. ANTHROPIC_API_KEY)

ai = Kani(engine)
response = await ai.chat_round_str("Hello!")
print(response)

Implementation details

• Translates kani ChatMessage to OpenAI-format dicts for litellm
• Supports tool/function calling (translates AIFunction to OpenAI tool specs, parses ToolCall from responses)
• Streaming via stream() with incremental content and tool call assembly
• Token counting via litellm.token_counter() with fallback estimation
• drop_params=True by default for cross-provider compatibility

Risk / Compatibility

• Additive only -- no existing engines modified
• litellm is an optional dependency (pip install kani[litellm])
• Follows the same engine pattern as OpenAI/Anthropic/Google engines

[RheagalFire]

cc @zhudotexe

[RheagalFire]

@zhudotexe do you have any update on this PR?

[zhudotexe]

Thanks for the PR. I haven't had the time to review it as I'm currently on vacation. Could you add E2E tests in the same vein as the other API-based models (https://github.com/zhudotexe/kani/blob/main/tests/e2e/conftest.py#L466)? The main thing to look out for is to replace the underlying HTTP transport with the AsyncCachingTransport so it saves test data to the E2E cache. Then you might need to update the parameterization of the e2e tests to include the new provider. LMK if you run into any trouble with this and I can provide some pointers.

[zhudotexe]

Is this handled by the litellm SDK?

[zhudotexe]

Is there a way to set this per-request rather than globally? We don't want to step on the user's toes if they're using litellm elsewhere in their application.

[zhudotexe]

Suggested change

	"litellm>=1.80.0,<1.87.0",
	"litellm>=1.80.0,<2.0.0",

We can allow a generous upper bound unless we're using any unstable APIs (which we shouldn't be)

[zhudotexe]

Another consideration: How does LiteLLM handle messages with embedded reasoning/multimodal content? Seems like this impl doesn't handle those especially, but I'm not sure if that's a LiteLLM limitation or not.