Auth Server: multi-upstream provider support

[jhrozek]

Overview

This epic tracks the implementation of RFC-0052: adding multi-upstream Identity Provider (IDP) support to the embedded OAuth authorization server (pkg/authserver/). The auth server currently hard-rejects more than one upstream provider configuration; this feature lifts that restriction by adding a providerName dimension to the storage layer and introducing a sequential authorization chain so a single auth server instance can acquire tokens from multiple upstream IDPs in one flow.

Background

Multi-upstream IDP support is a prerequisite for the full vMCP auth story. Without it, backends requiring credentials from different IDPs (e.g., a corporate SSO token for internal services and a GitHub token for public APIs) cannot be served by a single auth server instance. The RFC specifies a server-driven sequential authorization chain, eager token loading into Identity.UpstreamTokens at middleware time, and full backward compatibility for existing single-upstream deployments.

Task Breakdown

Task ID	Title	Depends On	GitHub Issue #
TASK-001	Phase 1: Storage Layer	—	#4136
TASK-002	Phase 2: Handler and Server Layer	TASK-001	#4137
TASK-003	Phase 3: Config and Operator	TASK-001	#4138
TASK-004	Phase 4: Identity Enrichment and upstreamswap Middleware	TASK-001	#4139

Acceptance Criteria

• A single auth server instance accepts and validates configuration with multiple upstream OAuth2/OIDC providers
• Sequential authorization chain drives the user through each configured upstream in turn; the client makes one GET /oauth/authorize and receives one authorization code
• All upstream tokens are eagerly loaded into Identity.UpstreamTokens at auth middleware time
• upstreamswap middleware reads upstream tokens from Identity.UpstreamTokens with no hidden storage calls
• Single-upstream deployments continue to work with no configuration changes required
• Redis breaking change (key pattern) and ProviderIdentity breaking change documented in migration guide
• All linked task issues completed

References

• RFC-0052: https://github.com/stacklok/toolhive-rfcs/blob/main/rfcs/THV-0052-multi-upstream-idp-authserver.md
• Related stacklok-epics issue: https://github.com/stacklok/stacklok-epics/issues/251

[tgrunnagle]

Intake: RFC-0052 Multi-Upstream IDP Support in the Embedded Auth Server

Original Requirements

Implement RFC-0052: Multi-Upstream IDP Support in the Embedded Auth Server. The auth server (pkg/authserver/) currently hard-rejects configuration with more than one upstream Identity Provider (IDP). This RFC lifts that restriction by:

1. Adding a providerName dimension to the UpstreamTokenStorage interface so multiple providers' tokens can coexist per session without overwriting each other.
2. Introducing a sequential authorization chain in the handler layer — the auth server drives the user through each configured upstream IDP in turn (transparent to the OAuth client), accumulating tokens before issuing the final authorization code.
3. Eagerly loading all upstream tokens into Identity.UpstreamTokens at the auth middleware layer so downstream components (upstreamswap middleware, outgoing auth strategies) receive tokens as explicit struct fields — no context extraction, no hidden storage calls.
4. Preserving full backward compatibility for existing single-upstream deployments.

A placeholder epic issue already exists at #3924. One sub-issue must be created per implementation phase (4 total).

Existing Documentation

• RFC-0052 — /Users/trey/Documents/GitHub/toolhive/docs/proposals/THV-0052-multi-upstream-idp-authserver.md — Primary design document. Covers problem statement, proposed solution (high-level and detailed design), API changes, data model changes, sequential authorization chain flow, security considerations, compatibility notes, 4-phase implementation plan, detailed testing strategy, and documentation requirements.
• Epic placeholder issue — Auth Server: multi-upstream provider support #3924 — Existing parent issue; sub-issues for each phase will be linked here.
• Related Stacklok epic — https://github.com/stacklok/stacklok-epics/issues/251

Clarifying Questions & Answers

Issue Style

• Format: Task-based engineering tickets (one per RFC phase, mirroring the RFC's "Implementation Plan" section exactly)
• Rationale: The RFC already defines clear, bounded engineering phases with explicit file-level changes and test requirements. Task-based tickets map directly to those phases and make it easy to track progress and assign work.

Repository & Project

• Source Code Repository: stacklok/toolhive (primary)
• Issue Repository: stacklok/toolhive (same)
• Existing Epic/Parent: Auth Server: multi-upstream provider support #3924
• Local Repository Clones:
  • stacklok/toolhive (primary — issues created here): /Users/trey/Documents/GitHub/toolhive — Main ToolHive service; all code changes for this RFC land here (pkg/authserver/, pkg/auth/, pkg/runner/, cmd/thv-operator/)

Scope & Boundaries

• Expected Outcome: A single embedded auth server instance can be configured with multiple upstream OAuth2/OIDC providers. During a single /oauth/authorize flow the server drives sequential upstream redirects transparently; after all providers are satisfied the authorization code is issued. On each subsequent MCP request, the auth middleware populates Identity.UpstreamTokens (keyed by provider name) so upstreamswap and outgoing auth strategies can inject the correct upstream token for each backend.
• In Scope:
  • UpstreamTokenStorage interface: add providerName to StoreUpstreamTokens and GetUpstreamTokens; add GetAllUpstreamTokens bulk-read method; update DeleteUpstreamTokens to remain session-scoped
  • In-memory storage backend: nested map (sessionID → providerName → entry); implement GetAllUpstreamTokens; update eviction TTL to max(accessToken.ExpiresAt, refreshToken.ExpiresAt)
  • Redis storage backend: new key pattern {prefix}upstream:{sessionID}:{providerName}; session index SET at {prefix}upstream:idx:{sessionID}; Lua scripts for atomic store and delete; SMEMBERS+MGET for GetAllUpstreamTokens
  • PendingAuthorization: add UpstreamProviderName and SessionID fields
  • Handler layer: replace single upstream field with upstreams map[string]upstream.OAuth2Provider + upstreamOrder []string; implement sequential chain in authorize and callback handlers; nextMissingUpstream helper
  • pkg/authserver/config.go: remove len > 1 rejection; add name-uniqueness and explicit-name requirements for multi-upstream
  • Proxy runner (pkg/runner/): add len > 1 upstream rejection to preserve single-upstream invariant for CLI proxy runner
  • Operator admission webhooks (MCPServer, MCPRemoteProxy): move len > 1 restriction from MCPExternalAuthConfig CRD-level validation to workload-specific webhooks, so VirtualMCPServer can reference multi-upstream auth configs
  • Identity struct: add UpstreamTokens map[string]string; add to redaction logic
  • Auth middleware: call GetAllUpstreamTokens after JWT validation; populate Identity.UpstreamTokens
  • upstreamswap middleware: add ProviderName config field; read from Identity.UpstreamTokens instead of calling storage; remove StorageGetter dependency
  • pkg/runner/middleware.go (addUpstreamSwapMiddleware): derive ProviderName from Upstreams[0].Name, falling back to "default"
  • Storage mock regeneration via mockgen
  • All unit, integration, and end-to-end tests called out in the RFC testing strategy
  • Architecture docs and config reference documentation updates
• Out of Scope:
  • Dynamic upstream IDP registration at runtime
  • User account linking across providers
  • Distributed locking for concurrent token refresh operations
  • Full Redis migration tooling for tokens stored under the old key pattern
  • Step-up authentication signaling mechanics
  • Transparent token refresh or re-authorization during middleware enrichment
  • Multi-upstream IDP support in the CLI proxy runner (thv proxy) and corresponding K8s workload types (MCPServer, MCPRemoteProxy)
• Type: Enhancement (extending existing auth server to support a previously blocked configuration)

Technical Context

• Affected Components:
  • pkg/authserver/storage/ — UpstreamTokenStorage interface, in-memory backend, Redis backend, PendingAuthorization types, storage mocks
  • pkg/authserver/server/handlers/ — Handler struct, authorize handler, callback handler, nextMissingUpstream logic
  • pkg/authserver/server_impl.go — upstream map/slice construction from config
  • pkg/authserver/config.go — validation changes
  • pkg/auth/ — Identity struct, auth middleware enrichment
  • pkg/auth/upstreamswap/ — Config, middleware handler
  • pkg/runner/ — proxy runner config validation, addUpstreamSwapMiddleware
  • cmd/thv-operator/api/v1alpha1/ — MCPExternalAuthConfig and workload webhook admission validation
• Known Constraints:
  • SessionID must be generated server-side only; never accepted from client input (security invariant)
  • Each PendingAuthorization leg must use a fresh InternalState for CSRF protection
  • Redis multi-replica deployments must route reads in handleCallback to the primary to avoid replication-lag false restarts (existing go-redis default behavior; do not enable ReadOnly routing)
  • Existing Redis tokens stored under the old key pattern will be invalid after upgrade — documented breaking change requiring re-authentication
  • ProviderIdentity records stored with providerID = "oauth2" will not match after upgrade — documented breaking change
  • Storage and middleware code must never log token values (access tokens, refresh tokens, ID tokens)
  • The proxy runner must continue to enforce exactly one upstream provider per instance
• Target Milestone: None specified in RFC

Dependencies & Integration

• Depends On: None. RFC-0052 explicitly does not depend on UC-06 (step-up auth signaling), which is a separate future effort.
• Downstream Consumers:
  • vMCP deployments — the primary consumer that requires multi-upstream support (backends requiring credentials from different IDPs)
  • upstreamswap middleware — reads Identity.UpstreamTokens after this change instead of calling storage directly
  • Outgoing auth strategies — receive upstream tokens via Identity.UpstreamTokens
• External Integrations:
  • OAuth2/OIDC upstream providers (e.g., corporate SSO, GitHub OAuth) — no changes to the OAuth protocol; changes are server-side only
  • Redis — key pattern changes (breaking) and new Lua scripts for atomic operations

Acceptance & Verification

• Success Criteria:
  • A single auth server instance can be configured with 2+ upstream providers and successfully drives all sequential redirects transparently to the OAuth client
  • Single-upstream deployments continue to work identically with no configuration changes
  • Identity.UpstreamTokens is populated for all upstream providers on every authenticated MCP request
  • upstreamswap middleware injects the correct upstream token without calling storage directly
  • Proxy runner and MCPServer/MCPRemoteProxy operator webhooks continue to reject multi-upstream configurations
  • VirtualMCPServer operator resource can reference an auth config with multiple upstreams
  • All unit, integration, and E2E tests pass
• Test Scenarios (from RFC testing strategy):
  • Storage: multi-provider store/get (two providers per session), session delete wipes both, GetAllUpstreamTokens returns all (including expired) and empty map for unknown session, eviction TTL uses max(access.ExpiresAt, refresh.ExpiresAt)
  • Handler: SessionID generated and threaded through legs, nextMissingUpstream returns correct next, all-satisfied path issues auth code, fresh InternalState per leg, full sequential chain integration test (authorize → leg 1 callback → leg 2 callback → auth code → both tokens present)
  • Config: len > 1 no longer rejected at auth server level; name uniqueness enforced; proxy runner rejects len > 1; operator webhooks for MCPServer/MCPRemoteProxy reject len > 1 but VirtualMCPServer does not
  • Auth middleware: GetAllUpstreamTokens called when tsid claim present; expired tokens included; empty map when no tsid
  • upstreamswap: reads from Identity.UpstreamTokens; returns 401 when token absent; StorageGetter no longer injected
  • Security: replayed state parameter rejected; SessionID not read from client query params
• Stakeholders: RFC author tgrunnagle; Stacklok engineering team

Dry Run Mode

• Enabled: false

Additional Notes

Phase Breakdown

The RFC defines 4 implementation phases with explicit ordering constraints:

• Phase 1: Storage Layer — Foundation phase. Must land first. Changes UpstreamTokenStorage interface, updates both storage backends (memory + Redis), adds PendingAuthorization fields, regenerates mocks, updates all existing call sites within pkg/authserver/. All subsequent phases depend on the new interface signatures.

• Phase 2: Handler and Server Layer — Depends on Phase 1 (new storage interface). Replaces single-upstream handler with multi-upstream map + sequential chain logic. Includes full integration test of the end-to-end chain.

• Phase 3: Config and Operator — Depends on Phase 1 (storage interface must already accept multiple providers for config to allow them). Updates config validation to lift the len > 1 authserver-library restriction, moves the restriction to proxy runner and operator webhooks.

• Phase 4: Identity Enrichment and upstreamswap Middleware — Depends on Phase 1 (GetAllUpstreamTokens must exist). Enriches Identity struct, updates auth middleware and upstreamswap middleware. Can be developed in parallel with Phases 2 and 3 after Phase 1 lands, though sequential ordering is recommended for clarity.

Issue Framing

Each sub-issue should:

• Be labeled as a child/sub-issue of Auth Server: multi-upstream provider support #3924
• Reference the RFC document path: docs/proposals/THV-0052-multi-upstream-idp-authserver.md
• List the exact file changes and test requirements from the RFC's implementation plan section for that phase
• Note dependencies on prior phases explicitly

Breaking Changes

Two documented breaking changes exist and must be called out in each relevant phase issue:

1. Redis key pattern change: existing tokens stored under {prefix}upstream:{sessionID} become unreachable; sessions require re-authentication after upgrade.
2. ProviderIdentity records stored with providerID = "oauth2" will not match post-upgrade; affected users get fresh User records.

[tgrunnagle]

Research: RFC-0052 Multi-Upstream IDP Support in the Embedded Auth Server

Local Repository Clones

• /Users/trey/Documents/GitHub/toolhive — stacklok/toolhive (primary); all code changes for this RFC land here

Reference Documentation

In Repository

• docs/proposals/THV-0052-multi-upstream-idp-authserver.md — Primary RFC. Full problem statement, high-level and detailed design, API changes, data model changes, sequential authorization chain pseudocode, security considerations, compatibility notes, 4-phase implementation plan with file-level change lists, detailed testing strategy table, and documentation requirements.
• docs/arch/11-auth-server-storage.md — Architecture doc for auth server storage. Documents current storage design, Redis Sentinel model, key design (including DeriveKeyPrefix hash-tag format), consistency model (Lua scripts vs. pipelines vs. best-effort commands), TTL table, and CRD configuration shape.
• CONTRIBUTING.md — Contribution guidelines: DCO/Signed-off-by requirement, squash-merge policy, review requirements, testing requirements, PR size limits (~1000 lines).
• CLAUDE.md — Engineering conventions: SPDX license headers, mock generation via go.uber.org/mock (mockgen), business logic in pkg/, CLI as thin wrappers, table-driven tests preferred, lint-fix over lint, conventional commits NOT used, PR titles under 70 chars imperative mood.
• cmd/thv-operator/CLAUDE.md — Operator-specific: after CRD modifications run task operator-generate, task operator-manifests, task crdref-gen (from inside cmd/thv-operator/).
• docs/arch/09-operator-architecture.md — Operator architecture (CRD and controller design).
• docs/arch/10-virtual-mcp-architecture.md — vMCP architecture (primary consumer of multi-upstream feature).

External URLs

• Fosite OAuth2 Framework — OAuth2 library providing fosite.OAuth2Provider, storage interfaces (AuthorizeCodeStorage, AccessTokenStorage, etc.), and compose builder used by the auth server.
• RFC 7636: PKCE — Used for upstream IDP authorization in PendingAuthorization.UpstreamPKCEVerifier/Challenge.
• RFC 8707: Resource Indicators — Governs AllowedAudiences validation for MCP compliance.
• RFC 7591: Dynamic Client Registration — Supported by ClientRegistry interface.

---

Organizational Standards

Coding Conventions

• All Go files require SPDX headers at the top:

  // SPDX-FileCopyrightText: Copyright 2025 Stacklok, Inc.
  // SPDX-License-Identifier: Apache-2.0
  

  Use task license-fix to add them automatically; task license-check to verify.
• Public methods at the top half of Go files; private methods at the bottom half.
• Prefer immutable variable assignment with immediately-invoked anonymous functions over mutable conditionals (see CLAUDE.md example).
• Use fmt.Errorf with %w to preserve error chains; use errors.Is() / errors.As() for inspection.
• Never log token values (access tokens, refresh tokens, ID tokens, credentials).
• Use slog for structured logging; silent success principle (no INFO logging on happy paths).

Review Process

• Every PR requires a review from the core ToolHive team before merging.
• PRs exceeding ~1000 lines may be blocked and require splitting or justification.
• All commits must include a Signed-off-by trailer (DCO).
• Commits are squash-merged with the PR title on approval.
• PR titles: imperative mood, under 70 chars, no trailing period, no conventional commit prefixes (feat:, fix:, etc.).

Documentation Standards

• Architecture docs live in docs/arch/; update when adding or changing components, transports, middleware, CRDs, or RunConfig schema.
• After modifying CRDs in cmd/thv-operator/api/v1alpha1/: run task operator-generate, task operator-manifests, task crdref-gen (from inside cmd/thv-operator/).
• Generate CLI documentation with task docs.

Testing Requirements

• Unit tests alongside source files (*_test.go); prefer table-driven tests.
• Mocks via go.uber.org/mock (mockgen); place in mocks/ subdirectory; regenerate with mockgen.
• Integration tests use //go:build integration build tag; task test excludes them by default.
• Use require.NoError(t, err) (testify) over t.Fatal.
• Tests that write state to disk must use isolated temp directories (t.TempDir() + filepath.EvalSymlinks() on macOS to resolve symlinks).
• Use very large port numbers (near math.MaxInt16) in tests to avoid collisions.
• E2E tests in test/e2e/ test the compiled binary; they are the primary testing strategy for CLI commands.

---

Architecture & Design

System Architecture

The embedded OAuth authorization server (pkg/authserver/) is a full OAuth2/OIDC AS built on the Fosite framework. It sits in front of upstream Identity Providers and issues its own JWTs (TH-JWTs) that include a tsid (token session ID) claim linking the JWT to stored upstream tokens.

Current single-upstream flow:

1. OAuth client calls GET /oauth/authorize.
2. Auth server validates client, generates PendingAuthorization (with InternalState for CSRF and UpstreamPKCEVerifier/UpstreamNonce), redirects user to the single configured upstream IDP.
3. Upstream IDP calls back to GET /oauth/callback with a code.
4. Auth server exchanges code, stores UpstreamTokens keyed by sessionID, resolves/creates the internal User, issues fosite authorization code, redirects to client's redirect_uri.
5. Client exchanges code for TH-JWT access token (includes tsid claim) via POST /oauth/token.
6. On each MCP request, auth middleware validates TH-JWT; upstreamswap middleware uses tsid to call InProcessService.GetValidTokens(ctx, sessionID) (storage lookup with transparent refresh via UpstreamTokenRefresher), injects upstream access token into outgoing request.

RFC-0052 change — sequential multi-upstream chain:

1. Client calls GET /oauth/authorize (same as before — single request).
2. Auth server generates SessionID (the TSID for this chain), redirects to upstreamOrder[0].
3. Each callback stores tokens for pending.UpstreamProviderName, calls GetAllUpstreamTokens(ctx, sessionID) to determine nextMissingUpstream, either redirects to the next upstream or issues the authorization code.
4. Client makes exactly one GET /oauth/authorize and receives exactly one authorization code; all intermediate upstream redirects are transparent.
5. On MCP requests, auth middleware calls GetAllUpstreamTokens once (bulk read) and populates Identity.UpstreamTokens map[string]string (providerName → access token). upstreamswap reads identity.UpstreamTokens[cfg.ProviderName] directly — no storage call, no context extraction.

Established Patterns

• Interface Segregation: fosite splits storage into AuthorizeCodeStorage, AccessTokenStorage, RefreshTokenStorage, TokenRevocationStorage, PKCERequestStorage; ToolHive adds UpstreamTokenStorage, PendingAuthorizationStorage, ClientRegistry, UserStorage. The composite Storage interface embeds all of them (pkg/authserver/storage/types.go).
• Factory Pattern: middleware factories registered in pkg/runner/middleware.go:GetSupportedMiddlewareFactories() with string-keyed map; each factory receives *types.MiddlewareConfig + types.MiddlewareRunner.
• Defensive Copies: Both memory and Redis storage backends make defensive copies on read and write to prevent caller mutations from affecting stored data (all UpstreamTokens copies in memory.go:StoreUpstreamTokens and GetUpstreamTokens).
• Lua Scripts for Atomicity: Redis storage uses Lua scripts for operations requiring atomic key + index updates (upstream token store, last-used timestamp update, session delete). Pipelines (MULTI/EXEC) for batched non-atomic operations. Individual commands with best-effort cleanup for revocation.
• Sentinel Errors: storage.ErrNotFound, storage.ErrExpired, storage.ErrAlreadyExists, storage.ErrInvalidBinding — always check with errors.Is().
• RunConfig Pattern: serializable, versioned, portable config structs (no secrets inline — file paths or env var names resolved at runtime). Mirrors pkg/runner.RunConfig pattern.
• CEL Validation + Runtime Defense-in-Depth: CRD types use kubebuilder CEL validation markers (+kubebuilder:validation:XValidation) as the admission-time layer, plus Validate() methods on the types for runtime defense-in-depth. MCPExternalAuthConfig is the exemplar of this pattern (see cmd/thv-operator/api/v1alpha1/mcpexternalauthconfig_types.go).

Prior Decisions

• tsid claim name: Uses "tsid" instead of "sid" to avoid confusion with OIDC Session Management spec (pkg/auth/upstreamtoken/types.go:TokenSessionIDClaimKey).
• Redis Sentinel only: Only Sentinel mode is supported (no standalone Redis, no Redis Cluster). Hash tag format {ns:name} in key prefix ensures slot co-location if ever migrated to Cluster, but has no functional effect in Sentinel mode.
• ACL authentication only: Legacy requirepass not supported. Redis ACL users (Redis 6+) provide fine-grained access control.
• Storage mock generated from types.go interface: mockgen -source=types.go generates pkg/authserver/storage/mocks/mock_storage.go. Must regenerate whenever the Storage interface or any sub-interface changes.
• Single-upstream restriction: Currently enforced at two layers — (1) pkg/authserver/config.go:validateUpstreams() with if len(c.Upstreams) > 1 and (2) cmd/thv-operator/api/v1alpha1/mcpexternalauthconfig_types.go:787-788 inside validateEmbeddedAuthServer(). RFC-0052 removes (1) from the library layer and moves the restriction to the consumer layers (proxy runner and operator workload types).
• InProcessService with singleflight: Token refresh operations are deduplicated via golang.org/x/sync/singleflight to prevent concurrent refresh races for the same session. Refresh timeout is 30s, detached from the triggering request context (pkg/auth/upstreamtoken/service.go).
• Eviction TTL in memory storage: Current TTL = tokens.ExpiresAt + DefaultRefreshTokenTTL. RFC-0052 changes this to max(accessToken.ExpiresAt, refreshToken.ExpiresAt) to prevent premature eviction of tokens that still have a valid refresh token.

API Contracts (Interfaces Affected by RFC-0052)

Current UpstreamTokenStorage interface (pkg/authserver/storage/types.go):

StoreUpstreamTokens(ctx context.Context, sessionID string, tokens *UpstreamTokens) error
GetUpstreamTokens(ctx context.Context, sessionID string) (*UpstreamTokens, error)
DeleteUpstreamTokens(ctx context.Context, sessionID string) error

Post-RFC-0052 UpstreamTokenStorage interface (from RFC):

StoreUpstreamTokens(ctx context.Context, sessionID, providerName string, tokens *UpstreamTokens) error
GetUpstreamTokens(ctx context.Context, sessionID, providerName string) (*UpstreamTokens, error)
GetAllUpstreamTokens(ctx context.Context, sessionID string) (map[string]*UpstreamTokens, error)
DeleteUpstreamTokens(ctx context.Context, sessionID string) error

Current PendingAuthorization struct (relevant fields): ClientID, RedirectURI, State, PKCEChallenge, PKCEMethod, Scopes, InternalState, UpstreamPKCEVerifier, UpstreamNonce, CreatedAt.
New fields: UpstreamProviderName string and SessionID string.

Identity struct (pkg/auth/identity.go) — new field to add: UpstreamTokens map[string]string (providerName → access token). Must also be added to MarshalJSON() redaction logic (token values must be redacted, though provider names are safe to log).

upstreamswap.Config (pkg/auth/upstreamswap/middleware.go) — new field: ProviderName string. The ServiceGetter type and its use in CreateMiddleware and createMiddlewareFunc will be removed.

Known Constraints

• SessionID in PendingAuthorization is generated server-side by the authorize handler and stored only server-side. It must never be accepted from any client-supplied query parameter or form field.
• Each chain leg must generate a fresh InternalState for the upstream state parameter (independent CSRF protection per redirect). The used PendingAuthorization must be deleted before creating the next leg's pending entry.
• Redis handleCallback reads (GetAllUpstreamTokens for nextMissingUpstream) must go to the primary — do not enable ReadOnly routing on the go-redis client. Standard go-redis failover client routes reads to master by default; this must not be changed.
• Token values (access tokens, refresh tokens, ID tokens) must never appear in log output, error messages, or HTTP responses.
• Storage operations must validate providerName is non-empty at entry points.
• The proxy runner must continue to enforce exactly one upstream per server instance. This invariant moves from the library layer (pkg/authserver/config.go) to the consumer layer (pkg/runner/).

---

Codebase Analysis

Relevant Directories

• pkg/authserver/storage/ — UpstreamTokenStorage interface (types.go), in-memory backend (memory.go), Redis backend (redis.go), Redis key helpers (redis_keys.go), storage config (config.go), mocks (mocks/mock_storage.go).
• pkg/authserver/server/handlers/ — Handler struct (handler.go), AuthorizeHandler (authorize.go), CallbackHandler (callback.go), TokenHandler (token.go), UserResolver (user.go), discovery/JWKS (discovery.go).
• pkg/authserver/server/ — AuthorizationServerConfig (provider.go); session/ — fosite session type.
• pkg/authserver/ — Config/RunConfig/UpstreamRunConfig structs and validateUpstreams() (config.go); server_impl.go — newServer() and createProvider(); server.go — Server interface.
• pkg/authserver/upstream/ — OAuth2Provider interface and implementations (OIDCProviderImpl, BaseOAuth2Provider).
• pkg/auth/ — Identity struct (identity.go); TokenValidator.Middleware and claimsToIdentity (token.go); GetAuthenticationMiddleware (utils.go); context.go (context helpers WithIdentity/IdentityFromContext).
• pkg/auth/upstreamswap/ — Middleware, Config, ServiceGetter, createMiddlewareFunc (middleware.go).
• pkg/auth/upstreamtoken/ — Service interface and TokenSessionIDClaimKey (types.go); InProcessService.GetValidTokens with singleflight refresh (service.go); sentinel errors (errors.go).
• pkg/runner/ — GetSupportedMiddlewareFactories() and PopulateMiddlewareConfigs() (middleware.go); Runner.GetUpstreamTokenService() and embedded auth server wiring (runner.go); RunConfig struct (includes EmbeddedAuthServerConfig, UpstreamSwapConfig).
• pkg/transport/types/ — MiddlewareRunner interface (includes GetUpstreamTokenService()) (transport.go); mocks (mocks/mock_transport.go).
• cmd/thv-operator/api/v1alpha1/ — MCPExternalAuthConfig CRD types including EmbeddedAuthServerConfig, UpstreamProviderConfig, AuthServerStorageConfig (mcpexternalauthconfig_types.go); MCPServer types with ExternalAuthConfigRef (mcpserver_types.go); MCPRemoteProxy types with ExternalAuthConfigRef (mcpremoteproxy_types.go).
• cmd/thv-operator/pkg/controllerutil/ — authserver.go (converts CRD EmbeddedAuthServerConfig to authserver.RunConfig; calls validateUpstreamProviders — this is the controller-side validation entry point).

Similar Implementations (Reference Patterns)

• pkg/authserver/storage/memory.go lines 685–777 — Current StoreUpstreamTokens, GetUpstreamTokens, DeleteUpstreamTokens. Shows defensive copy pattern, timedEntry wrapper, eviction TTL calculation, and error wrapping pattern. Phase 1 will transform upstreamTokens map[string]*timedEntry[*UpstreamTokens] to a nested map map[string]map[string]*timedEntry[*UpstreamTokens].
• pkg/authserver/storage/redis.go — Lua script patterns, SADD/SMEMBERS set-index pattern, JSON serialization, nullMarker for nil values, warnOnCleanupErr for best-effort secondary index cleanup.
• pkg/authserver/storage/redis_keys.go — Key generation helpers redisKey, redisProviderKey, redisSetKey. Phase 1 adds redisUpstreamKey(prefix, sessionID, providerName string).
• cmd/thv-operator/api/v1alpha1/mcpexternalauthconfig_types.go — validateEmbeddedAuthServer() and validateUpstreamProvider() are the exemplar pattern for CRD-level Validate() methods. Phase 3 adds a Validate() method (or a validation helper) on MCPServer and MCPRemoteProxy types following this same pattern.
• pkg/auth/upstreamswap/middleware.go — Current createMiddlewareFunc pattern: extracts tsid from identity.Claims, calls serviceGetter() at request time, calls svc.GetValidTokens(ctx, tsid). Phase 4 replaces this with a direct identity.UpstreamTokens[cfg.ProviderName] read.

Key Call Sites for Phase 4 (Identity Enrichment)

JWT validation + Identity construction — pkg/auth/token.go, TokenValidator.Middleware method (around line 1067):

func (v *TokenValidator) Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // ... extract and validate bearer token ...
        claims, err := v.ValidateToken(r.Context(), tokenString)
        // ...
        identity, err := claimsToIdentity(claims, tokenString)  // line ~1087
        // ...
        // NEW: call GetAllUpstreamTokens here if tsid claim is present
        // and populate identity.UpstreamTokens before:
        ctx := WithIdentity(r.Context(), identity)              // line ~1096
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

The GetAllUpstreamTokens call must be inserted between claimsToIdentity (line ~1087) and WithIdentity (line ~1096). The TokenValidator struct will need access to storage.UpstreamTokenStorage — this requires either adding a field to TokenValidator or threading storage through GetAuthenticationMiddleware in pkg/auth/utils.go.

GetAuthenticationMiddleware wiring — pkg/auth/utils.go line 64: creates NewTokenValidator(ctx, *oidcConfig) and returns jwtValidator.Middleware. The UpstreamTokenStorage dependency for enrichment needs to be threaded through here.

CreateMiddleware (auth factory) — pkg/auth/middleware.go line 48: called from PopulateMiddlewareConfigs in pkg/runner/middleware.go. Receives types.MiddlewareRunner; the runner has access to IDPTokenStorage() via r.embeddedAuthServer.IDPTokenStorage().

GetUpstreamTokenService() removal from upstreamswap — pkg/transport/types/transport.go line 86: GetUpstreamTokenService() func() upstreamtoken.Service is on the MiddlewareRunner interface. After Phase 4, this method may no longer be needed by upstreamswap. Assess whether it is still needed by any other middleware before removing.

Test Patterns

• pkg/authserver/storage/memory_test.go — Unit tests for memory storage; test helpers use NewMemoryStorage(). Pattern: call storage methods directly, check return values and errors.
• pkg/authserver/storage/redis_test.go — Unit tests using miniredis (in-process Redis mock); redis_integration_test.go uses //go:build integration with real Redis via testcontainers.
• pkg/authserver/server/handlers/helpers_test.go — Shared test fixtures for handler tests: testStorageState, mock upstream provider implementations, test token helpers.
• pkg/authserver/integration_test.go — Full end-to-end integration test of the auth server (authorize → callback → token → validation). Phase 2 adds a multi-upstream sequential chain test here.
• pkg/auth/upstreamswap/middleware_test.go — Tests CreateMiddleware factory and createMiddlewareFunc; uses MockMiddlewareRunner from pkg/transport/types/mocks/.

File Naming Conventions

• Storage mocks: pkg/authserver/storage/mocks/mock_storage.go — regenerated via mockgen -destination=mocks/mock_storage.go -package=mocks -source=types.go [interface names]
• Transport/runner mocks: pkg/transport/types/mocks/mock_transport.go
• Type files in cmd/thv-operator/api/v1alpha1/ follow <resourcename>_types.go convention.

---

Related Work

Open Issues

• #3924 — Placeholder epic for RFC-0052 implementation; all phase sub-issues are children of this issue.
• stacklok-epics#251 — Related Stacklok-level epic.

Known Tech Debt

• Secondary index cleanup in Redis storage is best-effort (stale entries are bounded by TTL but can accumulate). A background cleanup job is noted as a TODO in pkg/authserver/storage/redis.go:warnOnCleanupErr.
• TokenValidator currently has no access to UpstreamTokenStorage; threading this dependency for Phase 4 requires a small refactor of TokenValidator, GetAuthenticationMiddleware, and the auth.CreateMiddleware factory.
• GetUpstreamTokenService() on MiddlewareRunner interface exists solely for upstreamswap. After Phase 4 this dependency is removed from upstreamswap; assess whether the interface method should be retained or removed (requires updating the mock).

---

Gaps & Open Questions

• TokenValidator storage dependency (Phase 4): The TokenValidator struct in pkg/auth/token.go currently takes no storage dependency. To call GetAllUpstreamTokens after JWT validation, either: (a) add UpstreamTokenStorage as an optional field on TokenValidator (nil = no enrichment, consistent with non-embedded-auth-server deployments), or (b) add a post-validation enrichment hook. The RFC does not prescribe the exact wiring; this is an implementation decision for the Phase 4 planner. Option (a) is preferred given the existing patterns in the codebase.

• GetUpstreamTokenService() interface retention (Phase 4): After upstreamswap no longer calls storage directly, GetUpstreamTokenService() on MiddlewareRunner may have no remaining callers. If removed, pkg/transport/types/transport.go interface and pkg/transport/types/mocks/mock_transport.go must be updated. Confirm no other middleware depends on it before removing.

• Operator workload-type Validate() methods (Phase 3): Neither MCPServer nor MCPRemoteProxy currently has a Validate() method in their *_types.go files. Phase 3 must add validation to both types (following the MCPExternalAuthConfig.Validate() pattern) that rejects embeddedAuthServer auth configs referencing more than one upstream provider. The validation needs to resolve the referenced MCPExternalAuthConfig — clarify whether this validation lives in the CRD Validate() method (without cluster access) or in the controller's reconcile loop (with cluster access to fetch the referenced config). Given that MCPExternalAuthConfig.Validate() validates the config itself and the MCPServer/MCPRemoteProxy only hold a reference, the controller's reconcile logic (similar to how it checks ConditionTypeExternalAuthConfigValidated) is the more practical location.

• ProviderIdentity provider ID after RFC-0052: Currently the callback stores providerID = string(h.upstream.Type()) which resolves to "oidc" or "oauth2". After RFC-0052, it should store providerID = pending.UpstreamProviderName (the configured name, e.g., "github", "corporate-idp"). This is a documented breaking change for existing Redis deployments. The Phase 2 handler implementation must use pending.UpstreamProviderName not upstream.Type() as the providerID in h.userResolver.ResolveUser.