Skip to content

Phase 4: Operator reconciler, HTTP handler unit tests, and E2E test coverage for embedded AS #4143

New issue
New issue
Open
Open
Phase 4: Operator reconciler, HTTP handler unit tests, and E2E test coverage for embedded AS#4143
Labels
authenticationauthorizationenhancementNew feature or requestgoPull requests that update go codekubernetesItems related to KubernetesoperatortestingvmcpVirtual MCP Server related issues
@tgrunnagle

Description

@tgrunnagle
tgrunnagle
opened on Mar 13, 2026

Description

Wire the embedded authorization server into the Kubernetes operator and add comprehensive test coverage — unit tests for the vMCP HTTP handler, CLI E2E tests for Mode A/Mode B routing and negative validation, and operator E2E tests verifying the AuthServerConfigValid status condition lifecycle. This is the final phase of RFC-0053 that closes the loop between the CRD API, the operator reconciler, the config converter, and observable runtime behavior.

Context

RFC-0053 adds an optional embedded OAuth/OIDC authorization server to vMCP. Phase 1 (#4140) established structural types and CRD fields. Phase 2 (#4141) mounted the AS HTTP routes on the vMCP mux. Phase 3 (#4142) added startup validation rules V-01 through V-07. Phase 4 (this task) completes the Kubernetes operator path: the reconciler resolves spec.authServerConfigRef, performs cross-resource validation, surfaces the AuthServerConfigValid status condition, and the config converter converts the referenced MCPExternalAuthConfig into the AuthServerConfig that vMCP reads at startup. This phase also delivers all deferred test coverage and documentation updates.

Parent epic: #4120 — vMCP: add embedded authorization server
RFC document: docs/proposals/THV-0053-vmcp-embedded-authserver.md
Dependencies: #4141 (Phase 2: Server wiring), #4142 (Phase 3: Startup validation)
Blocks: nothing — this is the final phase. Full end-to-end token flow testing depends on RFC-0052 (#3924) for identity.UpstreamTokens population.

Acceptance Criteria

Operator Reconciler

  • cmd/thv-operator/controllers/virtualmcpserver_controller.go runValidations() includes a new validateAuthServerConfigRef step that, when spec.authServerConfigRef is non-nil: fetches the referenced MCPExternalAuthConfig from the same namespace; verifies its spec.type is "embeddedAuthServer" (surfacing a Failed phase with descriptive message if not); verifies spec.embeddedAuthServer.issuer matches spec.incomingAuth.oidcConfig.inline.issuer (V-04 check, surfacing Failed phase with message containing "issuer" and "mismatch"); and verifies that the audience derived from spec.incomingAuth.oidcConfig.inline.audience is non-empty (defense-in-depth empty-audience guard)
  • A new ConditionTypeAuthServerConfigValid = "AuthServerConfigValid" constant is defined in cmd/thv-operator/api/v1alpha1/virtualmcpserver_types.go
  • On successful authServerConfigRef validation, statusManager.SetCondition(ConditionTypeAuthServerConfigValid, "AuthServerConfigValid", "Auth server config is valid", metav1.ConditionTrue) is called
  • On validation failure, statusManager.SetCondition(ConditionTypeAuthServerConfigValid, "AuthServerConfigInvalid", <descriptive message>, metav1.ConditionFalse) is called and the phase is set to Failed; reconciliation does not proceed to ensureAllResources
  • When spec.authServerConfigRef is nil (Mode A), validateAuthServerConfigRef is skipped entirely and no AuthServerConfigValid condition is set

Config Converter

  • cmd/thv-operator/pkg/vmcpconfig/converter.go Convert() includes a new convertAuthServerConfig step: when vmcp.Spec.AuthServerConfigRef != nil, fetches the MCPExternalAuthConfig, verifies type is "embeddedAuthServer", converts the EmbeddedAuthServerConfig to authserver.RunConfig, derives allowedAudiences from vmcp.Spec.IncomingAuth.OIDCConfig.Inline.Audience, sets config.AuthServer = &vmcpconfig.AuthServerConfig{RunConfig: runConfig}, and returns an error on any failure
  • When spec.authServerConfigRef is nil (Mode A), config.AuthServer remains nil (unchanged from the deep-copied spec config)
  • The converter derives allowedAudiences from spec.incomingAuth.oidcConfig.inline.audience and does NOT read any allowedAudiences field from MCPExternalAuthConfig (the CRD intentionally omits that field)

HTTP Handler Unit Tests

  • pkg/vmcp/server/server_test.go (or pkg/vmcp/server/handler_test.go) contains unit tests for the server Handler() method covering Mode A routing: GET /.well-known/oauth-protected-resource returns HTTP 200; GET /.well-known/openid-configuration returns HTTP 404; GET /.well-known/oauth-authorization-server returns HTTP 404; GET /.well-known/jwks.json returns HTTP 404; GET /oauth/token returns HTTP 404
  • Mode B routing tests: GET /.well-known/openid-configuration is served by the AS handler (HTTP 200 with JSON body); GET /.well-known/oauth-authorization-server is served by the AS handler (HTTP 200); GET /.well-known/jwks.json is served by the AS handler (HTTP 200); GET /oauth/authorize is served by the AS handler (not 404)
  • Both Mode A and Mode B: GET /.well-known/oauth-protected-resource returns HTTP 200 (explicit registration, not catch-all)
  • Both Mode A and Mode B: GET /mcp without a bearer token returns HTTP 401 when AuthMiddleware is configured

vMCP CLI E2E Tests

  • test/e2e/vmcp_authserver_test.go exists and is in package e2e_test; it joins the existing Ginkgo suite in test/e2e/e2e_suite_test.go automatically
  • Positive — Mode B startup: start vMCP with a Mode B YAML config (using OIDCMockServer as the upstream IDP); GET /.well-known/openid-configuration returns HTTP 200 with a JSON body containing a non-empty issuer field matching the configured AS issuer
  • Positive — Mode B 401: GET /mcp (or configured endpoint path) without a bearer token returns HTTP 401 (auth middleware active)
  • Positive — Mode A: start vMCP with no authServer block; GET /.well-known/openid-configuration returns HTTP 404; GET /.well-known/oauth-protected-resource returns HTTP 200
  • Negative — V-04 issuer mismatch: start vMCP with authServer.runConfig.issuer different from incomingAuth.oidc.issuer; process exits non-zero; stderr contains a string matching "issuer" and "mismatch" (or the V-04 message text)
  • Negative — V-01 upstream_inject without AS: start vMCP with an upstream_inject backend auth strategy but no authServer block; process exits non-zero; stderr contains a string referencing V-01 or "upstream_inject" and "authServer"
  • Negative — V-02 unknown provider: start vMCP with an upstream_inject strategy whose providerName does not match any upstream in authServer.runConfig.upstreams; process exits non-zero; stderr contains a string referencing the unknown provider name

Operator E2E Tests

  • test/e2e/thv-operator/virtualmcp/virtualmcp_authserver_test.go exists in package virtualmcp; it joins the existing Ginkgo suite in test/e2e/thv-operator/virtualmcp/suite_test.go
  • Test creates an MCPExternalAuthConfig with spec.type: embeddedAuthServer and a valid spec.embeddedAuthServer configuration (with an upstream provider pointing to an in-cluster mock OIDC server or a test fixture)
  • Test creates a VirtualMCPServer with spec.authServerConfigRef.name pointing to the above MCPExternalAuthConfig; spec.incomingAuth.oidcConfig.inline.issuer matches the embeddedAuthServer.issuer
  • Eventually assertion verifies the AuthServerConfigValid condition transitions to True (status ConditionTrue) within the test timeout
  • Deployment is created and becomes available (following the pattern in virtualmcp_external_auth_test.go)
  • Negative test: create a VirtualMCPServer with an issuer mismatch; Eventually assertion verifies the AuthServerConfigValid condition transitions to False with a message containing "issuer"; phase is Failed; no Deployment is created for this VirtualMCPServer

Documentation

  • docs/arch/09-operator-architecture.md is updated to document the new AuthServerConfigRef field on VirtualMCPServerSpec, the validateAuthServerConfigRef reconciliation step, and the AuthServerConfigValid status condition
  • docs/arch/02-core-concepts.md is updated to describe the Mode A / Mode B distinction for vMCP auth (embedded AS vs. external IDP)
  • A new vMCP auth server guide is added under docs/ (e.g., docs/vmcp-auth-server.md) covering: config walkthrough for both CLI YAML and Kubernetes CRD paths; example YAML snippets; explanation of the jwksAllowPrivateIP: true requirement for in-cluster loopback OIDC discovery; troubleshooting the self-referencing OIDC discovery setup
  • task docs is run to regenerate CRD reference documentation after any CRD changes

General

  • All new Go files include // SPDX-FileCopyrightText: Copyright 2025 Stacklok, Inc. and // SPDX-License-Identifier: Apache-2.0 headers
  • task lint passes (or task lint-fix resolves all linting issues)
  • task test passes (unit tests)
  • All existing tests continue to pass (Mode A configs with nil AuthServer and nil AuthServerConfigRef must not be affected)

Technical Approach

Recommended Implementation

Work in four coordinated steps:

Step 1 — Reconciler validation (cmd/thv-operator/controllers/virtualmcpserver_controller.go): Add a validateAuthServerConfigRef method following the same structure as validateCompositeToolRefs and validateGroupRef. It is called inside runValidations() only when vmcp.Spec.AuthServerConfigRef != nil. The method uses r.Get to fetch the MCPExternalAuthConfig by name and namespace. After verifying spec.type == "embeddedAuthServer", perform the V-04 issuer consistency check (exact string comparison — no URL normalization). Use statusManager.SetCondition(ConditionTypeAuthServerConfigValid, ...) to surface results; on failure also call statusManager.SetPhase(VirtualMCPServerPhaseFailed) and statusManager.SetMessage(...). On validation failure, return (false, nil) from runValidations to stop reconciliation without requeueing (user must fix spec).

Step 2 — Converter step (cmd/thv-operator/pkg/vmcpconfig/converter.go): Add a convertAuthServerConfig private method that accepts ctx, vmcp, and the resolved MCPExternalAuthConfig. It converts EmbeddedAuthServerConfig fields to an authserver.RunConfig, derives allowedAudiences from vmcp.Spec.IncomingAuth.OIDCConfig.Inline.Audience, and returns *vmcpconfig.AuthServerConfig. Call this method from Convert() after the existing convertIncomingAuth block, gated on vmcp.Spec.AuthServerConfigRef != nil. The converter must also resolve the MCPExternalAuthConfig by calling c.k8sClient.Get (see existing convertBackendAuthConfig pattern for the Get call structure).

Step 3 — Unit tests: Add pkg/vmcp/server/server_test.go (or extend the existing file) with table-driven tests for Handler(). Construct a minimal Config with a mock AuthMiddleware (using httptest.NewRecorder and httptest.NewRequest). For Mode B tests, create an EmbeddedAuthServer via runner.NewEmbeddedAuthServer(ctx, runConfig) with a minimal dev-mode config (ephemeral keys, in-memory storage). After each test, defer as.Close(). Use httptest.NewServer(handler) to get a testable HTTP server. Assert HTTP status codes for each path.

Step 4 — E2E tests: For CLI E2E tests in test/e2e/vmcp_authserver_test.go, follow test/e2e/proxy_oauth_test.go exactly — Describe + BeforeEach/AfterEach, By() annotations, Eventually for async assertions. Start a OIDCMockServer as the upstream IDP and write YAML config files to t.TempDir(). For negative tests, start the vMCP binary and assert non-zero exit code and stderr content using exec.Command. For operator E2E tests in test/e2e/thv-operator/virtualmcp/virtualmcp_authserver_test.go, follow virtualmcp_external_auth_test.go — use k8sClient.Create, Eventually with k8sClient.Get to assert condition status.

Patterns & Frameworks

  • runValidations pattern: Follow the exact structure of validateCompositeToolRefs in virtualmcpserver_controller.go — return (false, nil) on user-fixable spec errors (to stop reconciliation without requeueing), return (false, err) for transient errors that should trigger requeue (e.g., API server unavailable)
  • Status condition surfacing: Use statusManager.SetCondition(conditionType, reason, message, status) directly (as in validateAndUpdatePodTemplateStatus) rather than a specialized helper, since AuthServerConfigValid does not warrant a dedicated method on StatusManager
  • Converter pattern: Follow convertBackendAuthConfig in converter.go — use c.k8sClient.Get with types.NamespacedName, check errors.IsNotFound separately from other errors, return fmt.Errorf with %w
  • omitempty and nil safety: All code paths gated on AuthServerConfigRef != nil; Mode A must pass zero new lines when the field is absent
  • Exact string match for V-04: Use == for issuer comparison; no URL normalization applied. Operators must use identical strings in both configs
  • SPDX headers: Every new Go file must start with // SPDX-FileCopyrightText: Copyright 2025 Stacklok, Inc. and // SPDX-License-Identifier: Apache-2.0. Use task license-fix to add missing headers automatically
  • require.NoError(t, err) for test assertions: Use require from github.com/stretchr/testify/require rather than t.Fatal in unit tests; use Expect(...).To(Succeed()) in operator E2E tests (Gomega)
  • Ginkgo for E2E tests: Use Describe, BeforeAll/AfterAll (for Ordered suites), By() annotations, and Eventually(...).Within(...).ProbeEvery(...) for async assertions. Follow the existing operator E2E test structure in test/e2e/thv-operator/virtualmcp/

Code Pointers

  • cmd/thv-operator/controllers/virtualmcpserver_controller.go — runValidations() (line 276): add the new validateAuthServerConfigRef call here, after validateEmbeddingServerRef. Pattern for the new method: follow validateCompositeToolRefs (line 385) for structure; follow validateGroupRef (line 327) for the SetPhase/SetMessage/SetCondition call pattern
  • cmd/thv-operator/api/v1alpha1/virtualmcpserver_types.go — condition type constants block (line 209): add ConditionTypeAuthServerConfigValid = "AuthServerConfigValid" alongside existing constants
  • cmd/thv-operator/pkg/vmcpconfig/converter.go — Convert() (line 68): add the convertAuthServerConfig step after the convertIncomingAuth block (line 81). Pattern: follow convertBackendAuthConfig (line 283) for the k8sClient.Get call structure
  • cmd/thv-operator/api/v1alpha1/mcpexternalauthconfig_types.go — EmbeddedAuthServerConfig struct (line 152): source of fields to map to authserver.RunConfig; note AllowedAudiences is absent by design (derived from the referencing VirtualMCPServer)
  • pkg/authserver/config.go — authserver.RunConfig: target type for the converter; key fields: Issuer string, AllowedAudiences []string, Upstreams []UpstreamRunConfig
  • pkg/vmcp/server/server.go — Config struct (line 83), Handler() method (line ~444): target for unit tests; the AuthServer *runner.EmbeddedAuthServer field added in Phase 2 (Phase 2: Server wiring — mount embedded auth server routes on vMCP mux #4141) is what Mode B tests populate
  • pkg/authserver/runner/embeddedauthserver.go — NewEmbeddedAuthServer(ctx, *authserver.RunConfig) and RegisterHandlers(mux) (added in Phase 2: Server wiring — mount embedded auth server routes on vMCP mux #4141): used to construct Mode B test fixtures
  • test/e2e/proxy_oauth_test.go — reference pattern for CLI E2E tests: OIDCMockServer, process start/stop, Eventually, By() annotations
  • test/e2e/oidc_mock.go — OIDCMockServer: reusable mock OIDC server backed by Ory Fosite; use as the upstream IDP in Mode B E2E tests
  • test/e2e/thv-operator/virtualmcp/virtualmcp_external_auth_test.go — reference pattern for operator E2E tests: k8sClient.Create, Eventually with condition assertions, BeforeAll/AfterAll cleanup
  • test/e2e/thv-operator/virtualmcp/helpers.go — shared helpers for operator E2E tests: CreateMCPGroupAndWait, CreateMockHTTPServer; check what is available before writing new helpers

Component Interfaces

New constant in cmd/thv-operator/api/v1alpha1/virtualmcpserver_types.go:

// ConditionTypeAuthServerConfigValid indicates whether the authServerConfigRef
// references a valid, compatible MCPExternalAuthConfig.
const ConditionTypeAuthServerConfigValid = "AuthServerConfigValid"

New method on VirtualMCPServerReconciler in cmd/thv-operator/controllers/virtualmcpserver_controller.go:

// validateAuthServerConfigRef validates the MCPExternalAuthConfig referenced by
// spec.authServerConfigRef. It verifies the type is "embeddedAuthServer", checks
// issuer consistency with IncomingAuth.OIDC.Issuer (V-04), and verifies the
// audience is non-empty (defense-in-depth guard).
// Returns (true, nil) to continue, (false, nil) for spec errors (no requeue),
// (false, err) for transient errors (triggers requeue).
func (r *VirtualMCPServerReconciler) validateAuthServerConfigRef(
    ctx context.Context,
    vmcp *mcpv1alpha1.VirtualMCPServer,
    statusManager virtualmcpserverstatus.StatusManager,
) (bool, error)

New private method on Converter in cmd/thv-operator/pkg/vmcpconfig/converter.go:

// convertAuthServerConfig resolves spec.authServerConfigRef to an AuthServerConfig.
// It fetches the MCPExternalAuthConfig, verifies its type, converts EmbeddedAuthServerConfig
// to authserver.RunConfig, and derives allowedAudiences from the VirtualMCPServer's
// incoming auth audience.
func (c *Converter) convertAuthServerConfig(
    ctx context.Context,
    vmcp *mcpv1alpha1.VirtualMCPServer,
) (*vmcpconfig.AuthServerConfig, error)

Key mapping from EmbeddedAuthServerConfig to authserver.RunConfig (in the converter):

// Derive allowedAudiences from the VirtualMCPServer incoming auth config.
// Note: AllowedAudiences is NOT a field on EmbeddedAuthServerConfig in the CRD.
// It is always derived here so the CRD does not need to duplicate the audience value.
var allowedAudiences []string
if vmcp.Spec.IncomingAuth != nil &&
    vmcp.Spec.IncomingAuth.OIDCConfig != nil &&
    vmcp.Spec.IncomingAuth.OIDCConfig.Inline != nil &&
    vmcp.Spec.IncomingAuth.OIDCConfig.Inline.Audience != "" {
    allowedAudiences = []string{vmcp.Spec.IncomingAuth.OIDCConfig.Inline.Audience}
}

runConfig := &authserver.RunConfig{
    Issuer:           embeddedCfg.Issuer,
    AllowedAudiences: allowedAudiences,
    // ... map SigningKeySecretRefs, HMACSecretRefs, TokenLifespans, Storage, Upstreams
}

Testing Strategy

Unit Tests — HTTP Handler (pkg/vmcp/server/server_test.go or pkg/vmcp/server/handler_test.go)

Use httptest.NewRecorder() and httptest.NewRequest() to test Handler() without starting a real server. Use httptest.NewServer for integration-style route tests.

  • Mode A (nil AuthServer): GET /.well-known/oauth-protected-resource → 200
  • Mode A: GET /.well-known/openid-configuration → 404
  • Mode A: GET /.well-known/oauth-authorization-server → 404
  • Mode A: GET /.well-known/jwks.json → 404
  • Mode A: GET /oauth/token → 404
  • Mode B (non-nil AuthServer from NewEmbeddedAuthServer with ephemeral dev config): GET /.well-known/openid-configuration → 200 with Content-Type: application/json
  • Mode B: GET /.well-known/jwks.json → 200
  • Mode B: GET /.well-known/oauth-protected-resource → 200 (still served in Mode B)
  • Both modes: GET /mcp without bearer token → 401 when AuthMiddleware is set to a simple "reject all" middleware

CLI E2E Tests (test/e2e/vmcp_authserver_test.go)

Follow test/e2e/proxy_oauth_test.go exactly. Use OIDCMockServer as the upstream IDP for Mode B. Write YAML config files to a temp dir. Start the vMCP binary with exec.Command.

  • Mode B positive: OIDC discovery returns 200 with valid JSON (non-empty issuer, non-empty jwks_uri)
  • Mode B positive: unauthenticated MCP request returns 401
  • Mode A positive: OIDC discovery path returns 404; oauth-protected-resource returns 200
  • Negative V-04: process exits non-zero; stderr contains issuer mismatch message
  • Negative V-01: process exits non-zero; stderr contains upstream_inject / AS required message
  • Negative V-02: process exits non-zero; stderr contains unknown provider name message

Operator E2E Tests (test/e2e/thv-operator/virtualmcp/virtualmcp_authserver_test.go)

Follow virtualmcp_external_auth_test.go. Use Ordered suite with BeforeAll/AfterAll for resource cleanup.

  • Positive: MCPExternalAuthConfig + VirtualMCPServer created; Eventually asserts AuthServerConfigValid condition is True; Eventually asserts Deployment exists and is available
  • Negative: VirtualMCPServer with issuer mismatch; Eventually asserts AuthServerConfigValid condition is False with message containing "issuer"; phase is Failed; no Deployment exists for this VirtualMCPServer

Edge Cases

  • spec.authServerConfigRef nil (Mode A): validateAuthServerConfigRef is skipped; Convert() does not set config.AuthServer; no AuthServerConfigValid condition appears
  • Referenced MCPExternalAuthConfig not found: reconciler returns a transient error (triggers requeue), phase is set to Failed
  • Referenced MCPExternalAuthConfig has wrong type (e.g., "tokenExchange"): reconciler surfaces AuthServerConfigValid=False with a descriptive type mismatch message; phase is Failed; no requeue (spec error)
  • spec.incomingAuth.oidcConfig.inline.audience is empty: converter sets allowedAudiences = nil (not an empty slice); reconciler audience guard logs a warning but does not fail (defense-in-depth, not blocking)
  • Issuer mismatch (V-04): exact string comparison; trailing slash difference ("https://as.example.com" vs "https://as.example.com/") must trigger the mismatch error

Out of Scope

  • upstream_inject outgoing auth strategy implementation (deferred to a follow-up RFC — the constant, config struct, and validation were added in Phase 3 but the actual token injection middleware is not implemented in RFC-0053)
  • Full end-to-end token flow via identity.UpstreamTokens — this requires RFC-0052 (Auth Server: multi-upstream provider support #3924) to be merged; the E2E tests here cover Mode A/Mode B routing and the AuthServerConfigValid condition, not the full token issuance and upstream token population chain
  • Hot-reload of AS configuration (requires pod restart; out of scope for this RFC)
  • Multi-upstream IDP support beyond what UpstreamRunConfig.Name already provides (defined by RFC-0052)
  • CLI-path upstream token swap (pkg/auth/upstreamswap middleware)
  • Step-up auth signaling

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    authenticationauthorizationenhancementNew feature or requestgoPull requests that update go codekubernetesItems related to KubernetesoperatortestingvmcpVirtual MCP Server related issues

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions