feat: add structured request logging with request IDs and configurable levels

[reldothescribe]

Summary

This PR adds structured request logging middleware to improve observability across the MCP server and proxy.

Features

1. HTTP Request Logging

• Logs all HTTP requests with structured fields:
  • method: HTTP method (GET, POST, etc.)
  • path: Request path
  • status: HTTP response status code
  • duration_ms: Request duration in milliseconds
  • user_agent: Client user agent
  • remote_addr: Client IP address
  • size: Response size in bytes

2. Request ID Propagation

• Automatically generates unique request IDs (UUID v4)
• Propagates via X-Request-ID header
• Preserves existing request IDs from incoming requests
• Returns request ID in response headers

3. Correlation IDs for Tracing

• Supports X-Correlation-ID header for distributed tracing
• Correlation IDs are propagated through the request lifecycle
• Enables tracking requests across multiple services

4. Configurable Log Levels

• Supports debug, info, warn, error levels
• Configurable via config file or CLI flags
• CLI flags override config file settings

5. JSON Logging Format

• Supports both text and json formats
• JSON format recommended for production environments
• Easy integration with log aggregation systems

Configuration

MCP Server (config.yaml)

observability:
  logging:
    level: info      # debug, info, warn, error
    format: text     # text or json
    output_path: ""  # Optional: file path (default: stdout)

Proxy Server (proxy-config.yaml)

logging:
  level: info      # debug, info, warn, error
  format: json     # text or json
  output_path: ""  # Optional: file path (default: stdout)

Usage

The logging middleware is automatically applied to all HTTP routes when using SSE or streamable-http transport:

• MCP Server: Applied to /sse, /message, /mcp, /health, /ready endpoints
• Proxy Server: Applied to all authenticated routes

Testing

All new code includes comprehensive tests:

• Request ID generation
• Logger configuration (levels, formats, file output)
• Middleware functionality
• Context propagation

go test -v ./pkg/observability/...

Example Output

Text Format

INFO[2025-02-01T23:16:57Z] HTTP request completed  duration=45.2ms duration_ms=45 method=GET path=/mcp request_id=abc-123 status=200 user_agent=mcp-client/1.0

JSON Format

{
  "level": "info",
  "msg": "HTTP request completed",
  "method": "GET",
  "path": "/mcp",
  "status": 200,
  "duration_ms": 45,
  "request_id": "abc-123",
  "user_agent": "mcp-client/1.0"
}