Engine: Task changes

.Jules/bolt.md

@@ -1,3 +1,18 @@
+# Bolt's Journal
+This journal documents critical performance learnings for the codebase.
+
+## 2024-05-23 - Lazy Loading Search Providers
+**Learning:** Moving heavy SDK imports (like `google.genai`) inside methods significantly improves startup time and robustness.
+**Action:** Use local imports inside factory methods for optional integrations or heavy dependencies. When testing, patch the class at its *source definition*, not the module namespace.
+
+## 2024-05-24 - [Initial Entry]
+**Learning:** Initial setup of Bolt's journal.
+**Action:** Use this file to record specific performance insights.
+
 ## 2025-05-18 - [Optimization of Fuzzy Matching]
 **Learning:** `difflib.SequenceMatcher.real_quick_ratio()` provides an O(1) upper bound check based on length, which is significantly faster than `quick_ratio()` (O(N)) or `ratio()` (O(N*M)). Using this as a first-pass filter for fuzzy matching large datasets can dramatically improve performance.
 **Action:** Always check for `real_quick_ratio()` when implementing fuzzy matching loops with `difflib`.
+
+## 2026-01-25 - [Memory vs Reality Discrepancy]
+**Learning:** The memory claimed `format_search_output` was already using list join, but the actual code on disk was using O(N^2) string concatenation.
+**Action:** Always verify code state with `read_file` before trusting memory or documentation about optimizations.

.Jules/sentinel.md

@@ -12,3 +12,8 @@
 **Vulnerability:** The `RateLimitMiddleware` blindly trusted the `X-Forwarded-For` header, allowing attackers to bypass rate limits by spoofing random IPs in this header.
 **Learning:** Middleware often defaults to "dev-friendly" (trusting headers) which is insecure for production. Implicit trust in headers creates silent vulnerabilities.
 **Prevention:** Always default to NOT trusting `X-Forwarded-For`. Require explicit configuration (`TRUST_PROXY_HEADERS`) to enable it.
+
+## 2026-01-25 - [HIGH] XSS Defense in Depth via Sanitization
+**Vulnerability:** The ChatMessagesView component rendered Markdown content using `ReactMarkdown` without explicit HTML sanitization (`rehype-sanitize`). While `ReactMarkdown` v9+ is safe by default, adding plugins or future configuration changes could introduce Stored XSS vulnerabilities.
+**Learning:** Security tools often require careful configuration to avoid breaking functionality. Default sanitization schemas can be too aggressive, stripping attributes needed for styling (like `className` for code highlighting or `align` for GFM tables).
+**Prevention:** Always include `rehype-sanitize` when rendering user content. Explicitly configure the schema to allow necessary safe attributes (e.g., `className` on `code` blocks) to balance security and functionality.

backend/src/agent/app.py

@@ -42,24 +42,26 @@ async def dispatch(self, request: Request, call_next):
             # 🛡️ Sentinel: Reject 'Transfer-Encoding: chunked' to prevent Content-Length bypass (Request Smuggling/DoS)
             transfer_encoding = request.headers.get("transfer-encoding", "").lower()
             if "chunked" in transfer_encoding:
-                logger.warning("Request blocked: Chunked encoding not allowed")
+                logger.warning("Rejected chunked transfer encoding (DoS protection)")
                 return Response("Chunked encoding not allowed", status_code=411)
 
             content_length = request.headers.get("content-length")
             if not content_length:
                 # 🛡️ Sentinel: Enforce Content-Length for state-changing methods to prevent streaming DoS
-                logger.warning("Request blocked: Content-Length required")
+                logger.warning("Rejected request missing Content-Length (DoS protection)")
                 return Response("Content-Length required", status_code=411)
 
             try:
                 # 🛡️ Sentinel: Prevent 500 crashes from malformed Content-Length headers
                 if int(content_length) > self.max_upload_size:
                     logger.warning(
-                        f"Request blocked: Request entity too large ({content_length} > {self.max_upload_size})"
+                        f"Rejected request with content length {content_length} > {self.max_upload_size}"
                     )
                     return Response("Request entity too large", status_code=413)
             except ValueError:
-                logger.warning("Request blocked: Invalid Content-Length")
+                logger.warning(
+                    f"Rejected request with invalid content length: {content_length}"
+                )
                 return Response("Invalid Content-Length", status_code=400)
         return await call_next(request)
 
@@ -149,7 +151,11 @@ async def validation_exception_handler(request: Request, exc: RequestValidationE
     limit=100,
     window=60,
     protected_paths=["/agent", "/threads"],
+<<<<<<< HEAD
     trust_proxy_headers=app_config.trust_proxy_headers,
+=======
+    trust_proxy_headers=app_config.trust_proxy_headers
+>>>>>>> dc0cbcc (feat(security): add logging for rejected requests in middleware)
 )
 
 # Add Security Headers (OUTERMOST - added last)

backend/src/agent/security.py

@@ -5,6 +5,7 @@
 import math
 import os
 import time
+import math
 from collections import defaultdict
 from typing import List, Set
 
@@ -266,6 +267,7 @@ async def dispatch(self, request: Request, call_next):
             # 🛡️ Sentinel: Support X-Forwarded-For for proxies (Render/Load Balancers)
             # Use trust-bound extraction to prevent IP spoofing attacks.
             forwarded = request.headers.get("X-Forwarded-For")
+<<<<<<< HEAD
             fallback_ip = request.client.host if request.client else "unknown"
 
             if forwarded and self.trust_proxy_headers:
@@ -276,6 +278,27 @@ async def dispatch(self, request: Request, call_next):
                 )
                 if client_ip is None:
                     client_ip = fallback_ip
+=======
+            if forwarded and self.trust_proxy_headers:
+                # 🛡️ Sentinel: Prevent spoofing by traversing from the end (trusted proxies)
+                # Proxies (like Render) append the verified client IP to the end.
+                # We traverse backwards to find the first non-private IP to avoid blocking the proxy itself.
+                try:
+                    ips = [ip.strip() for ip in forwarded.split(",")]
+                    client_ip = ips[-1]  # Default to last IP
+                    for ip in reversed(ips):
+                        try:
+                            # Check if IP is public (not private, not loopback)
+                            ip_obj = ipaddress.ip_address(ip)
+                            if not ip_obj.is_private and not ip_obj.is_loopback:
+                                client_ip = ip
+                                break
+                        except ValueError:
+                            continue  # Skip invalid IPs
+                except Exception:
+                    # Fallback to simple extraction if parsing fails
+                    client_ip = forwarded.split(",")[-1].strip()
+>>>>>>> dc0cbcc (feat(security): add logging for rejected requests in middleware)
 
                 # Truncate to 100 chars to prevent memory exhaustion attacks
                 client_ip = client_ip[:100]
@@ -297,17 +320,36 @@ async def dispatch(self, request: Request, call_next):
                 self.requests[client_key] = active_requests
 
                 # Calculate retry_after
+<<<<<<< HEAD
                 oldest_request_time = active_requests[0]
                 reset_time = oldest_request_time + self.window
                 retry_after = max(1, int(math.ceil(reset_time - now)))
 
-                logger.warning(f"Rate limit exceeded for {client_key} on {path}")
+                logger.warning(f"Rate limit exceeded for client {client_key} on path {path}")
+
+                return JSONResponse(
+                    status_code=429,
+                    content={"detail": "Too Many Requests", "retry_after": retry_after},
+                    headers={"Retry-After": str(retry_after)},
+                )
+=======
+                if active_requests:
+                    oldest_request_time = active_requests[0]
+                    reset_time = oldest_request_time + self.window
+                    retry_after = max(1, int(math.ceil(reset_time - now)))
+                else:
+                    retry_after = self.window
+
+                logger.warning(
+                    f"Rate limit exceeded for client {client_key} on path {path}"
+                )
 
                 return JSONResponse(
                     status_code=429,
                     content={"detail": "Too Many Requests", "retry_after": retry_after},
                     headers={"Retry-After": str(retry_after)},
                 )
+>>>>>>> dc0cbcc (feat(security): add logging for rejected requests in middleware)
 
             active_requests.append(now)
 
@@ -355,6 +397,9 @@ async def dispatch(self, request: Request, call_next):
                 ):  # This was a new entry (or re-entry after expiry)
                     # Safe delete using pop to avoid KeyErrors in race conditions
                     self.requests.pop(client_key, None)
+                    logger.warning(
+                        f"Server busy (max clients exceeded) for client {client_key} on path {path}"
+                    )
                     return Response("Server Busy", status_code=503)
 
             self.requests[client_key] = active_requests

backend/src/search/router.py

@@ -1,3 +1,4 @@
+"""Search router module for handling multiple search providers."""
 import logging
 import threading
 from enum import Enum
@@ -11,6 +12,7 @@
 
 
 class SearchProviderType(Enum):
+    """Enumeration of supported search providers."""
     GOOGLE = "google"
     DUCKDUCKGO = "duckduckgo"
     BRAVE = "brave"
@@ -25,9 +27,11 @@ def __init__(self, app_config: AppConfig = config):
         """Initialize router with config."""
         self.config = app_config
         self.providers: Dict[str, SearchProvider] = {}
+<<<<<<< HEAD
         self._providers_lock = threading.Lock()
 
     def _get_provider(self, name: str) -> SearchProvider | None:
+        """Lazily initialize and return a search provider instance."""
         # Quick check without lock
         if name in self.providers:
             return self.providers[name]
@@ -40,23 +44,18 @@ def _get_provider(self, name: str) -> SearchProvider | None:
             try:
                 if name == SearchProviderType.GOOGLE.value:
                     from .providers.google_adapter import GoogleSearchAdapter
-
                     self.providers[name] = GoogleSearchAdapter()
                 elif name == SearchProviderType.BRAVE.value:
                     from .providers.brave_adapter import BraveSearchAdapter
-
                     self.providers[name] = BraveSearchAdapter()
                 elif name == SearchProviderType.DUCKDUCKGO.value:
                     from .providers.duckduckgo_adapter import DuckDuckGoAdapter
-
                     self.providers[name] = DuckDuckGoAdapter()
                 elif name == SearchProviderType.TAVILY.value:
                     from .providers.tavily_adapter import TavilyAdapter
-
                     self.providers[name] = TavilyAdapter()
                 elif name == SearchProviderType.BING.value:
                     from .providers.bing_adapter import BingAdapter
-
                     self.providers[name] = BingAdapter()
             except Exception as e:
                 logger.debug(f"Provider {name} failed to init: {e}")
@@ -70,6 +69,42 @@ def _get_provider(self, name: str) -> SearchProvider | None:
                 )
 
             return self.providers.get(name)
+=======
+
+    def _load_provider(self, name: str) -> Optional[SearchProvider]:
+        """Lazily load provider class and instantiate."""
+        if name in self.providers:
+            return self.providers[name]
+
+        provider = None
+        try:
+            if name == SearchProviderType.GOOGLE.value:
+                from .providers.google_adapter import GoogleSearchAdapter
+                provider = GoogleSearchAdapter()
+            elif name == SearchProviderType.DUCKDUCKGO.value:
+                from .providers.duckduckgo_adapter import DuckDuckGoAdapter
+                provider = DuckDuckGoAdapter()
+            elif name == SearchProviderType.BRAVE.value:
+                from .providers.brave_adapter import BraveSearchAdapter
+                provider = BraveSearchAdapter()
+            elif name == SearchProviderType.TAVILY.value:
+                from .providers.tavily_adapter import TavilyAdapter
+                provider = TavilyAdapter()
+            elif name == SearchProviderType.BING.value:
+                from .providers.bing_adapter import BingAdapter
+                provider = BingAdapter()
+
+            if provider:
+                self.providers[name] = provider
+
+        except Exception as e:
+            logger.debug(f"Failed to lazy load provider {name}: {e}")
+
+        return provider
+
+    def _get_provider(self, name: str) -> Optional[SearchProvider]:
+        return self._load_provider(name)
+>>>>>>> 5d045b0 (Refactor SearchRouter to lazily load provider adapters on first use for faster startup.)
 
     def search(
         self,
@@ -96,6 +131,8 @@ def search(
             provider = self._get_provider(primary_name)
 
         if not provider:
+            # Fallback was also unavailable or failed to init
+            logger.error("No valid search provider available.")
             raise ValueError("No valid search provider available.")
 
         # Execute with reliability-first logic
@@ -117,8 +154,6 @@ def search(
                     fallback_provider = self._get_provider(fallback_name)
                     if fallback_provider:
                         # Fallback gets the same retry logic or just a single shot?
-                        # For simplicity, fallback is usually single shot untuned or standard.
-                        # Let's try standard (tuned=True default)
                         return fallback_provider.search(query, max_results=max_results)
 
                 # If we get here, all attempts failed

backend/tests/agent/test_rate_limiter.py

@@ -7,13 +7,18 @@
 from agent.rate_limiter import RateLimiter, PACIFIC_TZ
 
 class TestRateLimiter(unittest.TestCase):
-    def test_daily_reset_logic(self):
+    @patch('agent.rate_limiter.datetime')
+    def test_daily_reset_logic(self, mock_datetime):
         """Test that daily reset occurs at midnight Pacific Time."""
+        # Setup mock time
+        now_date = datetime(2023, 10, 2, 12, 0, 0, tzinfo=PACIFIC_TZ)
+        mock_datetime.now.return_value = now_date
+
         limiter = RateLimiter()
 
         # Manually set last reset date to yesterday
-        yesterday = datetime.now(PACIFIC_TZ).date() - timedelta(days=1)
-        limiter._last_reset_date = yesterday
+        yesterday_date = now_date.date() - timedelta(days=1)
+        limiter._last_reset_date = yesterday_date
 
         # Add some dummy requests
         limiter._requests_per_day.append(12345)
@@ -24,15 +29,19 @@ def test_daily_reset_logic(self):
 
         # Should be cleared
         self.assertEqual(len(limiter._requests_per_day), 0)
-        self.assertEqual(limiter._last_reset_date, datetime.now(PACIFIC_TZ).date())
+        self.assertEqual(limiter._last_reset_date, now_date.date())
 
-    def test_no_reset_same_day(self):
+    @patch('agent.rate_limiter.datetime')
+    def test_no_reset_same_day(self, mock_datetime):
         """Test that daily reset does not occur on the same day."""
+        # Setup mock time
+        now_date = datetime(2023, 10, 2, 12, 0, 0, tzinfo=PACIFIC_TZ)
+        mock_datetime.now.return_value = now_date
+
         limiter = RateLimiter()
 
         # Set last reset date to today
-        today = datetime.now(PACIFIC_TZ).date()
-        limiter._last_reset_date = today
+        limiter._last_reset_date = now_date.date()
 
         # Add some dummy requests
         limiter._requests_per_day.append(12345)
@@ -43,7 +52,7 @@ def test_no_reset_same_day(self):
 
         # Should NOT be cleared
         self.assertEqual(len(limiter._requests_per_day), 1)
-        self.assertEqual(limiter._last_reset_date, today)
+        self.assertEqual(limiter._last_reset_date, now_date.date())
 
     @patch("agent.rate_limiter.time")
     def test_wait_if_needed_rpm_limit(self, mock_time):