diff --git a/README.md b/README.md index 73e904d..2551df0 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) [![Python 3.11+](https://img.shields.io/badge/Python-3.11+-green.svg)](https://python.org) [![PyPI](https://img.shields.io/pypi/v/claude-ctx.svg)](https://pypi.org/project/claude-ctx/) -[![Tests](https://img.shields.io/badge/Tests-3894_collected-brightgreen.svg)](#) +[![Tests](https://img.shields.io/badge/Tests-3907_collected-brightgreen.svg)](#) [![Graph](https://img.shields.io/badge/Graph-102%2C928_nodes_/_2.9M_edges-red.svg)](graph/) [![Docs](https://img.shields.io/badge/docs-MkDocs_Material-blue.svg)](https://stevesolun.github.io/ctx/) [![Repo views](https://hits.sh/github.com/stevesolun/ctx.svg?label=repo%20views)](https://hits.sh/github.com/stevesolun/ctx/) diff --git a/docs/dashboard.md b/docs/dashboard.md index 7ae4500..a533e79 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -10,7 +10,7 @@ and harness wiki/graph browsing. ```bash ctx-monitor serve # http://127.0.0.1:8765 ctx-monitor serve --port 8888 # custom port -ctx-monitor serve --host 0.0.0.0 --port 8888 # LAN-visible HTML only (explicit opt-in) +ctx-monitor serve --host 0.0.0.0 --port 8888 # LAN read-only with startup token URL ``` Zero Python dependencies added by the dashboard. Everything runs on @@ -150,7 +150,7 @@ Every page shows the same nav bar. The eleven tabs cover the dashboard-supported observable surface of ctx: ``` -Home · Loaded · Skills · Wiki · Graph · Status · KPIs · Runtime · Sessions · Logs · Live +Home · Loaded · Skills · Wiki · Graph · Manage · Harness Setup · Docs · Config · Status · KPIs · Runtime · Sessions · Logs · Live ``` ### HTML views @@ -173,6 +173,10 @@ per-process monitor token injected into the rendered page. | `/wiki/?type=` | Dashboard-supported wiki entity page rendered: markdown body + full frontmatter table + grade banner + deep links to sidecar and graph-neighborhood views. The optional `type` query disambiguates duplicate slugs such as `langgraph`. | | `/graph` | **Graph explorer landing page** - node/edge count header, a "Popular seed slugs" block (18 highest-degree skill/agent/MCP/harness entities as clickable chips), search box for any skill/agent/MCP/harness slug, and the built-in graph list panel. Clicking a seed chip navigates to `/graph?slug=&type=`. | | `/graph?slug=&type=` | **Built-in** 1-hop neighborhood around the target skill/agent/MCP/harness slug. Entity pills identify skill, agent, MCP server, and harness rows. Tap any node to navigate to that entity's typed wiki page. Type and tag filters run client-side. | +| `/manage` | Search, inspect, edit, delete, and manually import skill/agent/MCP/harness wiki entities through the same safe-name and mutation-token checks as live load/unload. | +| `/harness` | Harness Setup wizard for non-Claude/custom API/local model users: collects model, goals, tool needs, safety constraints, and shows the harness recommendation/install path. | +| `/docs` | Local repo docs rendered inside the dashboard with MkDocs-like tabs, sidebar table of contents, in-dashboard search, and source links. | +| `/config` | Effective ctx config with defaults, required markers, field explanations, and editable user overrides where supported. | | `/status` | Durable queue and artifact status: job counts by state, recent queue jobs, graph/wiki artifact sizes, and crash-safe promotion metadata. | | `/kpi` | **KPI dashboard** — total entity count with subject breakdown, grade distribution pills, two-column tables for grade counts and lifecycle tiers (active · watch · demote · archive), hard-floor reasons with counts, **By category** table (count · avg score · A/B/C/D/F mix per category), **Top demotion candidates** (active/watch entries graded D or F, sorted by consecutive-D streak desc then score asc), and the **Archived** list. Same shape as `python -m kpi_dashboard render` but HTML | | `/runtime` | Generic harness runtime ledger from `CTX_RUNTIME_LIFECYCLE_DIR` or `~/.ctx/runtime/events.jsonl`: validation totals, failed/error checks, recent validation rows, and open escalations. | @@ -192,13 +196,18 @@ per-process monitor token injected into the rendered page. | `GET /api/graph/.json?type=&hops=1&limit=40` | Dashboard-shaped skill/agent/MCP/harness `{nodes, edges, center}`; `type` is optional but recommended for duplicate slugs, `hops` is [1, 3], `limit` is [5, 150]. | | `GET /api/kpi.json` | `DashboardSummary` passthrough — `{total, by_subject, grade_counts, lifecycle_counts, category_breakdown, hard_floor_counts, low_quality_candidates, archived, generated_at}`. Returns `{total: 0, detail: "no sidecars yet"}` when the quality directory is empty | | `GET /api/runtime.json` | Runtime lifecycle summary: source path, validation count, failed/error count, open-escalation count, latest validation, recent validations, open escalations, and session IDs. | +| `GET /api/config.json` | Effective/default/user config payload used by the Config tab. | +| `GET /api/entities/search.json?q=&type=&limit=80` | Wiki entity search results for Manage, Config, and entity picker flows. | +| `GET /api/entity/.json?type=` | Frontmatter and Markdown body for one wiki entity. | | `GET /api/events.stream` | Server-sent events tail of `~/.claude/ctx-audit.jsonl` | ### Mutation endpoints Dashboard GET views are read-only. When `ctx-monitor` is bound to a -non-loopback host, `/api/*` JSON and SSE routes are disabled; keep the -default loopback bind for local automation. Both POST endpoints enforce +non-loopback host, HTML, `/api/*` JSON, and SSE routes require the +read-token URL printed by `ctx-monitor`; the first successful token URL +sets an HttpOnly same-site cookie for dashboard navigation. Keep the +default loopback bind for local automation. POST endpoints enforce same-origin (browser tab open on another origin can't forge a request), require the per-process `X-CTX-Monitor-Token` injected into the dashboard page, and reject any slug failing the shared safe-name validator. That validator blocks path @@ -214,6 +223,9 @@ load/unload mutation endpoint yet. | `POST /api/unload` | `{"slug": "...", "entity_type": "skill"}` | `skill_unload.unload_from_session([slug])` | | `POST /api/unload` | `{"slug": "...", "entity_type": "agent"}` | remove the agent row from `skill-manifest.json` and append an unload row | | `POST /api/unload` | `{"slug": "...", "entity_type": "mcp-server"}` | `mcp_install.uninstall_mcp(slug, wiki_dir=...)` | +| `POST /api/config` | `{"updates": {...}}` | persist supported user config overrides after validation | +| `POST /api/entity/upsert` | entity metadata/body payload | write or update a wiki entity, then attach graph/recommendation metadata | +| `POST /api/entity/delete` | `{"slug": "...", "entity_type": "skill"}` | remove a dashboard-supported wiki entity after safe-name validation | Harness load/unload POSTs are rejected with the exact `ctx-harness-install ... --dry-run` command to run instead. Skill rows emit @@ -292,8 +304,10 @@ observability proof that ctx's telemetry pipeline is live. ## Security - **Binds to 127.0.0.1 by default**. Use `--host 0.0.0.0` only if - you actually want LAN-visible. No authentication; the server is - intended for a local developer's own machine. + you actually want LAN-visible read-only access. The startup output + prints a one-process read-token URL; without that token or the cookie + it sets, LAN HTML/API/SSE requests return 403. Mutations remain + disabled on non-loopback binds. - **Same-origin gating on mutation**. Any POST with an `Origin` header that doesn't match `Host` returns 403. Curl and direct tool calls are allowed (no Origin header at all). diff --git a/docs/entity-onboarding.md b/docs/entity-onboarding.md index e30978c..80b45b5 100644 --- a/docs/entity-onboarding.md +++ b/docs/entity-onboarding.md @@ -67,7 +67,9 @@ the update is treated like a release step. 11. Unpark and stage the graph artifacts once the release candidate is final: `python scripts/graph_artifact_guard.py unpark`, then `git add` the graph artifacts intentionally. Run `python scripts/graph_artifact_guard.py prune` - after interrupted Git/LFS runs or after release staging. + after interrupted Git/LFS runs or after release staging to clean prunable + local LFS cache entries. Add `--include-git-prune` only when you explicitly + want repo-wide dangling Git objects removed too. The durable wiki worker drains `entity-upsert`, `graph-export`, `skill-index-refresh`, `tar-refresh`, and `artifact-promotion` jobs. Use diff --git a/docs/index.md b/docs/index.md index e1e81b5..55e9d7e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -202,7 +202,7 @@ ones are flagged. New ones self-ingest. --- **v1.0.9** — MIT, CI-matrixed (Ubuntu 3.12 plus Windows/macOS 3.11/3.12), - 3,894 tests collected. Ships console scripts including `ctx-init`, + 3,907 tests collected. Ships console scripts including `ctx-init`, `ctx-monitor` (local dashboard with graph + wiki + load/unload for skills, agents, and MCP servers, plus Harness Setup for user-owned LLMs), `ctx-incremental-attach`, `ctx-incremental-shadow`, `ctx-dedup-check` diff --git a/graph/README.md b/graph/README.md index b5d10da..3402df6 100644 --- a/graph/README.md +++ b/graph/README.md @@ -206,14 +206,16 @@ unpark and stage the artifacts exactly once: ```bash python scripts/graph_artifact_guard.py unpark -git add graph/wiki-graph.tar.gz graph/wiki-graph-runtime.tar.gz graph/*.json.gz +git add graph/wiki-graph.tar.gz graph/wiki-graph-runtime.tar.gz \ + graph/skills-sh-catalog.json.gz graph/communities.json graph/entity-overlays.jsonl python scripts/graph_artifact_guard.py prune ``` If a local Git integration gets interrupted while artifacts are dirty, -`python scripts/graph_artifact_guard.py prune` removes unreachable local Git -objects and prunable local LFS cache entries. It does not delete tracked graph -files, rewrite history, or change the remote LFS store. +`python scripts/graph_artifact_guard.py prune` removes prunable local LFS cache +entries. It does not delete tracked graph files, rewrite history, or change the +remote LFS store. Repo-wide `git prune --expire=now` is intentionally opt-in via +`--include-git-prune` because it can discard unrelated dangling recovery objects. For a bulk skill refresh, update the existing shipped tarball through the release refresh path: diff --git a/scripts/ci_required.py b/scripts/ci_required.py index 63f61d9..13f6892 100644 --- a/scripts/ci_required.py +++ b/scripts/ci_required.py @@ -67,10 +67,6 @@ def failed_required_jobs( event_name == "pull_request" and _job_output(needs, "classify", "graph_only") == "true" ) - graph_changed_pr = ( - event_name == "pull_request" - and _job_output(needs, "classify", "graph_changed") == "true" - ) graph_artifact_changed_pr = ( event_name == "pull_request" and _job_output(needs, "classify", "graph_artifact_changed") == "true" @@ -109,7 +105,6 @@ def failed_required_jobs( and name == "graph-check" and result == "skipped" and not graph_artifact_changed_pr - and not (graph_changed_pr and not docs_only_pr) ): continue if ( diff --git a/scripts/graph_artifact_guard.py b/scripts/graph_artifact_guard.py index e116c83..3f2e63a 100644 --- a/scripts/graph_artifact_guard.py +++ b/scripts/graph_artifact_guard.py @@ -82,12 +82,15 @@ def _print_status(repo: Path, artifacts: list[str]) -> None: _run_git(repo, ["count-objects", "-vH"]) -def _prune(repo: Path, *, include_lfs: bool) -> None: - _run_git(repo, ["prune", "--expire=now", "--verbose"]) +def _prune(repo: Path, *, include_lfs: bool, include_git_prune: bool) -> None: + if include_git_prune: + _run_git(repo, ["prune", "--expire=now", "--verbose"]) if include_lfs: result = _run_git(repo, ["lfs", "prune", "--verbose"], check=False) if result.returncode != 0: raise SystemExit(result.returncode) + if not include_lfs and not include_git_prune: + print("No prune action selected.") def _clean_stale_graph_files(repo: Path, *, dry_run: bool) -> None: @@ -127,7 +130,7 @@ def _parser() -> argparse.ArgumentParser: help=( "status shows skip-worktree state; park hides generated archives from " "normal Git status/stage scans; unpark re-enables release staging; " - "prune removes unreachable local Git/LFS objects; clean-stale " + "prune removes prunable local LFS cache entries; clean-stale " "removes interrupted graph promotion leftovers." ), ) @@ -151,6 +154,14 @@ def _parser() -> argparse.ArgumentParser: action="store_true", help="For prune only: skip git lfs prune.", ) + parser.add_argument( + "--include-git-prune", + action="store_true", + help=( + "For prune only: also run repo-wide git prune --expire=now. " + "This can discard unrelated dangling recovery objects." + ), + ) parser.add_argument( "--dry-run", action="store_true", @@ -171,7 +182,11 @@ def main(argv: list[str] | None = None) -> int: elif args.command == "unpark": _set_skip_worktree(repo, artifacts, enabled=False) elif args.command == "prune": - _prune(repo, include_lfs=not args.skip_lfs) + _prune( + repo, + include_lfs=not args.skip_lfs, + include_git_prune=args.include_git_prune, + ) elif args.command == "clean-stale": _clean_stale_graph_files(repo, dry_run=args.dry_run) return 0 diff --git a/src/ctx/core/graph/resolve_graph.py b/src/ctx/core/graph/resolve_graph.py index 938091b..f3542db 100644 --- a/src/ctx/core/graph/resolve_graph.py +++ b/src/ctx/core/graph/resolve_graph.py @@ -194,22 +194,26 @@ def _apply_entity_overlays(G: nx.Graph, graph_path: Path) -> nx.Graph: for payload in active_overlay_records(records): authoritative_nodes = _authoritative_overlay_nodes(payload) + for node_id in authoritative_nodes: + if node_id in G: + G.remove_edges_from(list(G.edges(node_id))) nodes = payload.get("nodes", []) if isinstance(nodes, list): for node in nodes: if not isinstance(node, dict): continue - node_id = node.get("id") - if not isinstance(node_id, str) or not node_id: + incoming_node_id = node.get("id") + if not isinstance(incoming_node_id, str) or not incoming_node_id: continue + overlay_node_id: str = incoming_node_id attrs = {key: value for key, value in node.items() if key != "id"} - if node_id in G: + if overlay_node_id in G: attrs = ( - replace_node_attrs(G.nodes[node_id], attrs) - if node_id in authoritative_nodes - else merge_node_attrs(G.nodes[node_id], attrs) + replace_node_attrs(G.nodes[overlay_node_id], attrs) + if overlay_node_id in authoritative_nodes + else merge_node_attrs(G.nodes[overlay_node_id], attrs) ) - G.add_node(node_id, **attrs) + G.add_node(overlay_node_id, **attrs) applied_nodes += 1 edges = payload.get("edges", []) diff --git a/src/ctx_monitor.py b/src/ctx_monitor.py index ab73b9e..a9cbf52 100644 --- a/src/ctx_monitor.py +++ b/src/ctx_monitor.py @@ -66,12 +66,14 @@ import re import secrets import sqlite3 +import socket import sys import tarfile import threading import time import zlib from collections import defaultdict, deque +from http.cookies import CookieError, SimpleCookie from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path, PurePosixPath from typing import Any @@ -97,6 +99,7 @@ _GRAPH_REPORT_RE = re.compile(r"Nodes:\s*([\d,]+)\s*\|\s*Edges:\s*([\d,]+)") _MAX_POST_BODY_BYTES = 64 * 1024 _DASHBOARD_INDEX_MEMBER = "graphify-out/dashboard-neighborhoods.sqlite3" +_READ_TOKEN_COOKIE = "ctx_monitor_read_token" # ─── Data sources ──────────────────────────────────────────────────────────── @@ -132,6 +135,18 @@ def _origin_host_name(origin: str) -> str: return (parsed.hostname or "").rstrip(".").lower() +def _read_token_cookie(cookie_header: str) -> str: + if not cookie_header: + return "" + try: + cookie = SimpleCookie() + cookie.load(cookie_header) + except CookieError: + return "" + morsel = cookie.get(_READ_TOKEN_COOKIE) + return morsel.value if morsel is not None else "" + + def _claude_dir() -> Path: return Path(os.path.expanduser("~/.claude")) @@ -4997,7 +5012,7 @@ def _render_docs_markdown(markdown_text: str, page_anchor: str) -> str: try: import markdown as markdown_lib # type: ignore[import-untyped] - return str(markdown_lib.markdown( + rendered = str(markdown_lib.markdown( markdown_text, extensions=[ "admonition", @@ -5024,14 +5039,75 @@ def _render_docs_markdown(markdown_text: str, page_anchor: str) -> str: }, output_format="html5", )) + return _sanitize_docs_html(rendered) except Exception: return _render_wiki_markdown(markdown_text) +def _sanitize_docs_html(rendered_html: str) -> str: + """Remove active HTML from local docs before embedding in the dashboard.""" + dangerous_blocks = ( + "script", + "style", + "iframe", + "object", + "embed", + "form", + "textarea", + "select", + ) + dangerous_tags = ( + "base", + "button", + "input", + "link", + "meta", + ) + + def escape_match(match: re.Match[str]) -> str: + return html.escape(match.group(0)) + + for tag in dangerous_blocks: + rendered_html = re.sub( + rf"<\s*{tag}\b[^>]*>.*?<\s*/\s*{tag}\s*>", + escape_match, + rendered_html, + flags=re.IGNORECASE | re.DOTALL, + ) + rendered_html = re.sub( + rf"<\s*/?\s*{tag}\b[^>]*>", + escape_match, + rendered_html, + flags=re.IGNORECASE, + ) + for tag in dangerous_tags: + rendered_html = re.sub( + rf"<\s*/?\s*{tag}\b[^>]*>", + escape_match, + rendered_html, + flags=re.IGNORECASE, + ) + + rendered_html = re.sub( + r"\s+on[a-zA-Z0-9_-]+\s*=\s*(\"[^\"]*\"|'[^']*'|[^\s>]+)", + "", + rendered_html, + flags=re.IGNORECASE, + ) + rendered_html = re.sub( + r"\s+(href|src)\s*=\s*([\"'])\s*(javascript:|data:text/html)", + r" \1=\2#", + rendered_html, + flags=re.IGNORECASE, + ) + return rendered_html + + def _docs_search_text(entry: dict[str, str]) -> str: text = f"{entry['title']} {entry['path']} {entry['summary']} {entry['body']}" text = re.sub(r"```.*?```", " ", text, flags=re.DOTALL) text = re.sub(r"!!!\s+\w+(?:\s+\"[^\"]+\")?", " ", text) + text = re.sub(r"<[^>]+>", " ", text) text = re.sub(r"[*_`#>\[\]().!:-]+", " ", text) return re.sub(r"\s+", " ", text).strip().lower() @@ -6436,13 +6512,23 @@ def _read_authorized(self, qs: dict[str, str]) -> bool: request_host = _request_host_name(self.headers.get("Host", "")) if self._mutations_enabled(): return _host_allows_mutations(request_host) - token = self.headers.get("X-CTX-Monitor-Token") or qs.get("token", "") + token = ( + self.headers.get("X-CTX-Monitor-Token") + or qs.get("token", "") + or _read_token_cookie(self.headers.get("Cookie", "")) + ) return bool(_MONITOR_TOKEN) and secrets.compare_digest(token, _MONITOR_TOKEN) def _send_security_headers(self, *, html_response: bool = False) -> None: self.send_header("X-Content-Type-Options", "nosniff") self.send_header("Referrer-Policy", "no-referrer") self.send_header("X-Frame-Options", "DENY") + if getattr(self, "_ctx_set_read_cookie", False): + self.send_header( + "Set-Cookie", + f"{_READ_TOKEN_COOKIE}={_MONITOR_TOKEN}; Path=/; " + "HttpOnly; SameSite=Strict", + ) if html_response: self.send_header( "Content-Security-Policy", @@ -6507,6 +6593,7 @@ def do_GET(self) -> None: # noqa: N802 — stdlib signature from urllib.parse import parse_qs qs = {k: v[0] for k, v in parse_qs(raw_query).items()} try: + self._ctx_set_read_cookie = False read_authorized = getattr(self, "_read_authorized", lambda _qs: True) if not read_authorized(qs): if path.startswith("/api/"): @@ -6521,6 +6608,13 @@ def do_GET(self) -> None: # noqa: N802 — stdlib signature "

monitor read token required on non-loopback bind

", ) return + query_token = qs.get("token", "") + self._ctx_set_read_cookie = ( + not self._mutations_enabled() + and bool(query_token) + and bool(_MONITOR_TOKEN) + and secrets.compare_digest(query_token, _MONITOR_TOKEN) + ) if path == "/": self._send_html(_render_home()) elif path == "/sessions": @@ -6858,16 +6952,16 @@ def serve(host: str = "127.0.0.1", port: int = 8765) -> None: """Run the monitor. Blocks until Ctrl+C.""" global _MONITOR_TOKEN server = _make_monitor_server(host, port) - _MONITOR_TOKEN = ( - secrets.token_urlsafe(32) - if bool(getattr(server, "_ctx_mutations_enabled", False)) - else "" - ) - url = f"http://{host}:{port}/" + _MONITOR_TOKEN = secrets.token_urlsafe(32) + mutations_enabled = bool(getattr(server, "_ctx_mutations_enabled", False)) + url = f"http://{_monitor_display_host(host)}:{port}/" + if not mutations_enabled: + url = f"{url}?token={_MONITOR_TOKEN}" print(f"ctx-monitor serving at {url} (Ctrl+C to stop)", flush=True) - if not bool(getattr(server, "_ctx_mutations_enabled", False)): + if not mutations_enabled: print( - "ctx-monitor: non-loopback bind; load/unload mutations disabled", + "ctx-monitor: non-loopback bind; read token required and " + "load/unload mutations disabled", flush=True, ) try: @@ -6878,6 +6972,21 @@ def serve(host: str = "127.0.0.1", port: int = 8765) -> None: server.server_close() +def _monitor_display_host(host: str) -> str: + """Return a URL host users can paste into a browser.""" + if host in {"0.0.0.0", "::"}: + try: + candidate = socket.gethostbyname(socket.gethostname()) + except OSError: + candidate = "" + if candidate and not candidate.startswith("127."): + return candidate + return "localhost" + if ":" in host and not host.startswith("["): + return f"[{host}]" + return host + + def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser( prog="ctx-monitor", diff --git a/src/kpi_dashboard.py b/src/kpi_dashboard.py index 8c5ce90..9a59f30 100644 --- a/src/kpi_dashboard.py +++ b/src/kpi_dashboard.py @@ -35,6 +35,7 @@ from typing import Any, Iterable from ctx_lifecycle import ( + LifecycleState, LifecycleSources, STATE_ACTIVE, STATE_ARCHIVE, @@ -187,11 +188,16 @@ def _build_row( slug: str, *, score: QualityScore | None, + lifecycle_subject_type: str | None = None, lifecycle_state: str, consecutive_d_count: int, sources: LifecycleSources, ) -> EntityRow: - subject = score.subject_type if score is not None else _guess_subject(slug, sources) + subject = ( + score.subject_type + if score is not None + else lifecycle_subject_type or _guess_subject(slug, sources) + ) return EntityRow( slug=slug, subject_type=subject, @@ -218,16 +224,9 @@ def collect_rows( *, sources: LifecycleSources, ) -> list[EntityRow]: """Walk both sinks and return one row per known slug (union).""" - quality_sources = _quality_sources(sources.sidecar_dir) - quality_keys = set(quality_sources) - quality_slugs = {slug for slug, _sidecar_dir in quality_sources} - lifecycle_slugs = set(_iter_lifecycle_slugs(sources.sidecar_dir)) - row_sources = sorted(quality_keys, key=lambda item: (item[0], str(item[1]))) + [ - (slug, sources.sidecar_dir) - for slug in sorted(lifecycle_slugs - quality_slugs) - ] - rows: list[EntityRow] = [] - for slug, sidecar_dir in row_sources: + quality_rows: list[tuple[str, Path, QualityScore | None, LifecycleState | None]] = [] + quality_subjects: set[tuple[str, str]] = set() + for slug, sidecar_dir in _quality_sources(sources.sidecar_dir): try: score = load_quality( slug, @@ -236,17 +235,49 @@ def collect_rows( except (json.JSONDecodeError, ValueError, OSError) as exc: _logger.warning("kpi_dashboard: skipping %s: %s", slug, exc) score = None - lc = load_lifecycle_state(slug, sidecar_dir=sidecar_dir) + if score is not None: + quality_subjects.add((slug, score.subject_type)) + quality_rows.append((slug, sidecar_dir, score, None)) + + lifecycle_rows: list[tuple[str, Path, QualityScore | None, LifecycleState | None]] = [] + for slug in _iter_lifecycle_slugs(sources.sidecar_dir): + lc = load_lifecycle_state(slug, sidecar_dir=sources.sidecar_dir) + if lc is None: + continue + if (slug, lc.subject_type) not in quality_subjects: + lifecycle_rows.append((slug, sources.sidecar_dir, None, lc)) + + row_sources = sorted( + quality_rows + lifecycle_rows, + key=lambda item: (item[0], str(item[1]), item[3].subject_type if item[3] else ""), + ) + rows: list[EntityRow] = [] + for slug, sidecar_dir, score, lifecycle_override in row_sources: + lc = lifecycle_override + if lc is None and score is not None: + candidates = [sidecar_dir] + if sidecar_dir != sources.sidecar_dir: + candidates.append(sources.sidecar_dir) + for candidate_dir in candidates: + candidate = load_lifecycle_state(slug, sidecar_dir=candidate_dir) + if candidate is not None and candidate.subject_type == score.subject_type: + lc = candidate + break + elif lc is None: + lc = load_lifecycle_state(slug, sidecar_dir=sidecar_dir) if lc is not None: state = lc.state streak = lc.consecutive_d_count + lifecycle_subject_type = lc.subject_type else: state = STATE_ACTIVE streak = 0 + lifecycle_subject_type = None rows.append( _build_row( slug, score=score, + lifecycle_subject_type=lifecycle_subject_type, lifecycle_state=state, consecutive_d_count=streak, sources=sources, diff --git a/src/tests/test_ci_classifier.py b/src/tests/test_ci_classifier.py index cd34bec..98d8acc 100644 --- a/src/tests/test_ci_classifier.py +++ b/src/tests/test_ci_classifier.py @@ -544,7 +544,7 @@ def test_ci_required_rejects_missing_graph_check_on_graph_only_pr() -> None: } -def test_ci_required_rejects_missing_graph_check_on_unknown_graph_change() -> None: +def test_ci_required_allows_graph_check_skip_for_nonartifact_graph_change() -> None: needs = _required_needs( classify={ "result": "success", @@ -558,9 +558,7 @@ def test_ci_required_rejects_missing_graph_check_on_unknown_graph_change() -> No **{"graph-check": {"result": "skipped"}}, ) - assert failed_required_jobs(needs, event_name="pull_request") == { - "graph-check": "skipped", - } + assert failed_required_jobs(needs, event_name="pull_request") == {} def test_ci_required_allows_browser_skip_for_unrelated_pr_only() -> None: diff --git a/src/tests/test_ctx_monitor.py b/src/tests/test_ctx_monitor.py index c47b60c..72488e4 100644 --- a/src/tests/test_ctx_monitor.py +++ b/src/tests/test_ctx_monitor.py @@ -943,9 +943,11 @@ def fake_load(slug: str, entity_type: str = "skill") -> tuple[bool, str]: ) as response: loaded_html = response.read().decode("utf-8") csp = response.headers.get("Content-Security-Policy", "") + cookie = response.headers.get("Set-Cookie", "") assert "browser-token" not in loaded_html assert "Read-only mode" in loaded_html assert "script-src 'self' 'unsafe-inline'" in csp + assert "ctx_monitor_read_token=browser-token" in cookie with pytest.raises(urllib.error.HTTPError) as excinfo: urllib.request.urlopen( @@ -956,6 +958,17 @@ def fake_load(slug: str, entity_type: str = "skill") -> tuple[bool, str]: body = json.loads(excinfo.value.read().decode("utf-8")) assert "read token required" in body["detail"] + status, body = _get_raw( + port, + "/api/manifest.json", + headers={ + "Host": f"127.0.0.1:{port}", + "Cookie": "ctx_monitor_read_token=browser-token", + }, + ) + assert status == 200 + assert body["load"] == [] + status, body = _post_json( port, "/api/load", @@ -971,6 +984,34 @@ def fake_load(slug: str, entity_type: str = "skill") -> tuple[bool, str]: thread.join(timeout=2) +def test_serve_generates_read_token_for_non_loopback( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + class FakeServer: + _ctx_mutations_enabled = False + + def serve_forever(self) -> None: + raise KeyboardInterrupt + + def server_close(self) -> None: + return None + + monkeypatch.setattr(cm, "_MONITOR_TOKEN", "") + monkeypatch.setattr(cm, "_make_monitor_server", lambda _host, _port: FakeServer()) + monkeypatch.setattr(cm.secrets, "token_urlsafe", lambda _size: "lan-token") + monkeypatch.setattr(cm.socket, "gethostname", lambda: "devbox") + monkeypatch.setattr(cm.socket, "gethostbyname", lambda _name: "192.168.1.50") + + cm.serve(host="0.0.0.0", port=8765) + + assert cm._MONITOR_TOKEN == "lan-token" + out = capsys.readouterr().out + assert "http://192.168.1.50:8765/?token=lan-token" in out + assert "http://0.0.0.0:8765" not in out + assert "read token required" in out + + def test_host_allows_mutations_only_for_loopback() -> None: assert cm._host_allows_mutations("127.0.0.1") assert cm._host_allows_mutations("::1") @@ -2888,6 +2929,31 @@ def test_render_docs_lists_repo_docs( assert "doc-card" not in html_out +def test_render_docs_sanitizes_active_html( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + (tmp_path / "docs").mkdir() + (tmp_path / "mkdocs.yml").write_text( + "site_name: ctx\nnav:\n - Home: index.md\n", + encoding="utf-8", + ) + (tmp_path / "docs" / "index.md").write_text( + "# Home\n\n" + "\n\n" + "bad\n", + encoding="utf-8", + ) + monkeypatch.setattr(cm, "_docs_roots", lambda: [tmp_path]) + + html_out = cm._render_docs() + + assert "" not in html_out + assert "<script" in html_out + assert "onclick=" not in html_out + assert "href=\"javascript:" not in html_out + + def test_render_docs_falls_back_to_public_docs( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/src/tests/test_graph_artifact_guard.py b/src/tests/test_graph_artifact_guard.py index 7e8ae55..88258e4 100644 --- a/src/tests/test_graph_artifact_guard.py +++ b/src/tests/test_graph_artifact_guard.py @@ -85,3 +85,55 @@ def test_graph_artifact_guard_removes_only_stale_graph_files( assert not staged.exists() assert not partial.exists() assert outside.exists() + + +def test_graph_artifact_guard_prune_is_lfs_only_by_default( + tmp_path: Path, + monkeypatch, +) -> None: + repo_root = Path(__file__).resolve().parents[2] + guard = _load_guard(repo_root) + calls: list[tuple[str, ...]] = [] + + def fake_run_git( + _repo: Path, + args: list[str], + *, + check: bool = True, + capture: bool = False, + ) -> subprocess.CompletedProcess[str]: + calls.append(tuple(args)) + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + + monkeypatch.setattr(guard, "_run_git", fake_run_git) + + guard._prune(tmp_path, include_lfs=True, include_git_prune=False) + + assert ("lfs", "prune", "--verbose") in calls + assert not any(call[:1] == ("prune",) for call in calls) + + +def test_graph_artifact_guard_prune_requires_explicit_git_prune( + tmp_path: Path, + monkeypatch, +) -> None: + repo_root = Path(__file__).resolve().parents[2] + guard = _load_guard(repo_root) + calls: list[tuple[str, ...]] = [] + + def fake_run_git( + _repo: Path, + args: list[str], + *, + check: bool = True, + capture: bool = False, + ) -> subprocess.CompletedProcess[str]: + calls.append(tuple(args)) + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + + monkeypatch.setattr(guard, "_run_git", fake_run_git) + + guard._prune(tmp_path, include_lfs=False, include_git_prune=True) + + assert ("prune", "--expire=now", "--verbose") in calls + assert not any(call[:2] == ("lfs", "prune") for call in calls) diff --git a/src/tests/test_kpi_dashboard.py b/src/tests/test_kpi_dashboard.py index ecb9d29..1e42aea 100644 --- a/src/tests/test_kpi_dashboard.py +++ b/src/tests/test_kpi_dashboard.py @@ -193,6 +193,46 @@ def test_mcp_quality_subdir_is_included( ("filesystem", "mcp-server") ] + def test_lifecycle_only_mcp_keeps_subject_type( + self, sources: cl.LifecycleSources, + ) -> None: + _write_lifecycle( + sources.sidecar_dir, + "filesystem", + subject_type="mcp-server", + state=cl.STATE_ARCHIVE, + ) + + rows = kd.collect_rows(sources=sources) + + assert [(r.slug, r.subject_type, r.lifecycle_state) for r in rows] == [ + ("filesystem", "mcp-server", cl.STATE_ARCHIVE) + ] + + def test_lifecycle_only_mcp_is_not_dropped_by_same_slug_skill_quality( + self, sources: cl.LifecycleSources, + ) -> None: + _write_quality( + sources.sidecar_dir, + "langgraph", + subject_type="skill", + grade="B", + score=0.65, + ) + _write_lifecycle( + sources.sidecar_dir, + "langgraph", + subject_type="mcp-server", + state=cl.STATE_ARCHIVE, + ) + + rows = kd.collect_rows(sources=sources) + + assert sorted((r.slug, r.subject_type, r.grade, r.lifecycle_state) for r in rows) == [ + ("langgraph", "mcp-server", "", cl.STATE_ARCHIVE), + ("langgraph", "skill", "B", cl.STATE_ACTIVE), + ] + def test_duplicate_skill_and_mcp_slugs_are_distinct_rows( self, sources: cl.LifecycleSources, ) -> None: diff --git a/src/tests/test_resolve_graph_queries.py b/src/tests/test_resolve_graph_queries.py index dcee258..a719e37 100644 --- a/src/tests/test_resolve_graph_queries.py +++ b/src/tests/test_resolve_graph_queries.py @@ -341,6 +341,8 @@ def test_ann_overlay_replaces_existing_edge_scores_for_updated_node( assert edge["similarity_score"] == pytest.approx(0.8) assert edge["method"] == "ann_attach_v1" assert "shared_tags" not in edge + assert not G.has_edge("skill:A", "skill:C") + assert G.has_edge("skill:B", "skill:C") def test_entity_overlay_does_not_lower_existing_edge(self, tmp_path: Path) -> None: source = _build_simple_graph() diff --git a/src/tests/test_validate_graph_artifacts.py b/src/tests/test_validate_graph_artifacts.py index 16333d3..290d186 100644 --- a/src/tests/test_validate_graph_artifacts.py +++ b/src/tests/test_validate_graph_artifacts.py @@ -875,6 +875,20 @@ def test_scan_graph_json_rejects_out_of_range_edge_scores() -> None: _scan_graph_json(BytesIO(payload)) +@pytest.mark.parametrize("raw", ["NaN", "Infinity", "-Infinity"]) +def test_scan_graph_json_rejects_non_finite_edge_scores(raw: str) -> None: + payload = ( + b'{"nodes":[{"id":"skill:a","type":"skill"}],"edges":[' + b'{"source":"skill:a","target":"skill:b","semantic_sim":' + + raw.encode("ascii") + + b',"tag_sim":0.0,"token_sim":0.0,"weight":0.5,"final_weight":0.5}' + b"]}" + ) + + with pytest.raises(GraphArtifactError, match="semantic_sim must be finite"): + _scan_graph_json(BytesIO(payload)) + + def test_scan_graph_json_rejects_weight_final_weight_drift() -> None: graph = { "nodes": [{"id": "skill:a", "type": "skill"}], @@ -947,6 +961,29 @@ def test_scan_graph_json_rejects_score_component_drift() -> None: _scan_graph_json(BytesIO(payload)) +@pytest.mark.parametrize("value", [-0.1, 1.1]) +def test_scan_graph_json_rejects_out_of_range_score_components(value: float) -> None: + graph = { + "nodes": [{"id": "skill:a", "type": "skill"}], + "edges": [ + { + "source": "skill:a", + "target": "skill:b", + "weight": 0.5, + "final_weight": 0.5, + "score_components": { + "semantic": value, + "tag": 0.5 - value, + }, + }, + ], + } + payload = json.dumps(graph).encode("utf-8") + + with pytest.raises(GraphArtifactError, match="score_components must be 0..1"): + _scan_graph_json(BytesIO(payload)) + + @pytest.mark.parametrize("field", ["semantic_sim", "tag_sim", "token_sim"]) def test_overlay_validation_rejects_out_of_range_similarity_fields( tmp_path: Path, @@ -1019,6 +1056,35 @@ def test_overlay_validation_rejects_score_component_drift(tmp_path: Path) -> Non _validate_root_entity_overlay(tmp_path / "entity-overlays.jsonl") +@pytest.mark.parametrize("value", [-0.1, 1.1]) +def test_overlay_validation_rejects_out_of_range_score_components( + tmp_path: Path, + value: float, +) -> None: + (tmp_path / "entity-overlays.jsonl").write_text( + json.dumps({ + "nodes": [{"id": "skill:a"}], + "edges": [ + { + "source": "skill:a", + "target": "skill:b", + "weight": 0.5, + "final_weight": 0.5, + "score_components": { + "semantic": value, + "tag": 0.5 - value, + }, + }, + ], + }) + + "\n", + encoding="utf-8", + ) + + with pytest.raises(GraphArtifactError, match="score_components must be 0..1"): + _validate_root_entity_overlay(tmp_path / "entity-overlays.jsonl") + + def test_scan_graph_json_extracts_top_level_graph_export_id() -> None: graph = { "directed": False, diff --git a/src/validate_graph_artifacts.py b/src/validate_graph_artifacts.py index cc1df07..8d8ef81 100644 --- a/src/validate_graph_artifacts.py +++ b/src/validate_graph_artifacts.py @@ -6,6 +6,7 @@ import argparse import gzip import json +import math import re import sqlite3 import tarfile @@ -464,8 +465,35 @@ def _validate_graph_edge_score_fields(data: bytes) -> None: value = float(raw_value) except ValueError as exc: raise GraphArtifactError(f"graph.json edge {field} must be numeric") from exc + if not math.isfinite(value): + raise GraphArtifactError(f"graph.json edge {field} must be finite") if not 0 <= value <= 1: raise GraphArtifactError(f"graph.json edge {field} must be 0..1") + _validate_graph_edge_score_objects(data) + + +def _validate_graph_edge_score_objects(data: bytes) -> None: + try: + graph = json.loads(data) + except json.JSONDecodeError: + return + edges = graph.get("edges") if isinstance(graph, dict) else None + if not isinstance(edges, list): + return + for edge in edges: + if not isinstance(edge, dict): + continue + for field in _EDGE_SCORE_FIELDS: + if field not in edge: + continue + value = edge[field] + if not isinstance(value, int | float): + raise GraphArtifactError(f"graph.json edge {field} must be numeric") + numeric = float(value) + if not math.isfinite(numeric): + raise GraphArtifactError(f"graph.json edge {field} must be finite") + if not 0 <= numeric <= 1: + raise GraphArtifactError(f"graph.json edge {field} must be 0..1") def _validate_graph_edge_weight_drift(data: bytes) -> None: @@ -500,11 +528,18 @@ def _validate_score_component_mapping( return if not isinstance(final_weight, int | float) or not isinstance(components, dict): raise GraphArtifactError(f"{context} score_components must sum to final_weight") + if not math.isfinite(float(final_weight)): + raise GraphArtifactError(f"{context} final_weight must be finite") numeric_components: list[float] = [] for value in components.values(): if not isinstance(value, int | float): raise GraphArtifactError(f"{context} score_components must be numeric") - numeric_components.append(float(value)) + numeric = float(value) + if not math.isfinite(numeric): + raise GraphArtifactError(f"{context} score_components must be finite") + if numeric < 0.0 or numeric > 1.0: + raise GraphArtifactError(f"{context} score_components must be 0..1") + numeric_components.append(numeric) _validate_score_component_sum( float(final_weight), numeric_components, @@ -525,9 +560,14 @@ def _validate_score_component_bytes( component_values: list[float] = [] for field_match in _SCORE_COMPONENT_VALUE_RE.finditer(raw_components): try: - component_values.append(float(field_match.group(1))) + component = float(field_match.group(1)) except ValueError as exc: raise GraphArtifactError(f"{context} score_components must be numeric") from exc + if not math.isfinite(component): + raise GraphArtifactError(f"{context} score_components must be finite") + if component < 0.0 or component > 1.0: + raise GraphArtifactError(f"{context} score_components must be 0..1") + component_values.append(component) if not component_values: raise GraphArtifactError(f"{context} score_components must sum to final_weight") _validate_score_component_sum(final_weight, component_values, context=context) @@ -539,6 +579,8 @@ def _validate_score_component_sum( *, context: str, ) -> None: + if not math.isfinite(final_weight): + raise GraphArtifactError(f"{context} final_weight must be finite") component_total = min(sum(component_values), 1.0) if abs(component_total - final_weight) > _SCORE_COMPONENT_TOLERANCE: raise GraphArtifactError(f"{context} score_components must sum to final_weight")