diff --git a/authbridge/README.md b/authbridge/README.md index 7d8fa1424..1d9b2102d 100644 --- a/authbridge/README.md +++ b/authbridge/README.md @@ -387,8 +387,9 @@ The easiest way to get all prerequisites is to use the [Rossoctl Quickstart](htt - [Manual deployment](./demos/github-issue/demo-manual.md) — deploy everything via `kubectl` and YAML manifests - [UI deployment](./demos/github-issue/demo-ui.md) — import agent and tool via the Rossoctl dashboard - **[Token-Exchange Routes](./demos/token-exchange-routes/README.md)** - Configuration reference for the `authproxy-routes` ConfigMap; covers single-target (one route) and multi-target (one agent → many tools) patterns +- **[Lineage Demo](./demos/lineage/README.md)** - Per-request data lineage from the sidecar on the Weather Agent pair, attached to the running Deployments with the [lineage attach kit](./lineage-attach/README.md); shows the same turn fragmented and then as one trace (no Keycloak involved) -All demos cover configuring Keycloak, deploying, and testing. +All demos except the Lineage Demo cover configuring Keycloak, deploying, and testing. ### Route-Based Configuration diff --git a/authbridge/demos/README.md b/authbridge/demos/README.md index a31ce0403..8d78ea3cc 100644 --- a/authbridge/demos/README.md +++ b/authbridge/demos/README.md @@ -25,6 +25,8 @@ more AuthBridge capabilities. | **[abctl Walkthrough](weather-agent/demo-with-abctl.md)** | Reference | Watch the AuthBridge plugin pipeline live with the `abctl` TUI | Tooling only | | **[IBAC](ibac/README.md)** | Intermediate | Intent-Based Access Control: LLM judge denies outbound HTTP that doesn't align with the user's recorded intent. Reproduces the email-poison / prompt-injection attack from `huang195/ibac`; chat with the agent through the rossoctl UI and see the exfiltration blocked, then `make show-result` for a pipeline-level forensic | UI + kubectl | | **[SPARC (finance)](finance-sparc/README.md)** | Intermediate | SPARC pre-tool reflection: the `sparc` plugin blocks a hallucinated/ungrounded tool argument (an invented transaction id) before it executes and transparently asks the user to clarify, then approves the corrected call. Complements IBAC — SPARC verifies argument grounding, IBAC verifies intent alignment | UI + kubectl | +| **[Lineage](lineage/README.md)** | Intermediate | Per-request lineage on the Weather Agent pair: attach the sidecar with the lineage attach kit and see one turn first as 19 separate traces (the app forwards no `traceparent`), then as one trace of 70 spans with one root once the app's own propagation is switched on. Deploys the two stock images plain; edits nothing else | kubectl + scripts | +| **[Lineage attach kit](../lineage-attach/README.md)** | Reference | Attach per-request lineage to any existing Deployment: enable the `lineage-telemetry` plugin and every HTTP exchange becomes two facts-only spans (`request` + `response`, paired by `lineage.exchange.id`) sent to **any** OTLP consumer. A strategic-merge patch + ConfigMap, generated and validated; a propagate-only OTel shim for uninstrumented Python apps, activated by one env var | kubectl + scripts | | **[CPEX Bridge (HR)](hr-cpex/README.md)** | Advanced | CPEX/APL declarative policy: one route chains a coarse APL predicate, an embedded Cedar PDP, RFC 8693 token exchange with a post-check, PII redaction and audit plugins. Same request, different data per caller (Bob sees an SSN, Eve gets it redacted). Self-contained: its own kind cluster + namespace, deployed via `make` rather than operator injection | [kubectl (make)](hr-cpex/README.md#quick-start) | ## Recommended Path @@ -133,8 +135,8 @@ Cluster-backed demos (everything above except the session-budget local walkthrough) require: - A Kubernetes cluster with the Rossoctl platform installed ([Installation Guide](https://github.com/rossoctl/rossoctl/blob/main/docs/getting-started/install.md)) -- Keycloak deployed in the `keycloak` namespace -- SPIRE deployed (for demos using SPIFFE identity) +- Keycloak deployed in the `keycloak` namespace (not used by the Lineage demo) +- SPIRE deployed (for demos using SPIFFE identity; not used by the Lineage demo) UI-based demos additionally require: - The Rossoctl UI running at `http://rossoctl-ui.localtest.me:8080` diff --git a/authbridge/demos/lineage/README.md b/authbridge/demos/lineage/README.md new file mode 100644 index 000000000..3ae7c5f71 --- /dev/null +++ b/authbridge/demos/lineage/README.md @@ -0,0 +1,220 @@ +# Lineage demo — the weather agent and its tool, with per-request lineage + +The [Weather Agent](../weather-agent/demo-ui.md) pair from `rossoctl/examples` +— an A2A agent that asks an LLM and calls one MCP tool — deployed plain, then +given per-request lineage with the [lineage attach kit](../../lineage-attach/README.md) +and nothing else. Six steps. You will see the same turn twice: first as +**19 separate traces** (the entry alone and each of the app's 18 calls in a +trace of its own, because the app does not carry `traceparent`), then as +**one trace of 70 spans** with one root, after the +app's own propagation is switched on. Nothing about the app is edited except +one environment variable that the app itself defines. + +Read [the kit's README](../../lineage-attach/README.md) for what the spans +carry and [DESIGN](../../lineage-attach/DESIGN.md) for why propagation is the +app's job; this page is only the walk-through. + +## Prerequisites + +- The rossoctl platform on kind (`rossoctl` cluster, Kubernetes 1.29 or newer — + the kit attaches the sidecar as a native sidecar), namespace `team1` with + its platform-rendered `envoy-config` ConfigMap, and the platform collector + (`deploy/otel-collector` in `rossoctl-system`, stock `debug` exporter). +- A sidecar image that carries `lineage-telemetry` (cortex #761): until a release does, + [RECIPE step 1](../../lineage-attach/RECIPE.md#1-a-sidecar-image-that-carries-the-plugin-once-per-cluster-until-a-release-does) + builds and loads it (run it from the kit's directory; its `cd ..` is + relative to there). Then, back in this directory, for the whole session: + + ```sh + export SIDECAR_IMAGE=docker.io/library/authbridge-envoy:latest PROXY_INIT_IMAGE=docker.io/library/proxy-init:latest + export KIT=../../lineage-attach + ``` +- An LLM the agent can reach over **plaintext HTTP** (an HTTPS LLM is TLS + passthrough: the sidecar records no hop for it). Default: Ollama on the host + with `qwen2.5:7b` (`ollama pull qwen2.5:7b`); edit `k8s/weather.yaml`'s + `weather-llm` ConfigMap for anything else. +- Egress from the cluster to `https://wttr.in`, which the tool queries. +- `kubectl` and `python3` on the host (`ask.sh`, `show-trace.py`). +- Permissions, all free to a kind admin: `ask.sh` creates a pod in `team1`; + `show-trace.py` gets `deployments` and reads `pods/log` in `rossoctl-system`; + steps 2, 4 and 6 get and patch Deployments (step 2 with a server-side dry + run first) and watch the rollout, and steps 2 and 6 get, create or delete + ConfigMaps, in `team1`. + +Run everything from this directory. + +## 1. Deploy the pair, plain + +```sh +kubectl apply -f k8s/weather.yaml +kubectl -n team1 rollout status deploy/weather-tool && kubectl -n team1 rollout status deploy/weather-service +./ask.sh "What is the weather in Paris?" +``` + +`k8s/weather.yaml` is the two stock images (`weather_service`, `weather_tool`) +as two Deployments and two Services — no `AgentRuntime`, no platform sidecar, +no auth. `ask.sh` sends one A2A `message/send` from a pod inside the cluster +(a port-forward would bypass the sidecar) with a `traceparent` whose trace id +it prints. Pass: an `answer:` line with the weather. Nothing is captured yet: +no sidecar is attached. (The tool's own OTel spans do reach the collector +from step 1 — its code defaults the endpoint — but those are the app's spans, +not lineage.) + +## 2. Attach lineage (capture) + +```sh +for d in weather-tool weather-service; do NAMESPACE=team1 DEPLOY=$d CAPTURE_IO=true $KIT/sidecar-patch.sh; done +``` + +Pass — each prints, in this order (rollout progress lines omitted): + +``` +configmap/authbridge-lineage-config- created +deployment.apps/ patched +>> back out: kubectl -n team1 patch deploy/ --type strategic -p '' && kubectl -n team1 delete cm authbridge-lineage-config- +deployment "" successfully rolled out +>> lineage sidecar attached to deploy/ (self_id=, ns=team1) +``` + +Two `NOTE:` blocks precede that on stderr: the sidecar records every hop but +attribution is the app's job (which is what steps 3 to 5 show), and +`CAPTURE_IO=true` sends parsed content to the collector over plain gRPC +(in-cluster here). That is the whole attachment: a ConfigMap and a strategic-merge patch per Deployment, both +generated by the kit. Both pods are now `2/2`. `CAPTURE_IO=true` is the demo's +choice, not the kit's default: the spans then carry the question, the tool +arguments and the prompts, so the trace reads as a story. Demo only — with the +stock `debug` exporter that content lands in the collector's pod log, readable +by anyone with `pods/log` in `rossoctl-system`, for the log's lifetime. The +back-out line is right at any later time; step 6 uses it. + +## 3. One turn — every hop alone + +```sh +./ask.sh "What is the weather in Paris?" # prints: trace id: +./show-trace.py +``` + +`show-trace.py` reads the collector's log and lists the sidecar spans of one +trace. Measured: + +``` +2 sidecar spans, 1 exchanges: 1 inbound a2a +parent.source: 1 wire, 0 tracestate, 0 none +traces begun by an unparented outbound hop while this one was in flight: 18 +shape: ENTRY ONLY — nothing the app called landed here; its calls are the stray traces above +``` + +The sidecar saw everything the turn did — 35 exchanges: the A2A entry, 16 MCP +exchanges to the tool (session handshakes, tool listing, the call), 2 LLM +calls, and the tool's 16 inbound sides — and recorded all 70 spans. But the +app forwarded no `traceparent`, so the entry is alone in your trace and each +of the app's 18 calls started a trace of its own: its sidecar found nothing on +the wire to parent on (`parent.source=none`), forwarded a `traceparent` of its +own making, and the tool's side of each MCP call joined *that* trace — 19 +traces, each internally consistent and each useless, because nothing links a +call to the question that caused it. This is the case DESIGN calls *the one +that looks fine and is not*: count spans and it passes; read the shape and it +fails. (The counts are one run's — the LLM may plan a turn differently and +add or drop an MCP exchange; the shape lines are what hold from run to run.) + +## 4. Switch the app's propagation on + +The weather agent ships its own OpenTelemetry setup, activated by one +variable it defines: when `OTEL_EXPORTER_OTLP_ENDPOINT` is set it extracts the +inbound `traceparent` and instruments `httpx`, so its LLM and tool calls carry +it. Point it at the platform collector's OTLP/HTTP receiver — port 8335 on the +stock chart, not 4318 — on the app container only: + +```sh +kubectl -n team1 set env deploy/weather-service -c agent OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.rossoctl-system.svc.cluster.local:8335 +kubectl -n team1 rollout status deploy/weather-service +``` + +Three collector ports are in play and two of them are right: the apps export +OTLP/HTTP to `8335`, the plugin exports OTLP/gRPC to `4317` (the kit's +default `OTEL_ENDPOINT`), and the Service's `4318` has nothing behind it on +the stock chart. Both apps' own exports go out through their lineage sidecars +without an `OUTBOUND_PORTS_EXCLUDE`; that works because the collector's host +is on the plugin's default `bypass_hosts`, so the sidecar records no span for +them. + +> **Why not the kit's shim here?** Because the interlock refuses this image, +> correctly: after `podman pull ghcr.io/rossoctl/examples/weather_service:latest` +> (the interlock probes a local image; an absent one is refused with a +> different message), `$KIT/build-otel-shim.sh ghcr.io/rossoctl/examples/weather_service:latest` +> exits 3 with `REFUSING to bake …: it already instruments httpx`. An app that +> brings its own instrumentation gets its own switch; the shim is for the app +> that brings none ([RECIPE step 2](../../lineage-attach/RECIPE.md#2-bake-the-propagation-shim-onto-the-app-image-once-per-image)). +> The tool image bakes, but it does not need to: its one call that matters is +> HTTPS to `wttr.in`, which the sidecar passes through unseen, and its own +> OTLP export (the tool's code defaults the endpoint to the collector's 8335; +> `k8s/weather.yaml` states it) is bypassed as above, so capture is all it +> needs. + +## 5. The same turn — one trace + +```sh +./ask.sh "What is the weather in Paris?" +./show-trace.py +``` + +Measured: + +``` +70 sidecar spans, 35 exchanges: 1 inbound a2a, 16 inbound mcp, 2 outbound inference, 16 outbound mcp +parent.source: 1 wire, 34 tracestate, 0 none +traces begun by an unparented outbound hop while this one was in flight: 0 +shape: OK — one root, unstamped only at the entry, the app's calls are in this trace +``` + +The table above those lines is the turn, hop by hop, in time order: the A2A +entry (`wire` — the caller minted the trace), then each MCP exchange seen +twice (outbound at the agent, inbound at the tool, both `tracestate`), the two +LLM calls (`inference`, peer `host.containers.internal:11434`), and the A2A +response last. The apps' own spans arrive in the same trace too — the agent's +A2A server, LangChain and `openai.chat` spans and its `httpx` `POST`s, the +tool's `tools/list` and `tools/call` — 82 of them on this turn, 78 from the +agent and 4 from the tool; the sidecar's are the ones with `lineage.*` +attributes. + +## 6. Back out + +Run the two back-out lines step 2 printed. Each is a strategic-merge reverse +patch that deletes, by name, exactly what the attach added, then deletes the +ConfigMap; it is right at any later revision, so the roll step 4 caused does +not matter. If the printed lines are gone, the kit regenerates them: + +```sh +for d in weather-tool weather-service; do + kubectl -n team1 patch deploy/$d --type strategic -p "$(EMIT=undo NAME=$d NAMESPACE=team1 $KIT/attach-lineage.sh)" \ + && kubectl -n team1 delete cm authbridge-lineage-config-$d +done +kubectl -n team1 set env deploy/weather-service -c agent OTEL_EXPORTER_OTLP_ENDPOINT- +``` + +Pass: `kubectl -n team1 get pods` shows both pods `1/1` again. The last line +removes step 4's variable, so the agent is plain again too. Then the app: +`kubectl delete -f k8s/weather.yaml`. + +## Files + +| file | what | +|---|---| +| `k8s/weather.yaml` | the pair, plain: two Deployments, two Services, one ConfigMap for the LLM | +| `ask.sh` | one A2A turn from inside the cluster with a chosen trace id | +| `show-trace.py` | the shape of one trace from the collector log, with a verdict; exit 0 only for one root with the app's calls inside it, 1 for no spans, 2 for the wrong shape | + +Everything that attaches lineage is the kit's; this directory holds only the +application and the two readers. + +## If it does not work + +| symptom | cause | +|---|---| +| `ask.sh` prints no answer, or the agent logs `Cannot connect to MCP` | the tool is not ready, or `MCP_URL` in `k8s/weather.yaml` does not match the Service name | +| the answer is an LLM error | `weather-llm` ConfigMap: the base URL is not reachable from a pod (podman kind: `host.containers.internal`; docker kind: `host.docker.internal`), or the model is not pulled | +| `show-trace.py` says `no sidecar spans for … in the last 10m` | most often the window: `--since` defaults to `10m` and a turn can take minutes — pass `--since 1h`. Otherwise the sidecar image predates the plugin, or the collector was restarted — `kubectl -n team1 logs deploy/weather-service -c envoy-proxy` | +| step 5 still says `ENTRY ONLY` (or `FRAGMENTED`) | the agent did not restart with the variable — `kubectl -n team1 logs deploy/weather-service -c agent \| grep 'httpx instrumented'` | +| the agent logs export failures after step 4 | wrong collector port: the stock chart serves OTLP/HTTP on `8335`, not `4318`; propagation works regardless, but the agent's own spans do not arrive | +| the answer is a tool error about `wttr.in` | the tool needs egress to `https://wttr.in`; the sidecar passes HTTPS through, so this is cluster egress, not lineage | +| anything about the attachment itself | the kit's [Troubleshooting](../../lineage-attach/README.md#troubleshooting) | diff --git a/authbridge/demos/lineage/ask.sh b/authbridge/demos/lineage/ask.sh new file mode 100755 index 000000000..b70b38403 --- /dev/null +++ b/authbridge/demos/lineage/ask.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# ask.sh — one A2A turn to the weather agent, sent from inside the cluster (a +# port-forward would bypass the sidecar) with a trace id of our choosing, so +# the spans it produces can be found by that id. +# +# The request is the A2A 0.3 JSON-RPC shape (`message/send`, `parts: [{kind: +# text}]`); the agent runs a2a-sdk 1.x and answers it through its 0.3 +# compatibility routes. The result is a Task (answer in status.message or the +# artifacts) or a Message (answer in parts); both are read. +# +# Usage: ./ask.sh ["What is the weather in Paris?"] +# NS=team1 SVC=weather-service PORT=8080 override the target. +# A turn is given --max-time 300 (curl); the trace exists even if the +# answer arrives later than that. +set -euo pipefail +NS="${NS:-team1}"; SVC="${SVC:-weather-service}"; PORT="${PORT:-8080}" +question="${1:-What is the weather in Paris?}" +trace_id="$(python3 -c 'import secrets; print(secrets.token_hex(16))')" +body="$(python3 -c 'import json, sys, uuid +print(json.dumps({"jsonrpc": "2.0", "id": "1", "method": "message/send", "params": {"message": { + "role": "user", "messageId": uuid.uuid4().hex, "parts": [{"kind": "text", "text": sys.argv[1]}]}}}))' "$question")" +echo "trace id: ${trace_id}" +kubectl -n "$NS" run "ask-${trace_id:0:12}" --rm -i --quiet --restart=Never --image=curlimages/curl:8.11.1 -- \ + curl -sS --max-time 300 -H 'content-type: application/json' \ + -H "traceparent: 00-${trace_id}-0000000000000001-01" \ + -d "$body" "http://${SVC}:${PORT}/" \ + | python3 -c 'import json, sys +raw = sys.stdin.read() +try: + r = json.loads(raw) +except ValueError: + # curl failed (nothing came back) or the agent answered with non-JSON: + # show what arrived, not a traceback; kubectl/curl said why above. + sys.exit("no answer: " + (raw.strip()[:300] or "empty response — see the error above")) +res = r.get("result", r) +parts = (res.get("status", {}).get("message", {}).get("parts", []) + or [p for a in res.get("artifacts", []) for p in a.get("parts", [])] + or res.get("parts", [])) +print("answer:", " ".join(p.get("text", "") for p in parts) or json.dumps(r)[:300])' +echo "trace id: ${trace_id} (kubectl -n rossoctl-system logs deploy/otel-collector | grep -c ${trace_id})" diff --git a/authbridge/demos/lineage/k8s/weather.yaml b/authbridge/demos/lineage/k8s/weather.yaml new file mode 100644 index 000000000..625f9db6b --- /dev/null +++ b/authbridge/demos/lineage/k8s/weather.yaml @@ -0,0 +1,202 @@ +# The weather pair from rossoctl/examples, deployed plain: no AgentRuntime, no +# platform sidecar, no auth — exactly the shape the lineage attach kit adopts. +# Two stock images, mirroring demos/weather-agent/k8s/*-advanced.yaml minus the +# enrollment labels and the auth ConfigMaps. One deliberate difference: the +# stock agent annotates its LLM port (11434) out of the sidecar's redirect; +# here it is left in, because the LLM hop is exactly what lineage should see. +# Point the agent at your LLM in the ConfigMap (host Ollama by default; +# qwen2.5:7b answers the demo's question with one tool call). +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: weather-llm + namespace: team1 +data: + # podman-backed kind reaches the host as host.containers.internal; + # docker-backed kind as host.docker.internal. OpenAI: https://api.openai.com/v1 + # (an HTTPS LLM is TLS passthrough — the sidecar records no hop for it). + LLM_API_BASE: "http://host.containers.internal:11434/v1" + # "ollama" is a placeholder, not a credential. A real key belongs in a Secret + # (kubectl create secret generic weather-llm-key --from-literal=LLM_API_KEY=…) + # referenced from the agent's envFrom as secretRef, not in this ConfigMap. + LLM_API_KEY: "ollama" + LLM_MODEL: "qwen2.5:7b" +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: weather-service + namespace: team1 + labels: + app.kubernetes.io/name: weather-service +spec: + # Cold pulls on kind can exceed the 600s default (as the stock manifests note). + progressDeadlineSeconds: 1800 + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app.kubernetes.io/name: weather-service + template: + metadata: + labels: + app.kubernetes.io/name: weather-service + spec: + # The image runs as 1001 (its Dockerfile); match it at pod level. + securityContext: + runAsNonRoot: true + runAsUser: 1001 + runAsGroup: 1001 + fsGroup: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: agent + image: ghcr.io/rossoctl/examples/weather_service:latest + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + ports: + - containerPort: 8000 + name: http + env: + - name: PORT + value: "8000" + - name: HOST + value: "0.0.0.0" + - name: UV_CACHE_DIR + value: "/app/.cache" + - name: MCP_URL + value: "http://weather-tool-mcp:8000/mcp" + - name: LOG_LEVEL + value: "INFO" + envFrom: + - configMapRef: + name: weather-llm + volumeMounts: + - name: cache + mountPath: /app/.cache + - name: marvin + mountPath: /.marvin + readinessProbe: + tcpSocket: + port: 8000 + initialDelaySeconds: 10 + periodSeconds: 5 + failureThreshold: 30 + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + volumes: + - name: cache + emptyDir: {} + - name: marvin + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: weather-service + namespace: team1 +spec: + selector: + app.kubernetes.io/name: weather-service + ports: + - name: http + port: 8080 + targetPort: 8000 +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: weather-tool + namespace: team1 + labels: + app.kubernetes.io/name: weather-tool +spec: + progressDeadlineSeconds: 1800 + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app.kubernetes.io/name: weather-tool + template: + metadata: + labels: + app.kubernetes.io/name: weather-tool + spec: + securityContext: + runAsNonRoot: true + runAsUser: 1001 + runAsGroup: 1001 + fsGroup: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: mcp + image: ghcr.io/rossoctl/examples/weather_tool:latest + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + ports: + - containerPort: 8000 + name: http + env: + - name: PORT + value: "8000" + - name: HOST + value: "0.0.0.0" + - name: UV_NO_CACHE + value: "1" + # The tool's own OTel export. Its code defaults to exactly this + # value when the variable is absent; stated here so the reader can + # see where the tool's spans go without opening the image. + - name: OTEL_EXPORTER_OTLP_ENDPOINT + value: "http://otel-collector.rossoctl-system.svc.cluster.local:8335" + volumeMounts: + - name: cache + mountPath: /app/.cache + - name: tmp + mountPath: /tmp + readinessProbe: + tcpSocket: + port: 8000 + initialDelaySeconds: 5 + periodSeconds: 5 + failureThreshold: 24 + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 500m + memory: 512Mi + volumes: + - name: cache + emptyDir: {} + - name: tmp + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: weather-tool-mcp + namespace: team1 +spec: + selector: + app.kubernetes.io/name: weather-tool + ports: + - name: http + port: 8000 + targetPort: 8000 diff --git a/authbridge/demos/lineage/show-trace.py b/authbridge/demos/lineage/show-trace.py new file mode 100755 index 000000000..d0e03cdaa --- /dev/null +++ b/authbridge/demos/lineage/show-trace.py @@ -0,0 +1,185 @@ +#!/usr/bin/env python3 +"""Print the shape of one trace from the platform collector's debug log. + +The stock collector prints every span it receives (debug exporter, +verbosity: detailed). This reads that log, keeps the sidecar's spans for one +trace id and lists them in time order — self id, direction, protocol, role, +where the parent came from, the peer — then says whether the shape is right: +one unstamped hop at the entry (an inbound: `wire` when the caller sent a +`traceparent`, `none` when it sent nothing), `tracestate` everywhere else, and +the app's own outbound calls *in this trace*. An entry alone is not a good +shape: it means the app's calls went to traces of their own, and those are +counted too. + +Usage: ./show-trace.py [--since 10m] [--namespace rossoctl-system] + +Exit status: 0 the good shape (one root, the app's calls inside), 1 no sidecar +spans for that id in the window (or the collector log could not be read), 2 +the wrong shape (ENTRY ONLY or FRAGMENTED). +""" + +import argparse +import collections +import re +import subprocess +import sys + +ATTR = re.compile(r"-> ([\w.]+): Str\((.*)\)$", re.M) +TRACE = re.compile(r"Trace ID\s*:\s*(\w+)") +START = re.compile(r"Start time\s*:\s*(\S+ \S+)") + +# Blocks the parser could not trust, reported once at the end: no parseable +# start time, or a lineage.* key seen twice with different values. +skipped_no_start = 0 +ambiguous = 0 + + +def facts(block: str) -> dict: + """The block's attributes, robust to captured content. + + With capture_io on, input.value / output.value hold user content printed + verbatim, newlines included; a line of it shaped like `-> lineage.role: + Str(response)` would otherwise be read as a fact. The plugin emits every + fact except lineage.parent.source BEFORE the captured value, and + lineage.parent.source after it, so the first occurrence wins for the + former and the last for the latter; a key seen twice with different + values is counted as ambiguous.""" + global ambiguous + attrs: dict = {} + seen_twice = False + for key, value in ATTR.findall(block): + if key in attrs and attrs[key] != value: + seen_twice = True + if key == "lineage.parent.source": + attrs[key] = value + else: + attrs.setdefault(key, value) + if seen_twice: + ambiguous += 1 + return attrs + + +def sidecar_blocks(log: str): + """Yield the debug-exporter span blocks that carry a `lineage.*` attribute.""" + for block in re.split(r"\n(?=Span #\d+)", log): + if "lineage.role" in block: + yield block + + +def stray_outbound_traces(log: str, trace_id: str, first: str, last: str) -> int: + """Traces other than trace_id that begin with an outbound hop that had no + stamp to parent on (`wire` or `none`) — an app call that started a trace + of its own — started while trace_id was in flight (between its first and + last span).""" + strays = set() + for block in sidecar_blocks(log): + attrs = facts(block) + tid = TRACE.search(block) + start = START.search(block) + if ( + tid + and start + and tid.group(1) != trace_id + and first <= start.group(1) <= last + and attrs.get("lineage.role") == "request" + and attrs.get("lineage.direction") == "outbound" + and attrs.get("lineage.parent.source") in ("wire", "none") + ): + strays.add(tid.group(1)) + return len(strays) + + +def spans_for(log: str, trace_id: str): + """The sidecar spans of one trace as rows, in start-time order. + + A block whose start time does not parse is skipped (and counted) rather + than sorted first as an empty string: the first and last timestamps bound + the stray window, and an empty one would open it to the start of the log.""" + global skipped_no_start + rows = [] + for block in sidecar_blocks(log): + tid = TRACE.search(block) + if not tid or tid.group(1) != trace_id: + continue + start = START.search(block) + if not start: + skipped_no_start += 1 + continue + attrs = facts(block) + rows.append( + ( + start.group(1), # full timestamp: sorts across midnight + attrs.get("lineage.self.id", ""), + attrs.get("lineage.direction", ""), + attrs.get("lineage.protocol", ""), + attrs.get("lineage.role", ""), + attrs.get("lineage.parent.source", ""), + attrs.get("lineage.peer.host", "")[:30], + attrs.get("lineage.outcome", ""), + ) + ) + rows.sort() + return rows + + +def main() -> int: + """Read the collector log, print the trace's rows and totals, judge the shape.""" + ap = argparse.ArgumentParser() + ap.add_argument("trace_id") + ap.add_argument("--since", default="10m", help="collector log window (kubectl --since)") + ap.add_argument("--namespace", default="rossoctl-system") + args = ap.parse_args() + try: + log = subprocess.run( # nosec B603 B607 — fixed argv, no shell + ["kubectl", "-n", args.namespace, "logs", "deploy/otel-collector", "--since", args.since], + check=True, + capture_output=True, + text=True, + ).stdout + except subprocess.CalledProcessError as exc: + # kubectl's own message is the useful one (no such deployment, no + # pods/log permission, no such namespace) — show it, not a traceback. + print(f"kubectl logs failed (exit {exc.returncode}): {exc.stderr.strip()}", file=sys.stderr) + return 1 + rows = spans_for(log, args.trace_id) + if skipped_no_start: + print(f"warning: {skipped_no_start} span block(s) of this trace had no start time; skipped", file=sys.stderr) + if not rows: + print(f"no sidecar spans for {args.trace_id} in the last {args.since}", file=sys.stderr) + return 1 + print(f"{'time':<13}{'self':<17}{'dir':<10}{'proto':<11}{'role':<10}{'parent':<12}{'peer':<31}outcome") + for r in rows: + print(f"{r[0][11:23]:<13}{r[1]:<17}{r[2]:<10}{r[3]:<11}{r[4]:<10}{r[5]:<12}{r[6]:<31}{r[7]}") + requests = [r for r in rows if r[4] == "request"] + parents = collections.Counter(r[5] for r in requests) + by_proto = collections.Counter(f"{r[2]} {r[3]}" for r in requests) + strays = stray_outbound_traces(log, args.trace_id, rows[0][0], rows[-1][0]) + if ambiguous: + print( + f"warning: {ambiguous} span block(s) repeated a lineage.* key with different values;" + " captured content may hold a fact-shaped line (first occurrence used, last for parent.source)", + file=sys.stderr, + ) + print() + mix = ", ".join(f"{n} {k}" for k, n in sorted(by_proto.items())) + print(f"{len(rows)} sidecar spans, {len(requests)} exchanges: {mix}") + wire, stamped, none = parents.get("wire", 0), parents.get("tracestate", 0), parents.get("none", 0) + print(f"parent.source: {wire} wire, {stamped} tracestate, {none} none") + print(f"traces begun by an unparented outbound hop while this one was in flight: {strays}") + has_outbound = any(r[2] == "outbound" for r in requests) + # The one unstamped hop must be the first request AND an inbound: an + # unstamped outbound root is a stray trace, whatever else it holds. + unstamped = parents.get("wire", 0) + parents.get("none", 0) + one_root = unstamped == 1 and requests[0][5] in ("wire", "none") and requests[0][2] == "inbound" + if one_root and has_outbound and strays == 0: + print("shape: OK — one root, unstamped only at the entry, the app's calls are in this trace") + return 0 + if not has_outbound: + print("shape: ENTRY ONLY — nothing the app called landed here; its calls are the stray traces above") + else: + print("shape: FRAGMENTED — an unstamped non-entry hop, an outbound root or strays mark un-propagated calls") + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/authbridge/lineage-attach/README.md b/authbridge/lineage-attach/README.md index 329889181..b3c9815e8 100644 --- a/authbridge/lineage-attach/README.md +++ b/authbridge/lineage-attach/README.md @@ -35,7 +35,8 @@ remembers writing. > gives a graceful parsers-only sidecar (the parsers predate the plugin). **Start here:** [RECIPE.md](RECIPE.md) — six steps, expected output, back -out. **Why it works and where it stops:** [DESIGN.md](DESIGN.md). +out. **Why it works and where it stops:** [DESIGN.md](DESIGN.md). **See it +run:** the [lineage demo](../demos/lineage/README.md) on the Weather Agent pair. ---