AIOps incident-intelligence reference architecture: HAVE/PARTIAL/GAP conformance + 3-pipeline build program

[mdheller]

AIOps incident-intelligence reference architecture — estate conformance + build program

Audit of the estate against an IBM Watson-AIOps-style incident-intelligence reference architecture, grouped into its 3 pipelines, plus a build program for the sovereign delta. Consume-not-fork: we build our own equivalents (MIT/Apache, stdlib) — no IBM/Watson proprietary code is vendored. SHA-256 references are the FIPS-180-4 algorithm, not a FIPS-140 module claim.

Grounded in ~/dev files + prior live-cluster facts (telemetry #1161, self-heal #518, wordops #1171). Paths cited are canonical prophet-platform unless noted; many capabilities appear duplicated across *.wt worktrees.

Tally

• HAVE (14): Kafka bus, MinIO object store, Postgres/persistence, Loki/Prometheus/promtail/OTel + observability ns, telemetry-capture-liveness, log-stream input, PagerDuty/Alertmanager alert ingestion, Localization Service (GBRG blast-radius), receipt/provenance spine, The Assay (ok/sad/bad), self-heal control loop, resilience runner, administrator actor (WordOps A0–A4 lease governance).
• PARTIAL (10): Similar Incidents Service (contract slice landed — PR feat(story_services): incident-similarity contract + deterministic scorer (AI4IT Pipeline 3) #33), Topology Service/ASM, Elastic/OpenSearch, Tempo traces, ChatOps Integrator+Orchestrator, Connection Controller/gateway, Training Console + model-train, chat-user actor, Orchestrator (named), NOI note.
• GAP (5): Anomaly (Flink) Core / stream-processing engine, Log Anomaly Detector (drain3/LAD), Event Grouping Service (alert correlation → stories), NOI/Netcool event adapter, near-duplicate/dedup (MinHash/SimHash) primitive.

Pipeline verdicts

Pipeline	Verdict	One-line
1. Log Anomaly	PARTIAL	Ingestion/transport/storage HAVE; the anomaly detector itself is GAP (spec-only)
2. Event Grouping	GAP	Alerts land as individual receipts; no correlation/clustering into stories (clustering primitives PARTIAL)
3. Incident Similarity	PARTIAL	Generic IR engines HAVE; incident-typed contract slice landed (PR #33); dedup + live ASM GAP

---

Pipeline 1 — Log Anomaly (ingest logs → detect anomalies per log)

Component	Status	Evidence
Log stream ingestion	HAVE	infra/k8s/observability/base/promtail.yaml → Loki; services/telemetry-producer/app.py (OTLP→EventEnvelope→Kafka); services/mlog-projection-loki/
Kafka bus	HAVE	infra/fabric/helm/helmfile.yaml (bitnami/kafka KRaft); topics infra/k8s/eventbus/base/topic-bootstrap.yaml (telemetry.logs/metrics/traces)
Object store (MinIO)	HAVE	helmfile minio; infra/k8s/workspace-minio/base/
Persistence (Postgres)	HAVE	helmfile cnpg/postgres; infra/datastores/postgres/*.sql
Elastic / OpenSearch	PARTIAL	client refs only (infra/k8s/sherlock-search/base/deployment.yaml OPENSEARCH_URL); no server chart anywhere
Anomaly (Flink) Core	GAP	no Flink/Kafka-Streams/Faust/Spark; stream work is plain Kafka consumers (Kappa)
Log Anomaly Detector	GAP	spec-only: GDI open-ai4it-spec/docs/glossary/TERMS.md (LAD), docs/devops/metering-anomaly-feedback-pipeline.md; no drain3/LogBERT/DeepLog impl
telemetry-capture-liveness	HAVE	infra/k8s/observability/base/telemetry-liveness.yaml + telemetry-canary.yaml
Tempo (traces)	PARTIAL	alerted-on (TempoNotReceivingSpans) but not deployed

Pipeline 2 — Event Grouping (events/alerts + anomalies → incident "stories")

Component	Status	Evidence
Alert ingestion (PagerDuty/Alertmanager)	HAVE	infra/k8s/alert-delivery/base/sink.py (each alert → EvidenceReceipt); observability/base/telemetry-alert-routing.yaml (pagerdutyConfigs)
NOI-style event adapter	GAP	ingestion is Prometheus/Alertmanager-native; no Netcool/NOI adapter
event-envelope w/ story_id	HAVE	GDI open-ai4it-spec/contracts/schemas/event-envelope.schema.json (story_id/features/explainability/feedback); topics derived-alert-groups/derived-stories declared
Event Grouping Service	GAP	alert-delivery only emits 1 receipt/alert — no grouping/dedup/story assembly
Clustering primitives	PARTIAL	hellgraph hg_analytics connected-components (+planned Louvain), link-prediction.ts Jaccard, personalizedPageRank — unassembled
near-duplicate / dedup	GAP	no MinHash/SimHash anywhere in estate

Pipeline 3 — Incident Similarity (match incidents to SRE search terms)

Component	Status	Evidence
Generic IR engine	HAVE	apps/sherlock-engine/src/main.rs (Tantivy BM25 ⊕ dense/Qdrant via RRF); hellgraph/ts/src/{bm25,ann,retrieval,hybrid,link-prediction}.ts
Vector substrate	HAVE	apps/memoryd/.../qdrant_index.py, apps/embeddings/app.py
Similar Incidents Service (incident-typed)	PARTIAL — landed	PR #33: deterministic incident-similarity contract + scorer + teeth (open-ai4it-spec/modules/story_services/incident_similarity/)
Dedup/near-dup	GAP	as above
Topology Service / ASM	PARTIAL	graph substrate HAVE (hellgraph crates); live ASM/service-discovery GAP. (Note: hg_analytics/src/topology.rs is deployment-placement, NOT service ASM)
Localization Service (blast-radius)	HAVE	sociosphere/gbrg/crates/gbrg-core/src/scoring.rs (BlastRadiusProofArtifact, deterministic [0,1])
Receipt/provenance spine	HAVE	contracts/EvidenceReceipt.v0.1.json; apps/receipt-gateway/tools/*receipt*.py; hellgraph/crates/hg_proof

Cross-cutting components & actors

Component	Status	Evidence
ChatOps Integrator + Orchestrator	PARTIAL	apps/matrix-qes-operator/ (thin, no live homeserver); WordOps lease fabric #1171
Connection Controller / gateway	PARTIAL/HAVE	orchestrators HAVE (sp-orchestrator/crates/*, apps/deepdive-orchestrator); gateways apps/wordops-mcp-gateway, apps/arcticdb-gateway, apps/search-gateway; no component literally "Connection Controller"
Training Console + training-data ingest	PARTIAL	deploy/training/ray-lora/ray_lora_train.py (real LoRA on verified traces); apps/lattice-studio/.../annotation_training.py; no console UI
The Assay (ok/sad/bad)	HAVE	apps/reasoning-failure-runner/src/reasoning_failure_runner/domain.py (Verdict ok/sad/bad); sourceos-spec ReasoningAssay
Self-heal control loop	HAVE	sociosphere/automation/{executors,responder,scheduler}.py (loop closed, honest liveness — #518)
Resilience program	HAVE (runner) / PARTIAL (fabric spec)	apps/reasoning-failure-runner/; docs/CHAOS_RESILIENCE_FABRIC_V0.md
Actor: administrator	HAVE	WordOps A0–A4 lease governance (docs/WORDOPS_APPROVAL_TO_LEASE_GOVERNANCE.md)
Actor: chat user	PARTIAL	matrix-qes-operator command seam; no live bot

---

Build program (sovereign delta) — acceptance criteria & sequencing

Step 0 (DONE) — Pipeline 3 seed. PR #33. Deterministic incident-similarity contract + scorer + teeth. Reuses receipt spine, the Assay, HellGraph topology field. Acceptance: make validate/make test green; teeth fire on perturbed golden. ✅

Step 1 — Pipeline 2: Event Grouping Service (highest gap value).

• Build: deterministic grouper contract — group normalized alerts + log anomalies into story_id-keyed incident stories by (time-window ∩ shared entity/service key ∩ topology proximity). Sovereign equivalent of the Event Grouping Service.
• Reuse: GDI event-envelope (story_id), hellgraph connected-components/Louvain + Jaccard edges, EvidenceReceipt spine, the Assay for grouping-confidence.
• Acceptance: schema (story.schema.json) + deterministic grouper + valid/invalid fixtures + validator wired into make validate; grouping is reproducible + receipt-sealed; teeth both ways. Emits derived-alert-groups/derived-stories topics already declared in topics.yaml.
• Blocker: none — buildable now in GDI, same pattern as PR feat(story_services): incident-similarity contract + deterministic scorer (AI4IT Pipeline 3) #33.

Step 2 — Pipeline 1: Log Anomaly Detector (sovereign LAD).

• Build: our own drain3-style log-template miner + per-template anomaly scorer (rarity/rate deviation), MIT/stdlib. Feeds derived-log-ad-grouping.
• Reuse: Kafka telemetry.logs (HAVE), Loki as store (HAVE), EvidenceReceipt, the Assay verdict.
• Acceptance: template contract + deterministic miner + fixtures + validator; emits anomalies as event-envelope insight messages with explainability; teeth.
• Blocker: choose stream substrate — no Flink (Kappa Kafka-consumer pattern is the estate norm); do NOT introduce Flink (bloat) unless justified.

Step 3 — supporting primitives.

• near-duplicate/dedup (MinHash or SimHash) — absent estate-wide; needed by both grouping (alert dedup) and similarity. Deterministic, stdlib, fixture-tested.
• Topology/ASM ingestion layer over hellgraph (live service-dependency graph) so the topology signal in Steps 0–1 is fed by real topology, not a supplied field.

Step 4 — wiring / actors.

• Wire matrix-qes-operator to a live homeserver so the chat-user actor is real (ChatOps PARTIAL→HAVE).
• Deploy OpenSearch server OR formally retire the client refs (Elastic PARTIAL→decision).
• Deploy Tempo or remove its liveness alert (Tempo PARTIAL→decision).

Consume-not-fork constraint (applies to every step): build sovereign equivalents; MIT/Apache only; no IBM/Watson proprietary. Every new gate ships a negative fixture proving it can fail (never-fired = suspect).

/cc @mdheller