Epic: Prophet Understand / Repo Intelligence v0

[mdheller]

Purpose

Create the SocioProphet repo-understanding layer: a governed, searchable, agent-usable codebase and knowledge-base graph artifact for every important repository.

This is the strategic absorption path from the current open-source "Understand Anything" style product pattern. We should borrow the product lesson — interactive graph, guided architecture tours, semantic search, diff impact analysis, and multi-agent packaging — but implement it as a governed SocioProphet substrate rather than vendoring an ungoverned local JSON/dashboard tool.

Product thesis

Every repo should carry a machine-readable self-model that humans and agents can trust:

• structural graph of files, modules, services, contracts, endpoints, schemas, docs, domains, and flows
• human-readable summaries for each important node
• guided tours ordered by dependency and risk
• semantic/fuzzy/graph search over repo contents and concepts
• PR diff impact analysis with affected nodes, edges, tests, docs, and policies
• provenance receipts tying graph facts to commit, parser, agent, source hash, and confidence
• policy gates that detect stale, unsigned, invalid, or drifted repo-intelligence artifacts

Core artifact

Define a versioned prophet-understanding.json / sociograph.json artifact family with at least:

• schema_version
• repo, commit, default_branch, generated_at
• generator and parser versions
• agent_identity / future DID binding placeholder
• source_hashes
• nodes[]
• edges[]
• summaries[]
• tours[]
• diff_impact[]
• policy_status
• provenance_receipts[]
• validation_results[]

Cross-repo ownership

• smart-tree: structural scanner and local graph emitter
• lampstand: local index/search over repo graph artifacts
• sherlock-search: hybrid lexical/vector/graph query over repo intelligence
• ontogenesis: ontology, JSON-LD, SHACL, Avro/TriTRPC contract generation
• agent-registry: multi-agent skill manifests from one registry contract
• agentplane: graph-aware dispatch, bounded repair, and impact-aware agent work orders
• policy-fabric: validation gates for graph freshness, provenance, signatures, and PR impact risk
• delivery-excellence: metrics for freshness, coverage, graph drift, and agent repair quality
• socioprophet: /repo-map or /understand UI workbench

v0 acceptance criteria

• A first formal artifact contract exists and validates in CI.
• At least one fixture repo graph is committed as an example.
• A scanner can emit a deterministic graph from one repo.
• Search can answer at least: "what owns this file?", "what depends on this contract?", "what changes in this PR matter?"
• UI workbench has a minimal graph + node detail + evidence panel path.
• Policy gates detect stale/missing/invalid artifacts without blocking unrelated work by default.
• Metrics expose graph freshness, coverage, drift, and PR impact radius.

Non-goals for v0

• No autonomous code mutation from graph output.
• No blind vendoring of Lum1104/Understand-Anything.
• No unreviewed post-commit hooks in sensitive repos.
• No claims of semantic certainty without evidence/confidence.

Implementation stance

Build our own governed substrate. Use external projects as benchmarks and UX references, not as the trust root.

[mdheller]

Tranche 1 dispatch map

Created the first cross-repo work orders for Prophet Understand / Repo Intelligence v0:

• Index Prophet Understand repo graph artifacts lampstand#8 — local indexing over repo graph artifacts
• Add hybrid repo-intelligence search lane for Prophet Understand v0 sherlock-search#37 — hybrid repo-intelligence search lane
• Define Prophet Understand ontology and validation contracts ontogenesis#41 — ontology and validation contracts
• Generate multi-agent Prophet Understand skill manifests agent-registry#20 — multi-agent skill manifests
• Add graph-aware dispatch and PR impact work orders agentplane#106 — graph-aware dispatch and PR impact work orders
• Add Prophet Understand graph artifact policy gates policy-fabric#59 — graph artifact policy gates
• Add Repo Intelligence metrics for Prophet Understand v0 delivery-excellence#15 — repo-intelligence metrics and scorecards
• Add Prophet Understand repo-map workbench to SocioProphet UI socioprophet#310 — /repo-map / /understand UI workbench

Smart Tree note: SocioProphet/smart-tree currently has GitHub Issues disabled, so the scanner/emitter work could not be created there as an issue through the connector. Track its required lane here until Issues are enabled or until a direct PR is opened:

• Smart Tree should own the deterministic structural scanner and local graph emitter.
• Required command shape: smart-tree understand --repo . --out .prophet/prophet-understanding.json.
• Required outputs: stable IDs, source anchors, parser receipts, skip receipts, provenance placeholders, validation results, and deterministic sorted output.
• No unreviewed post-commit hook should be enabled by default.

Dispatch posture: all lanes are bounded v0 artifacts/docs/fixtures/contracts work. No autonomous mutation, no third-party vendoring, and no claims of semantic certainty without evidence/confidence.

[mdheller]

Tranche 2 platform contract landed

Implemented the first concrete Prophet Understand / Repo Intelligence v0 contract artifacts in prophet-platform:

• docs/PROPHET_UNDERSTAND_REPO_INTELLIGENCE.md — platform spec, previously committed as 2a34df47474048d22d89159427d4796c26c97b8c
• schemas/repo-intelligence/prophet-understanding.schema.json — v0 JSON Schema, committed as bcc8895ed4a7d38a82246f33991fe3b54986c0af
• examples/repo-intelligence/prophet-understanding.fixture.json — v0 fixture graph artifact, committed as 567c16efc50c5cec8344900e0a147b5bda6b48c0
• tools/validate_prophet_understand.py — schema/fixture/provenance/reference/policy/doc validator, committed as 6afebc6edc0cf494551eab7507b3566836520d9c
• tools/validate_repo.py — main repo validation now calls the Prophet Understand validator, committed as 69fd13283433a31e9a0ecb25a9da8e9c55370bef

Current validation lane:

make validate-repo now includes Prophet Understand validation through tools/validate_repo.py.

Remaining immediate work:

1. Verify the new validator in CI/local checkout.
2. Add a dedicated Makefile target only if we want a separate validate-prophet-understand command in addition to validate-repo.
3. Move from fixture artifact to Smart Tree emitter output.
4. Consume the fixture in Lampstand and Sherlock.

Caveat: Smart Tree issues remain disabled, so scanner/emitter implementation is still tracked here until a direct PR or issue-enabled repo path exists.

[mdheller]

Tranche 3 implementation update

Moved Prophet Understand / Repo Intelligence from platform contract into the first producer/index/search/policy/metrics/UI lanes.

Platform contract hardening

• Updated schemas/repo-intelligence/prophet-understanding.schema.json to include freeform metadata support for repo metadata, nodes, and edges: 8104074f750920bdf2c2795ccdfd952c07f3293e
• Updated tools/validate_prophet_understand.py so directory graph nodes do not require source anchors while factual file-like nodes still do: 61ce187dba3906fa6f4f8261dcbb0c481dc02ff4

Smart Tree producer lane

• Added SocioProphet/smart-tree/docs/PROPHET_UNDERSTAND_EMITTER.md: 988f0b8f6d9603a28cf29f67042a2fed1dec2300
• Added SocioProphet/smart-tree/tools/emit_prophet_understanding.py: 09f3164f697d7044522bd46f9f8e3812208d7a19

This creates the first direct scanner/emitter path toward .prophet/prophet-understanding.json.

Lampstand index lane

• Added SocioProphet/lampstand/docs/PROPHET_UNDERSTAND_INDEXING.md: c8cf06a9f24f02a297e8261e8a2c59ae4b9bf23d
• Added SocioProphet/lampstand/tools/index_prophet_understanding.py: 0f2b4be9cb653ab6afde4b10b98580239f67109d

Sherlock query lane

• Added SocioProphet/sherlock-search/docs/PROPHET_UNDERSTAND_SEARCH.md: c4bf7f2f0283adc399c3c885320347fea28a47c3
• Added SocioProphet/sherlock-search/tools/search_prophet_understanding.py: 79304a8704ecd718c98487ea1a56fabe9e724745

Ontology / agent / policy / metrics / UI lanes

• Added SocioProphet/ontogenesis/docs/PROPHET_UNDERSTAND_ONTOLOGY.md: 273d1787d215be9080f2ad0206586f99321bc3c3
• Added SocioProphet/agent-registry/docs/PROPHET_UNDERSTAND_AGENT_SKILLS.md: 767d5bb48b0ab05b7a0a32d96699574d910e17e9
• Added SocioProphet/policy-fabric/docs/PROPHET_UNDERSTAND_POLICY.md: 7f2f94b3d8329503005766579f2e762c8d229195
• Added SocioProphet/policy-fabric/tools/evaluate_prophet_understand_policy.py: 20066ee5a55932189b27e5cf3da68b398a52a1ef
• Added SocioProphet/delivery-excellence/docs/PROPHET_UNDERSTAND_METRICS.md: a395191ba69540a0d34cc556ceaef15e23a26293
• Added SocioProphet/socioprophet/docs/PROPHET_UNDERSTAND_WORKBENCH.md: 1d33d3e34f47b848c537f7bd8623d192fe5586e0

AgentPlane protected branch handling

Direct main write was blocked by branch protection, so I opened PR #113:

• SocioProphet/agentplane#113 — Document Prophet Understand graph-aware dispatch
• branch: prophet-understand-dispatch-v0
• head: f10d5b5eb94fa6eb411e95d83241a83bdf9252d1
• observed checks: ci, validate, and lint all completed successfully
• merge attempt still failed with GitHub saying required status check lint is expected, so this remains open for branch-protection resolution or a subsequent merge attempt after GitHub check propagation.

Current caveat

The emitter/index/search/policy tools are now in repo as first-pass scripts, but I have not run them in a live checkout inside this tool session. The next tranche should run the end-to-end vertical slice:

smart-tree emit -> prophet-platform validate -> lampstand index -> sherlock search -> policy-fabric evaluate -> delivery-excellence scorecard.

[mdheller]

Tranche 4 vertical slice gate added

Added the first platform-local Prophet Understand vertical-slice harness and wired it into the repo validation gate.

New platform vertical slice

• tools/run_prophet_understand_vertical_slice.py: 4ccb3cecf81b98c87d85129a8d7fbcfa4ea065fb

The harness runs the contract validator and produces deterministic local outputs from the fixture:

• build/prophet-understand/lampstand-index.json
• build/prophet-understand/sherlock-search-contract-query.json
• build/prophet-understand/policy-decision.json
• build/prophet-understand/delivery-scorecard.json
• build/prophet-understand/vertical-slice-summary.json

This gives us a single repo-local proof path for:

contract fixture -> validation -> Lampstand-style index -> Sherlock-style search -> Policy Fabric-style decision -> Delivery Excellence-style scorecard

Validation gate wiring

• Updated tools/validate_repo.py to call tools/run_prophet_understand_vertical_slice.py: 1015ab5423bb4e36dfe19cd138811fd96758921c

So make validate-repo now gates both the Prophet Understand contract and the fixture-backed vertical slice.

Delivery Excellence scoring helper

• Added SocioProphet/delivery-excellence/tools/score_prophet_understand.py: 7d671515885231fafbbc2fd3381f3cfd5f318fda

This gives Delivery Excellence an executable scorecard helper for real graph artifacts.

AgentPlane PR status

Retried merge for SocioProphet/agentplane#113 after GitHub reported the PR as mergeable. The merge still failed with branch protection reporting required status check lint is expected, despite observed lint success on the head commit. Leave PR open for branch-protection/check propagation resolution.

Remaining gap

The next tranche should run these scripts in a checked-out repo and then replace fixture-local mimics with cross-repo calls:

• Smart Tree emits the artifact from a real repo.
• Prophet Platform validates the emitted artifact.
• Lampstand indexes the emitted artifact.
• Sherlock searches the Lampstand index.
• Policy Fabric evaluates the same emitted artifact.
• Delivery Excellence scores it.

At that point the vertical slice moves from fixture-backed to repo-backed.

[mdheller]

Tranche 5 repo-backed vertical slice runner added

Added the first cross-repo runner for a real repo-backed Prophet Understand / Repo Intelligence vertical slice.

New runner

• tools/run_prophet_understand_repo_backed_slice.py: 6bb5d3e1b218a02bb73d259eb3883ee3c88d7bcc

This runner expects sibling repos under ~/dev and executes:

1. smart-tree/tools/emit_prophet_understanding.py
2. prophet-platform/tools/validate_prophet_understand.py --artifact <emitted-artifact> --skip-doc
3. lampstand/tools/index_prophet_understanding.py
4. sherlock-search/tools/search_prophet_understanding.py
5. policy-fabric/tools/evaluate_prophet_understand_policy.py
6. delivery-excellence/tools/score_prophet_understand.py

Default target:

• target repo: ~/dev/smart-tree
• emitted artifact: ~/dev/smart-tree/.prophet/prophet-understanding.json
• outputs: prophet-platform/build/prophet-understand/repo-backed/

Validator hardening

• tools/validate_prophet_understand.py now accepts arbitrary artifacts through --artifact and can skip platform-doc marker checks through --skip-doc: 939a9fb7a37b172b893e7032036250449a1ce588

This is required so Prophet Platform can validate artifacts emitted by Smart Tree or other producers, not only the platform fixture.

Documentation

• Added docs/PROPHET_UNDERSTAND_REPO_BACKED_SLICE.md: 946e75b5e3f52f7f841d32fc0a59fafa9507df2a

Current status

We now have:

• fixture-backed gate wired into validate_repo.py
• repo-backed runner ready for local execution across sibling repos
• producer/index/search/policy/score helper scripts in their respective repos

Next step is terminal execution of the repo-backed runner, then fixing whatever real-world emitter/index/search/policy issues surface.

[mdheller]

Tranche 6 hardening update

Hardened the repo-backed vertical-slice path before terminal execution.

Smart Tree emitter hardening

• SocioProphet/smart-tree/tools/emit_prophet_understanding.py: acbd693f658f8036db2ea3e65b2ffd8ae2156e52

Changes:

• Records skipped directories before filtering traversal.
• Moves repo.metadata.skipped_paths into the artifact before final artifact hash calculation.
• Computes repo.artifact_hash from the artifact with artifact_hash normalized to sha256:pending.
• Normalizes missing remotes to local/<repo> instead of an invalid single-name repo identifier.
• Keeps skipped-path policy evidence tied to receipt:skipped-paths when skips exist.

Smart Tree emitter smoke

• SocioProphet/smart-tree/tools/smoke_prophet_understanding_emitter.py: 85146ec54ffb5fb0fe94d8ae36427d77cce85120

Creates a temporary fixture repo, emits .prophet/prophet-understanding.json, and checks schema version, finalized artifact hash, expected node kinds, skipped-path receipt, skipped-path metadata, and edge endpoint integrity.

Lampstand index smoke

• SocioProphet/lampstand/tools/smoke_prophet_understanding_index.py: 443e72cc016c1155437f1b2ce1879b6647feafd9

Creates a small v0 artifact fixture and verifies the indexer emits node, edge, summary, policy, and receipt record families.

Sherlock search smoke

• SocioProphet/sherlock-search/tools/smoke_prophet_understanding_search.py: 52474ca6c0f39ea43db234636dae5977ecf9222d

Builds a tiny Lampstand-style index and verifies Sherlock returns graph/evidence search results while preserving provenance IDs.

Policy Fabric smoke

• SocioProphet/policy-fabric/tools/smoke_prophet_understand_policy.py: 8b4a6a50f915685634855e754063b5b7e7aca490

Builds a graph artifact containing a workflow path and verifies Policy Fabric returns require_review for high-risk graph scope.

Delivery Excellence score hardening and smoke

• SocioProphet/delivery-excellence/tools/score_prophet_understand.py: 614060be7bcbcae717dc0d2bfa5a3d8126585cda
• SocioProphet/delivery-excellence/tools/smoke_prophet_understand_score.py: 1ea32c16aa2e5fdbb212fe659f607619ba1988f2

Changes:

• Scorecard state now accounts for ownership gaps, not only schema/policy/coverage ratios.
• Smoke verifies schema validity, expected impact radius, and yellow status when ownership metadata is missing.

Net effect

The repo-backed runner now has smoke-test support at each participating repo boundary before the full end-to-end terminal run:

• producer smoke: Smart Tree
• index smoke: Lampstand
• query smoke: Sherlock
• policy smoke: Policy Fabric
• score smoke: Delivery Excellence

Next terminal tranche should run the five repo-local smokes first, then the cross-repo runner.