converters: reference converters emitting the MIF Container Profile (start: MemPalace) (#77) - #222
Open
perlowja wants to merge 5 commits into
Open
Conversation
…odeled-information-format#77) First of the reference converters the Container Profile section (modeled-information-format#205) references. Reads a MemPalace palace (ChromaDB) directly and emits a MIF Container Profile corpus: each drawer becomes a kind:memory record whose payload is a schema-valid MIF memory unit, typed conceptType:episodic per the ai-memory MemPalace mapping (modeled-information-format#206). Source-specific fields round-trip under extensions.mempalace; ids are UUIDv5-stable. Ships a schema-validated example (examples/converters/mempalace.corpus.json); both emitted units validate against schema/mif.schema.json. converters/README documents the suite; Mem0/Letta/Cognee/Graphiti follow the same output-layer port, each as its own PR with a validated example.
…/Graphiti/Obsidian) (modeled-information-format#77) Adds the remaining reference converters alongside MemPalace, all emitting the MIF Container Profile natively (not the retired MPF envelope) via a shared _mif_common helper: mem0 — Qdrant (+SQLite history, +Platform): fact/obs -> semantic, history -> episodic letta — sqlite/server: core+archival -> semantic, messages -> episodic cognee — graph+vector+catalog: entities/chunks -> semantic; source docs -> kind:document graphiti — Neo4j temporal KG: entity -> semantic, episode -> episodic, edges -> relationships[] (bitemporal validity kept in extensions, deferred to a Level-3 sidecar) obsidian — a vault of Markdown notes: note -> semantic, journal -> episodic, [[wikilinks]] -> relationships[] (converter only; MIF core stays vendor-neutral per ADR-017) Each keeps its source-reading faithful to the CHARON adapter (stdlib + the one source client; no MIF runtime) and re-targets only the output. Every converter ships a schema-validated example under examples/converters/; all emitted memory units validate against schema/mif.schema.json.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| headers["X-Organization"] = self.org | ||
| req = urllib.request.Request(url, headers=headers, method="GET") | ||
| try: | ||
| with urllib.request.urlopen(req, timeout=60) as resp: |
…llib) The Letta server-mode GET builds its URL from the operator-supplied --base; guard the scheme to http(s) so a misconfigured base cannot become a file:// arbitrary-file read, and annotate the (now-mitigated) urllib call for the Semgrep dynamic-urllib audit rule.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new converters/ suite of reference adapters that export various AI-memory backends into MIF’s Container Profile corpus format (*.corpus.json), alongside example corpora and a top-level converters README to document intended usage and mappings.
Changes:
- Introduces multiple Python converters (MemPalace, Mem0, Letta, Cognee, Graphiti, Obsidian) plus a shared
_mif_common.pyhelper for emitting corpus envelopes and MIF memory payloads. - Adds schema-validated example converter outputs under
examples/converters/. - Documents the converter suite and its mapping/usage expectations in
converters/README.md.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
converters/_mif_common.py |
Shared emission helpers for corpus envelope + memory payload shaping (currently not aligned with repo Container Profile schema). |
converters/README.md |
Suite-level documentation and usage examples (includes a broken spec anchor link and an over-broad “all converters use _mif_common” claim). |
converters/mempalace/mempalace_to_mif.py |
MemPalace (ChromaDB) reader and exporter (currently hand-builds a non-conformant envelope). |
converters/mem0/mem0_to_mif.py |
Mem0 (Qdrant + optional SQLite history, or hosted API) exporter using _mif_common. |
converters/letta/letta_to_mif.py |
Letta (SQLite or REST) exporter using _mif_common. |
converters/cognee/cognee_to_mif.py |
Cognee exporter emitting both document and memory records using _mif_common. |
converters/graphiti/graphiti_to_mif.py |
Graphiti (Neo4j) exporter with relationship mapping, using _mif_common. |
converters/obsidian/obsidian_to_mif.py |
Obsidian vault exporter (currently derives wikilink targets inconsistently with note IDs). |
examples/converters/mempalace.corpus.json |
MemPalace example corpus (top-level envelope fields don’t match repo Container Profile schema). |
examples/converters/mem0.corpus.json |
Mem0 example corpus (top-level envelope fields don’t match repo Container Profile schema). |
examples/converters/letta.corpus.json |
Letta example corpus (top-level envelope fields don’t match repo Container Profile schema). |
examples/converters/cognee.corpus.json |
Cognee example corpus (top-level envelope fields don’t match repo Container Profile schema). |
examples/converters/graphiti.corpus.json |
Graphiti example corpus (top-level envelope fields don’t match repo Container Profile schema). |
examples/converters/obsidian.corpus.json |
Obsidian example corpus (top-level envelope fields don’t match repo Container Profile schema; wikilink targets don’t resolve within the corpus). |
Comment on lines
+91
to
+113
| def build_corpus( | ||
| records: List[Dict[str, Any]], *, source_system: str, source_instance: Optional[str] = None, | ||
| generated_at: Optional[str] = None, | ||
| ) -> Dict[str, Any]: | ||
| """Wrap `records` (each `{"kind": "memory"|"document", "payload": ...}`) in a | ||
| MIF Container Profile corpus.""" | ||
| corpus: Dict[str, Any] = { | ||
| "@context": MIF_CONTEXT, | ||
| "@type": "MemoryCorpus", | ||
| "mif_version": MIF_VERSION, | ||
| "records": records, | ||
| "provenance": { | ||
| "@type": "prov:Entity", | ||
| "prov:wasGeneratedBy": { | ||
| "@type": "prov:Activity", | ||
| "prov:used": source_system, | ||
| "prov:generatedAtTime": generated_at or datetime.now(timezone.utc).isoformat(), | ||
| }, | ||
| }, | ||
| } | ||
| if source_instance: | ||
| corpus["provenance"]["prov:wasDerivedFrom"] = source_instance | ||
| return corpus |
Comment on lines
+5
to
+8
| Reference converters that read a source AI-memory system and emit a MIF | ||
| **[Container Profile](../SPECIFICATION.md#container-profile-corpus-envelope--proposed)** | ||
| corpus (`*.corpus.json`) — many MIF memory units (and, where applicable, their | ||
| source `document` records) as one transportable artifact. |
Comment on lines
+24
to
+28
| All MIF emission goes through the shared [`_mif_common.py`](_mif_common.py) | ||
| helper, so every converter produces consistent, schema-valid output. Each | ||
| converter ships a **schema-validated example** under | ||
| [`../examples/converters/`](../examples/converters/); every emitted memory unit | ||
| is validated against [`../schema/mif.schema.json`](../schema/mif.schema.json). |
Comment on lines
+1
to
+5
| { | ||
| "@context": "https://mif-spec.dev/schema/context.jsonld", | ||
| "@type": "MemoryCorpus", | ||
| "mif_version": "1.2.2", | ||
| "records": [ |
Comment on lines
+22
to
+31
| "relationships": [ | ||
| { | ||
| "type": "links-to", | ||
| "target": "urn:mif:d5845a39-f8d4-5d6d-a3d7-5f96889acd99", | ||
| "metadata": { | ||
| "obsidian": { | ||
| "wikilink": "Container Profile" | ||
| } | ||
| } | ||
| } |
Comment on lines
+1
to
+5
| { | ||
| "@context": "https://mif-spec.dev/schema/context.jsonld", | ||
| "@type": "MemoryCorpus", | ||
| "mif_version": "1.2.2", | ||
| "records": [ |
Comment on lines
+1
to
+5
| { | ||
| "@context": "https://mif-spec.dev/schema/context.jsonld", | ||
| "@type": "MemoryCorpus", | ||
| "mif_version": "1.2.2", | ||
| "records": [ |
Comment on lines
+1
to
+5
| { | ||
| "@context": "https://mif-spec.dev/schema/context.jsonld", | ||
| "@type": "MemoryCorpus", | ||
| "mif_version": "1.2.2", | ||
| "records": [ |
Comment on lines
+202
to
+224
| def build_container( | ||
| palace_path: Path, *, source_instance: Optional[str] = None, | ||
| restore_original_ids: bool = True, | ||
| ) -> Dict[str, Any]: | ||
| """Assemble a MIF Container Profile corpus from a palace snapshot.""" | ||
| records = list(iter_memory_records(palace_path, restore_original_ids=restore_original_ids)) | ||
| corpus: Dict[str, Any] = { | ||
| "@context": MIF_CONTEXT, | ||
| "@type": "MemoryCorpus", | ||
| "mif_version": MIF_VERSION, | ||
| "records": records, | ||
| "provenance": { | ||
| "@type": "prov:Entity", | ||
| "prov:wasGeneratedBy": { | ||
| "@type": "prov:Activity", | ||
| "prov:used": SOURCE_SYSTEM, | ||
| "prov:generatedAtTime": datetime.now(timezone.utc).isoformat(), | ||
| }, | ||
| }, | ||
| } | ||
| if source_instance: | ||
| corpus["provenance"]["prov:wasDerivedFrom"] = source_instance | ||
| return corpus |
Comment on lines
+133
to
+149
| # [[wikilinks]] -> relationships (target = urn of the linked note) | ||
| rels: List[Dict[str, Any]] = [] | ||
| seen = set() | ||
| for target_name in _WIKILINK_RE.findall(body): | ||
| tn = target_name.strip() | ||
| if not tn or tn in seen: | ||
| continue | ||
| seen.add(tn) | ||
| rels.append({ | ||
| "type": "links-to", | ||
| "target": "urn:mif:" + str(uuid.uuid5(OBSIDIAN_NS, tn)), | ||
| "metadata": {"obsidian": {"wikilink": tn}}, | ||
| }) | ||
|
|
||
| mem = mif_memory( | ||
| mif_id=mif_uuid(OBSIDIAN_NS, rel.as_posix(), restore=fm.get("id") or fm.get("uid")), | ||
| concept_type=concept, |
zircote
requested changes
Jul 15, 2026
zircote
left a comment
zircote
left a comment
Collaborator
There was a problem hiding this comment.
Same envelope issue applies to all six corpus files here (cognee, graphiti, letta, mem0, mempalace, obsidian): mif_version needs to become containerProfileVersion, and @context needs to be https://mif-spec.dev/schema/container-context.jsonld rather than the bare memory-unit context.
Two file-specific issues on top of that:
- cognee.corpus.json records[0]: the document payload includes @id, created, extensions and namespace. None of those are allowed on DocumentReference. id (without the @) is already present separately, so @id can simply be removed.
- obsidian.corpus.json records[0]: created: "2026-06-25" is not a valid date-time. It needs a full timestamp, for example 2026-06-25T00:00:00Z.
One unrelated note: issue #274 (a gap where memoryType was not accepted as a conceptType alternative, despite documentation stating it should be) has merged, but it does not affect this PR either way. None of these files hit that path; your memory-kind records already validate correctly as written.
…converters Per @zircote's review: - _mif_common.py build_corpus(): emit containerProfileVersion (not mif_version) and the container-context @context, per schema/container.schema.json. Fix provenance to use the plain keys the schema defines (wasGeneratedBy/used/generatedAtTime/ wasDerivedFrom), not the prov:-prefixed variants. - mempalace_to_mif.py: route build_container() through the shared _mif_common.build_corpus() instead of reimplementing it locally, so the README's "every converter" claim is actually true. - All 6 checked-in examples/converters/*.corpus.json: same envelope + provenance-key fixes. - cognee_to_mif.py + its example: DocumentReference is additionalProperties:false with no extensions mechanism, so @id, namespace, created, and extensions are all schema-violations there (confirmed empirically via scripts/validate_container.py, not just the @id the review named) -- dropped from the document payload. - obsidian.corpus.json: fix the memory's created date to a valid date-time; fix a links-to relationship pointing at a non-existent record id, and drop a second relationship whose target record isn't present in this minimal example. - converters/README.md: fix the Container Profile link, which pointed at a SPECIFICATION.md anchor that doesn't exist (the section lives at docs/CONTAINER-PROFILE.md). Validated: python3 scripts/validate_container.py examples/ -> PASS (7 files).
Author
|
Fixed in 48a18ad: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Starts the reference-converter suite the Container Profile section (#205) references (#77).
This PR: a MemPalace → MIF Container Profile converter (
converters/mempalace/mempalace_to_mif.py) + a schema-validated example (examples/converters/mempalace.corpus.json) +converters/README.mddocumenting the suite.chromadbneeded, no MemPalace runtime).kind: memoryrecord whose payload is a valid MIF memory unit, typedconceptType: episodicper the ai-memory MemPalace mapping (docs(ai-memory): MemPalace → MIF Level-1 mapping + examples #206).extensions.mempalace;@ids are UUIDv5-stable (recovering an original UUID from YAML front matter when present).schema/mif.schema.json.Follow-ups (same output-layer port, each its own PR + validated example): Mem0, Letta, Cognee, Graphiti — currently MPF-era adapters in the CHARON tooling, being re-targeted to emit this envelope. Pairs with #205 (Container Profile spec) and #206 (ai-memory mapping).