Skip to content

refactor(vdb)!: carry the vector as a top-level record embedding, not metadata["embedding"] - #2807

Draft
jioffe502 wants to merge 1 commit into
mainfrom
devin/1792789112-vdb-record-top-level-embedding
Draft

jioffe502 wants to merge 1 commit into
mainfrom
devin/1792789112-vdb-record-top-level-embedding

Conversation

@jioffe502

Copy link
Copy Markdown
Collaborator

Summary

Follow-up to #2794. Graph rows already carry one typed embedding column; this removes the last nested copy point: client VDB records now hold the vector at the top level.

# before                                   # after
{"document_type": "text",                  {"document_type": "text",
 "metadata": {"embedding": [...],           "embedding": [...],
              "content": "...", ...}}       "metadata": {"content": "...", ...}}

Scope (breaking):

  • to_client_vdb_records writes record["embedding"]; readers lancedb._create_lancedb_result, lancedb_collections._collection_rows, and the missing-embedding check in to_client_vdb_records read only that. No fallback to metadata["embedding"]; code that builds records by hand for LanceDB.run must move the vector.
  • MetadataSchema.embedding is removed.
  • Docs (vdbs.md, content-metadata.md) and examples/building_vdb_operator.ipynb updated; test fixtures moved to the new shape.

Unchanged: values stored in LanceDB (vector column, float32), graph-row columns, retrieval.

Tests: VDB / LanceDB / service / ingest / records / embedding-input suites: 1680 passed, 8 skipped (CPU-only). black 25.9.0 and flake8 7.3.0 clean on touched files.

Link to Devin session: https://nvidia-cloud.devinenterprise.com/sessions/34802976322840ce95c617ad48813bcd
Open in Devin Desktop: https://nvidia-cloud.devinenterprise.com/desktop/session/34802976322840ce95c617ad48813bcd?variant=devin
Requested by: @jioffe502

… metadata["embedding"]

to_client_vdb_records now emits {document_type, embedding, metadata}; LanceDB,
collection writes, and the canonical-record check read record["embedding"].
MetadataSchema drops its unused embedding field; docs and the custom VDB
notebook follow.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant