Skip to content

fix: harden vector store reliability#570

Merged
zxuexingzhijie merged 12 commits into
spring-ai-alibaba:mainfrom
zxuexingzhijie:agent/fix-vector-store-reliability
Jul 26, 2026
Merged

fix: harden vector store reliability#570
zxuexingzhijie merged 12 commits into
spring-ai-alibaba:mainfrom
zxuexingzhijie:agent/fix-vector-store-reliability

Conversation

@zxuexingzhijie

Copy link
Copy Markdown
Collaborator

What changed

  • Replace synthetic similarity queries used for metadata-only reads with provider-specific exact metadata retrieval for SimpleVectorStore, Milvus, and Elasticsearch.
  • Check schema completeness by datasource and the complete requested table set instead of treating one recalled document as successful initialization.
  • Make vector replacement write the new documents before deleting the old set, assign fresh IDs when required, and roll back newly written documents on failure.
  • Validate embedding dimensions/model compatibility before activation and before vector operations can corrupt an existing collection.
  • Recover pending and stale embedding work at startup, with mapper-level integration coverage.
  • Persist SimpleVectorStore snapshots through a temporary file and atomic move.
  • Add bounded timeouts to hybrid retrieval and expose the timeout through vector-store configuration.
  • Reuse the upstream MetadataAwareSimpleVectorStore introduced by refactor: harden DataAgent backend and reuse Spring AI #562 rather than adding a parallel implementation.

Why

The previous metadata path was implemented as similaritySearch("default"). That made existence checks and deletion depend on the active embedding model and similarity ranking, so exact metadata matches could be missed. Replacement also deleted old vectors before the new set was safely established. A model/dimension change, interrupted persistence, stuck embedding status, or an unbounded hybrid request could therefore cause missing recall, data loss, or permanently blocked work.

Validation

  • ./mvnw -pl data-agent-management -DskipTests test-compile
  • ./mvnw -pl data-agent-management -Dtest='SchemaVectorRecallIntegrationTest,VectorReplacementIntegrationTest,SimpleVectorStorePersistenceIntegrationTest,EmbeddingModelCompatibilityValidatorTest,VectorStoreConfigurationTest,EmbeddingRecoveryMapperIntegrationTest,MetadataAwareSimpleVectorStoreTest' test — 14 passed
  • ./mvnw -q -pl data-agent-management test — 1819 passed, 0 failures, 0 errors, 1 skipped
  • The new vector regression tests use a deterministic real EmbeddingModel and real SimpleVectorStore; they do not use Mockito.
  • MilvusVectorStoreIntegrationTest exercises add/search/exact-filter/delete against an actual Milvus server and is gated by -Ddataagent.milvus.integration=true. It was not rerun during the final rebase because the local Docker daemon was unavailable.

Notes

  • This branch is rebased directly onto the current upstream main (ff49a5f).
  • Local datasource credentials in application.yml are intentionally excluded from the commit.

@zxuexingzhijie
zxuexingzhijie marked this pull request as ready for review July 25, 2026 14:58
@zxuexingzhijie
zxuexingzhijie merged commit cba7bd8 into spring-ai-alibaba:main Jul 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant