From 61e5d4134977632da2c5bf4a4e8a22a96f35548f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Istv=C3=A1n=20Zolt=C3=A1n=20Szab=C3=B3?= Date: Mon, 19 Oct 2026 16:44:36 +0200 Subject: [PATCH 1/5] Update skills for 1.20 docs. --- skills/qdrant-edge/SKILL.md | 2 +- skills/qdrant-model-migration/SKILL.md | 6 ++- skills/qdrant-monitoring/setup/SKILL.md | 4 +- skills/qdrant-multitenancy/SKILL.md | 1 + .../memory-usage-optimization/SKILL.md | 8 ++-- .../search-speed-optimization/SKILL.md | 3 +- .../vertical-scaling/SKILL.md | 5 ++- .../qdrant-search-quality/diagnosis/SKILL.md | 4 +- .../search-strategies/hybrid-search/SKILL.md | 4 +- .../hybrid-search/combining-searches/SKILL.md | 2 + .../hybrid-search/search-types/SKILL.md | 38 +++++++++++++------ skills/qdrant-sizing/SKILL.md | 6 ++- 12 files changed, 56 insertions(+), 27 deletions(-) diff --git a/skills/qdrant-edge/SKILL.md b/skills/qdrant-edge/SKILL.md index eaf6850..ea4236f 100644 --- a/skills/qdrant-edge/SKILL.md +++ b/skills/qdrant-edge/SKILL.md @@ -26,7 +26,7 @@ There is no built-in `.sync()`. Sync is a pattern you assemble from shard helper Use when: you need exact-term or BM25 matching, alone or alongside vectors. -- BM25 is built into Edge (`Bm25`, `Bm25Config`, `embed_document`, `embed_query`) with the IDF `Modifier` on `EdgeSparseVectorParams`, and is wire-compatible with server BM25: a shard seeded from a server snapshot answers local BM25 queries without re-indexing. Do not ship a second BM25 library [Edge BM25](https://skills.qdrant.tech/md/documentation/edge/edge-bm25/) +- BM25 is built into Edge (`Bm25`, `Bm25Config`, `embed_document`, `embed_query`) with the IDF `Modifier` on `EdgeSparseVectorParams`, and is wire-compatible with the server's sparse vector BM25 model (Edge BM25 docs cover only the sparse vector variant): a shard seeded from a server snapshot answers local BM25 queries without re-indexing. Do not ship a second BM25 library [Edge BM25](https://skills.qdrant.tech/md/documentation/edge/edge-bm25/) - Dense embeddings are NOT in Edge: generate them on device with the separate `fastembed` package [FastEmbed embeddings](https://skills.qdrant.tech/md/documentation/edge/edge-fastembed-embeddings/) - For hybrid search, run the dense and sparse legs as `prefetches` and fuse them with a `Fusion` query in a single `query` call, instead of combining rankings in application code [Reading data](https://skills.qdrant.tech/md/documentation/edge/edge-api/reading-data/) diff --git a/skills/qdrant-model-migration/SKILL.md b/skills/qdrant-model-migration/SKILL.md index 7b6a461..9cb4148 100644 --- a/skills/qdrant-model-migration/SKILL.md +++ b/skills/qdrant-model-migration/SKILL.md @@ -18,6 +18,7 @@ You MUST re-embed if: changing model provider (OpenAI to Cohere), changing archi You do NOT need to re-embed existing dense vectors if: +- Adding BM25 for hybrid search on v1.20+: no vectors to generate. Create a `text` payload index with `scoring: true` on the payload field; Qdrant builds it from the stored payload [BM25 scoring](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=bm25-scoring) - Adding sparse vectors for hybrid search: generate only the sparse vectors. On v1.18+, add the sparse field to the existing collection and backfill it with `UpdateVectors`. On v1.17 or earlier, copy the dense vectors into the new collection instead of recomputing them [Update vectors](https://skills.qdrant.tech/md/documentation/manage-data/points/?s=update-vectors) - Using Matryoshka models: use the `dimensions` parameter to output lower-dimensional embeddings (some recall loss, good for 100M+ datasets) - Changing quantization (binary to scalar): Qdrant re-quantizes automatically [Quantization](https://skills.qdrant.tech/md/documentation/manage-data/quantization/) @@ -73,6 +74,9 @@ If you anticipate future model migrations, define both vector fields upfront at Use when: adding sparse/BM25 vectors to an existing dense-only collection. Most common migration pattern. +- If the cluster is v1.20 or later and the text is in the payload, prefer BM25 on a full-text index over sparse vectors: create a `text` index with `scoring: true` (plus the language's stemmer and stopwords) on the text field and add a `text` query prefetch next to the dense one. It needs no schema change, no vector backfill, and no `avg_len`, and works on a collection of any age. Re-indexing a large collection can take a while [BM25 on a full-text index](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=bm25) +- Use the sparse vector options below on v1.19 or earlier, or when the text isn't stored in the payload. + - If the cluster is v1.18 or later, add the sparse vector field directly, even if the dense vector is unnamed [Update vector schema](https://skills.qdrant.tech/md/documentation/manage-data/collections/?s=update-vector-schema) - Generate sparse vectors for existing points and backfill with `UpdateVectors`; existing dense vectors stay as they are [Update vectors](https://skills.qdrant.tech/md/documentation/manage-data/points/?s=update-vectors) @@ -82,7 +86,7 @@ Use when: adding sparse/BM25 vectors to an existing dense-only collection. Most - Scroll the old collection with `with_vectors=True` to copy the dense vectors, and generate only the sparse vectors - Migrate payloads, swap alias -Sparse vectors at chunk level have different TF-IDF characteristics than document level. Test retrieval quality after migration, especially for non-English text without stop-word removal. +Sparse vectors and full-text BM25 statistics at chunk level have different TF-IDF characteristics than document level. Test retrieval quality after migration, especially for non-English text without stop-word removal. ## Re-embedding Is Too Slow diff --git a/skills/qdrant-monitoring/setup/SKILL.md b/skills/qdrant-monitoring/setup/SKILL.md index 6b0fe2a..499e8a5 100644 --- a/skills/qdrant-monitoring/setup/SKILL.md +++ b/skills/qdrant-monitoring/setup/SKILL.md @@ -39,7 +39,7 @@ Use when: configuring Kubernetes health checks. Use when: setting up alerts for production or Hybrid Cloud deployments. -- Qdrant Cloud provides pre-configured email alerts for 11 cluster issues. Use scraped metrics when you need custom alerts or routing to other targets. See [Cloud cluster monitoring](https://skills.qdrant.tech/md/documentation/cloud/cluster-monitoring/) +- Qdrant Cloud provides pre-configured email alerts for 11 cluster issues. Use scraped metrics when you need custom alerts or routing to other targets. See [Cloud cluster monitoring](https://skills.qdrant.tech/md/documentation/cloud/cluster-monitoring/) - Use AlertmanagerConfig to route alerts to Slack, PagerDuty, or other targets based on labels - At minimum, alert on: optimizer errors, node not ready, replication factor below target, disk usage >80% @@ -50,7 +50,7 @@ Use when: enterprise compliance requires centralized logs or audit trails. - Enable JSON log format for structured analysis: set `logger.format` to `json` in config [Configuration](https://skills.qdrant.tech/md/documentation/ops-configuration/configuration/) - Use FluentD/OpenSearch for log aggregation -- Audit logs (v1.17+) write to local filesystem (`/qdrant/storage/audit/`), not stdout. Mount a Persistent Volume and deploy a sidecar container to tail these files to stdout so DaemonSets can pick them up. [Audit logging](https://skills.qdrant.tech/md/documentation/security/?s=audit-logging) +- Audit logs (v1.17+) write to local filesystem (`/qdrant/storage/audit/`) by default. On v1.20 or newer, set `audit.output` to `stdout`, `stderr`, `file_and_stdout`, or `file_and_stderr` (env var `QDRANT__AUDIT__OUTPUT`) so a DaemonSet can collect them with no sidecar or Persistent Volume. Each line carries `"kind": "audit"` to tell it from regular logs, and the `/audit/logs` API returns nothing on a node that writes only to a stream. On v1.19 or older, mount a Persistent Volume and deploy a sidecar container to tail the files to stdout. [Audit logging](https://skills.qdrant.tech/md/documentation/security/?s=audit-log-output) ## What NOT to Do diff --git a/skills/qdrant-multitenancy/SKILL.md b/skills/qdrant-multitenancy/SKILL.md index 447abdd..61d447c 100644 --- a/skills/qdrant-multitenancy/SKILL.md +++ b/skills/qdrant-multitenancy/SKILL.md @@ -21,6 +21,7 @@ One collection holds every tenant. A payload field marks ownership, and a filter - Create a keyword payload index on the tenant field with `is_tenant=true` (the flag requires v1.11+). `is_tenant` tells Qdrant the field identifies tenants, so each tenant's vectors are stored together and served by sequential reads. Check [](https://api.qdrant.tech/api-reference/indexes/create-field-index). - At query time, isolate each tenant with a `must` filter on the tenant field. Without it, a query searches every tenant's data. Check [Payload-based multitenancy](https://skills.qdrant.tech/md/documentation/manage-data/multitenancy/?s=partition-by-payload). +- Text relevance (BM25 or sparse vectors with the IDF modifier) is scored with statistics computed over the whole shard, so tenants with different vocabularies distort each other's scores. On Qdrant 1.19 or newer, pass the `idf` search parameter with the tenant filter and index the tenant field; on 1.20 or newer this covers BM25 `text` queries on a full-text index too [Per-tenant scoring statistics](https://skills.qdrant.tech/md/documentation/manage-data/multitenancy/?s=per-tenant-scoring-statistics). - With this strategy, the indexing speed might become a bottleneck at scale because every tenant indexes into the same collection. To avoid this, you can disable the global HNSW creation (for the entire collection) and only build per-tenant indexes: set `m=0` and `payload_m` to a non-zero value. Although this accelerates the indexing process, keep in mind that requests without a tenant filter will become slower as they must scan all groups. So only make this trade if you hit the bottleneck and cross-tenant search is rare. [Calibrate performance](https://skills.qdrant.tech/md/documentation/manage-data/multitenancy/?s=calibrate-performance). diff --git a/skills/qdrant-performance-optimization/memory-usage-optimization/SKILL.md b/skills/qdrant-performance-optimization/memory-usage-optimization/SKILL.md index e9f96e1..bf03d39 100644 --- a/skills/qdrant-performance-optimization/memory-usage-optimization/SKILL.md +++ b/skills/qdrant-performance-optimization/memory-usage-optimization/SKILL.md @@ -7,7 +7,7 @@ description: "Diagnoses and reduces Qdrant memory usage. Use when someone report Qdrant operates with two types of memory: -- Resident memory (aka RSSAnon) - memory used for internal data structures like the ID tracker, plus components that stay fully in RAM. On Qdrant 1.19 or newer this is controlled per-component with `memory: pinned` (e.g. quantized vectors, payload indexes); see [memory tier legacy settings](https://skills.qdrant.tech/md/documentation/ops-configuration/memory-tiers?s=legacy-settings) for deployments on version 1.18 or older. +- Resident memory (aka RSSAnon) - memory used for internal data structures like the ID tracker, plus components that stay fully in RAM. As of Qdrant 1.20 the ID tracker has its own `memory` tier (default `pinned`; `cached` lets the OS evict it, `cold` needs no up-front RAM), where it was always resident before. On Qdrant 1.19 or newer this is controlled per-component with `memory: pinned` (e.g. quantized vectors, payload indexes); see [memory tier legacy settings](https://skills.qdrant.tech/md/documentation/ops-configuration/memory-tiers?s=legacy-settings) for deployments on version 1.18 or older. - OS page cache - memory used for caching disk reads, which can be released when needed. Original vectors are normally stored in page cache, so the service won't crash if RAM is full, but performance may degrade. On Qdrant 1.19 or newer this corresponds to `memory: cached` (pre-warmed into page cache at startup) or `memory: cold` (lazy disk reads, not pre-warmed); on 1.18 or older it's controlled via the `on_disk` boolean on vectors, HNSW config, sparse vector index, and payload index. See [Memory Tiers docs](https://skills.qdrant.tech/md/documentation/ops-configuration/memory-tiers/) (available on 1.19+). @@ -41,7 +41,7 @@ There are some scenarios, however, when it can be a good option: - Deployments with low latency disks - local NVMe or similar. - Multi-tenant deployments, where only a subset of tenants is frequently accessed, so that only a fraction of data & index is loaded in RAM at a time. -- For deployments with [inline storage](https://skills.qdrant.tech/md/documentation/ops-optimization/optimize/?s=inline-storage-in-hnsw-index) enabled. +- For deployments with [inline storage](https://skills.qdrant.tech/md/documentation/ops-optimization/optimize/?s=inline-storage-in-hnsw-index) enabled. On Qdrant 1.20 or newer, inline storage keeps the original vectors only in the HNSW index file, so `hnsw_config.memory` decides their tier and `vectors.memory` is ignored for those segments. Quantized vectors follow `hnsw_config.memory` unless quantization has its own explicit `memory`, so set it explicitly (for example `pinned`) when the HNSW index is `cold` or `cached` [Where Qdrant stores the vectors](https://skills.qdrant.tech/md/documentation/ops-optimization/optimize/?s=where-qdrant-stores-the-vectors). ## How to minimize memory footprint @@ -49,9 +49,9 @@ There are some scenarios, however, when it can be a good option: The main challenge is to put on disk those parts of data, which are rarely accessed. Here are the main techniques to achieve that: -- Use quantization to store only compressed vectors in RAM [Quantization docs](https://skills.qdrant.tech/md/documentation/manage-data/quantization/) +- Use quantization to store only compressed vectors in RAM. On Qdrant 1.20 or newer, 8-bit TurboQuant (`bits8`, 4x compression) is the recommended default over scalar quantization at the same compression (use scalar for the Manhattan metric), and it doesn't rescore by default, unlike `bits2`, `bits1_5`, and `bits1` [Quantization docs](https://skills.qdrant.tech/md/documentation/manage-data/quantization/) -- Use float16 or uint8 datatypes to reduce memory usage of vectors by 2x or 4x respectively, with some tradeoff in precision. On Qdrant 1.19 or newer, the `turbo4` datatype (TurboQuant-based, 4 bits/dimension, dense vectors only) reduces memory by ~8x, and can be paired with 1-bit quantization for cheaper rescoring than pairing 1-bit quantization with full-precision vectors. Read more about vector datatypes in [documentation](https://skills.qdrant.tech/md/documentation/manage-data/vectors/?s=datatypes) +- Use float16 or uint8 datatypes to reduce memory usage of vectors by 2x or 4x respectively, with some tradeoff in precision. `uint8` only makes sense for vectors already quantized to 0-255. On Qdrant 1.19 or newer, the TurboQuant datatypes replace the original vectors with a compressed form (dense vectors and multivectors, not sparse): `turbo4` (4 bits/dimension, ~8x, from 1.19), `turbo8` (~4x, closer to `float32` recall, from 1.20), and `turbo16` (~2x, from 1.20, closer to `float32` than `float16`). Any of them can be paired with 1-bit quantization for cheaper rescoring than pairing 1-bit quantization with full-precision vectors. A TurboQuant datatype is not TurboQuant quantization, which keeps the originals and adds a copy. Read more about vector datatypes in [documentation](https://skills.qdrant.tech/md/documentation/manage-data/vectors/?s=datatypes) - Leverage Matryoshka Representation Learning (MRL) to store only small vectors in RAM while keeping large vectors on disk. Examples of how to use MRL with Qdrant Cloud inference: [MRL docs](https://skills.qdrant.tech/md/documentation/inference/matryoshka-models/?s=reduce-vector-dimensionality-with-matryoshka-models) diff --git a/skills/qdrant-performance-optimization/search-speed-optimization/SKILL.md b/skills/qdrant-performance-optimization/search-speed-optimization/SKILL.md index cc351ef..f528fa2 100644 --- a/skills/qdrant-performance-optimization/search-speed-optimization/SKILL.md +++ b/skills/qdrant-performance-optimization/search-speed-optimization/SKILL.md @@ -49,7 +49,8 @@ Use when: filtered search is significantly slower than unfiltered. Most common S - Create payload index on the filtered field [Payload index](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=payload-index) - Use `is_tenant=true` for primary filtering condition: [Tenant index](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=tenant-index) - Try ACORN algorithm for complex filters: [ACORN](https://skills.qdrant.tech/md/documentation/search/search/?s=acorn-search-algorithm) -- Avoid using `nested` filtering conditions as a primary filter. It might force qdrant to read raw payload values instead of using index. +- Avoid using `nested` filtering conditions as a primary filter. It might force qdrant to read raw payload values instead of using index. On Qdrant 1.20 or newer, create per-element payload indexes (`nested` parameter set to the array key) on every field the nested filter uses; if all its conditions are match, range, geo, full-text, `is_empty`, or `is_null` on per-element-indexed fields of the same array, Qdrant answers it from the indexes alone. Note `is_empty` semantics differ on these indexes, and budget ~4 bytes per point per index in RAM [Per-element index](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=per-element-index-for-arrays-of-objects) [Nested object filter](https://skills.qdrant.tech/md/documentation/search/filtering/?s=nested-object-filter) +- Substring match (`match` with `substring`, 1.20+) is expensive: it checks every distinct value of the field, and Qdrant rejects it whenever strict mode is enabled (the Qdrant Cloud default). Prefer prefix match (keyword index with `prefix: true`) or full-text match where the use case allows. A keyword index, especially with `prefix`, speeds it up; a full-text index doesn't help [Substring match](https://skills.qdrant.tech/md/documentation/search/filtering/?s=substring-match) - If payload index was added after HNSW build, trigger re-index to create filterable subgraph links diff --git a/skills/qdrant-scaling/scaling-data-volume/vertical-scaling/SKILL.md b/skills/qdrant-scaling/scaling-data-volume/vertical-scaling/SKILL.md index abbae72..8a441f1 100644 --- a/skills/qdrant-scaling/scaling-data-volume/vertical-scaling/SKILL.md +++ b/skills/qdrant-scaling/scaling-data-volume/vertical-scaling/SKILL.md @@ -5,7 +5,7 @@ description: "Guides Qdrant vertical scaling decisions. Use when someone asks 'h # What to Do When Qdrant Needs to Scale Vertically -Vertical scaling means increasing CPU, RAM, or disk on existing nodes rather than adding more nodes. This is the recommended first step before considering horizontal scaling. Vertical scaling is simpler and avoids distributed system complexity. +Vertical scaling means increasing CPU, RAM, or disk on existing nodes rather than adding more nodes. This is the recommended first step before considering horizontal scaling. Vertical scaling is simpler and avoids distributed system complexity. - Vertical scaling for Qdrant Cloud is done through the [Qdrant Cloud Console](https://cloud.qdrant.io/) - For self-hosted deployments, resize the underlying VM or container resources @@ -35,7 +35,8 @@ Use when: current node resources (RAM, CPU, disk) are insufficient, but the work - Estimate RAM per component (vectors, HNSW index, ID tracker, payload indexes) for the structures you keep in RAM, using the formulas in [Capacity planning](https://skills.qdrant.tech/md/documentation/capacity-planning/?s=calculating-ram-and-disk-size). For a full sizing walkthrough, see [qdrant-sizing](../../../qdrant-sizing/SKILL.md) - Quantization adds a compressed copy alongside the original vectors. RAM for vectors drops only when the originals move to disk (`memory: cold`) and the quantized copy stays in RAM. With scalar quantization, the copy is 1/4 the size (INT8 reduces each float32 to 1 byte) [Quantization](https://skills.qdrant.tech/md/documentation/manage-data/quantization/) - With binary quantization, the copy is 1/32 the size [Binary quantization](https://skills.qdrant.tech/md/documentation/manage-data/quantization/?s=binary-quantization) -- On Qdrant 1.19 or newer, the `turbo4` datatype (dense vectors only) divides by ~8 on its own, without needing separate quantization [Vector datatypes](https://skills.qdrant.tech/md/documentation/manage-data/vectors/?s=datatypes) +- On Qdrant 1.19 or newer, the TurboQuant datatypes shrink vectors on their own, without separate quantization: `turbo4` by ~8x (1.19+), `turbo8` by ~4x and `turbo16` by ~2x (both 1.20+, higher recall). They apply to dense vectors and multivectors [Vector datatypes](https://skills.qdrant.tech/md/documentation/manage-data/vectors/?s=datatypes) +- On Qdrant 1.20 or newer, the ID tracker can move to `cached` or `cold` memory tiers (previously always resident), which frees RAM at very high point counts [Memory tiers](https://skills.qdrant.tech/md/documentation/ops-configuration/memory-tiers/) - Add overhead for the HNSW index (about 150 bytes per point at the default `m: 16`) and payload indexes; the WAL counts against disk, not RAM [Capacity planning](https://skills.qdrant.tech/md/documentation/capacity-planning/) - Reserve 20% headroom for optimizer operations and OS cache - Monitor actual usage via Grafana/Prometheus before and after resizing [Monitoring](../../../qdrant-monitoring/SKILL.md) diff --git a/skills/qdrant-search-quality/diagnosis/SKILL.md b/skills/qdrant-search-quality/diagnosis/SKILL.md index 1894f96..99468d8 100644 --- a/skills/qdrant-search-quality/diagnosis/SKILL.md +++ b/skills/qdrant-search-quality/diagnosis/SKILL.md @@ -31,7 +31,7 @@ Use when: results are irrelevant or missing expected matches and you need to iso - Check if filters are too restrictive (then you might need to use ACORN) - If duplicate results from chunked documents, use Grouping API to deduplicate [Grouping](https://skills.qdrant.tech/md/documentation/search/search/?s=grouping-api) -Payload filtering and sparse vector search are different things. Metadata (dates, categories, tags) goes in payload for filtering. Text content goes in sparse vectors for search. +Payload filtering and ranked text search are different things. Metadata (dates, categories, tags) goes in payload for filtering. To rank text content, on Qdrant 1.20 or newer use BM25 on a full-text index with `scoring: true` (queried with a `text` query); on older versions, or for learned sparse models, use sparse vectors. A full-text index without `scoring` only filters. ## Approximate Search Worse Than Exact @@ -72,4 +72,4 @@ Use when: exact search also returns bad results and model choice is confirmed by - Skip payload indexes on filtered fields then blame quality (HNSW can't traverse filtered-out nodes, and filterable HNSW is built only if payload indexes were set up prior) - Deploy without baseline recall or other search relevance metrics (no way to measure regressions) - Compare two configs on a query set too small to resolve the difference between them (the result is noise, not evidence) -- Confuse payload filtering with sparse vector search (different things, different config) +- Confuse payload filtering with ranked text search (different things, different config) diff --git a/skills/qdrant-search-quality/search-strategies/hybrid-search/SKILL.md b/skills/qdrant-search-quality/search-strategies/hybrid-search/SKILL.md index 7205187..7f7bc7f 100644 --- a/skills/qdrant-search-quality/search-strategies/hybrid-search/SKILL.md +++ b/skills/qdrant-search-quality/search-strategies/hybrid-search/SKILL.md @@ -1,6 +1,6 @@ --- name: qdrant-hybrid-search -description: "Explains hybrid search in Qdrant. Use when someone asks 'how do I setup hybrid search?', 'how to combine keyword and semantic search?', 'sparse plus dense vectors?', 'missing keyword matches', 'how to combine results from multiple searches?' and 'combining multiple representations'. Also use for how a hybrid query is scoped: 'how is IDF scoped?', 'can one tenant's data contaminate another tenant's scoring?'" +description: "Explains hybrid search in Qdrant. Use when someone asks 'how do I setup hybrid search?', 'how to combine keyword and semantic search?', 'sparse plus dense vectors?', 'missing keyword matches', 'how to combine results from multiple searches?' and 'combining multiple representations'. Also use for how a hybrid query is scoped: 'how is IDF scoped?', 'can one tenant's data contaminate another tenant's scoring?'. Also 'BM25 on a text index or sparse vectors?'" allowed-tools: - Read - Grep @@ -29,7 +29,7 @@ Use when: different tenants share one collection and you need to understand hybr If user wants to isolate/share hybrid search pipelines between tenants, consider that: -- Indexes (sparse, payload and dense) and [IDF modifier](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=idf-modifier) for sparse vectors are computed independently **per shard**, not per **tenant**, by default — payload-based tenant partitioning alone does not isolate IDF statistics. On Qdrant 1.19 or newer, the `idf` search param can scope IDF statistics to a payload-filtered corpus (requires a payload index on the filtered field), giving each tenant properly isolated BM25 scoring instead of shard-wide statistics. +- Indexes (sparse, payload and dense) and [IDF modifier](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=idf-modifier) for sparse vectors are computed independently **per shard**, not per **tenant**, by default — payload-based tenant partitioning alone does not isolate IDF statistics. On Qdrant 1.19 or newer, the `idf` search param can scope IDF statistics to a payload-filtered corpus (requires a payload index on the filtered field), giving each tenant properly isolated BM25 scoring instead of shard-wide statistics. On 1.20 or newer it applies to BM25 `text` queries on a full-text index too, where it scopes the average document length as well [Per-tenant scoring statistics](https://skills.qdrant.tech/md/documentation/manage-data/multitenancy/?s=per-tenant-scoring-statistics). - Prefetch runs independently per shard to retrieve #limit results, so for collection-level prefetches if collection has several shards, Qdrant will always prefetch under the hood #limit * #shard results. Final results are merged based on scores. - In nested prefetches (deeper than 1 level), methods described in "Combining Searches" might be done on a shard level first, then per-shards results once again will be merged based on scores. diff --git a/skills/qdrant-search-quality/search-strategies/hybrid-search/combining-searches/SKILL.md b/skills/qdrant-search-quality/search-strategies/hybrid-search/combining-searches/SKILL.md index c26a701..1a10234 100644 --- a/skills/qdrant-search-quality/search-strategies/hybrid-search/combining-searches/SKILL.md +++ b/skills/qdrant-search-quality/search-strategies/hybrid-search/combining-searches/SKILL.md @@ -37,6 +37,8 @@ If you want to implement custom fusion on `score` of each prefetch: When using `FormulaQuery` over multiple prefetches (e.g. per-representation weighting): - `$score[i]` indexes prefetches in declaration order. Reordering the `prefetch=` list silently shifts which weight applies to which retriever. - Provide `defaults` for every `$score[i]` so the formula still evaluates for candidates that surfaced from only a subset of prefetches. +- On Qdrant 1.19.1 or newer, `max` and `min` expressions take the best or worst of several `$score[i]` (for example, keep each point's best score across per-field BM25 prefetches), cap a boost (`min(cap, boost)`), or clamp a value with `max(lo, min(hi, x))`. A point absent from a prefetch has `$score[i]` of `0.0`, which wins `max` over negative scores and `min` over positive ones, so set `defaults` (keys include the `$`, placed beside `formula`, never inside it) to a value outside the expected score range such as `-1e30` for `max` or `1e30` for `min` [Max and min](https://skills.qdrant.tech/md/documentation/search/search-relevance/?s=max-and-min). +- On 1.19.1 or newer, `acosh` supports exact geodesic-distance rescoring for hyperbolic embeddings (Lorentz model with `Dot` distance, `acosh(-$score)`, negated so closest ranks first). The input must be at least 1 [Hyperbolic distance](https://skills.qdrant.tech/md/documentation/search/search-relevance/?s=hyperbolic-distance). - Start with RRF when scores are on incomparable scales (e.g. BM25 + cosine). Reach for `FormulaQuery` only when explicit per-representation weighting or payload-driven boosts are required, and normalize each `$score[i]` (decay or min-max on a sampled distribution) before combining linearly. ## Need Good Ranking of Fused Candidates and Ready To Spend More Resources diff --git a/skills/qdrant-search-quality/search-strategies/hybrid-search/search-types/SKILL.md b/skills/qdrant-search-quality/search-strategies/hybrid-search/search-types/SKILL.md index d3efa7b..911f192 100644 --- a/skills/qdrant-search-quality/search-strategies/hybrid-search/search-types/SKILL.md +++ b/skills/qdrant-search-quality/search-strategies/hybrid-search/search-types/SKILL.md @@ -1,6 +1,6 @@ --- name: qdrant-hybrid-search-prefetches -description: "Constructing prefetch queries for hybrid retrieval, including sparse/dense and multi-field setups, and choosing a sparse embedding model. Use when someone asks 'dense and sparse in one search?', 'how to combine multiple fields for retrieval?', 'payloads or sparse vectors for lexical?', 'which sparse embedding model to use?', or 'BM25 vs SPLADE?'" +description: "Constructing prefetch queries for hybrid retrieval, including BM25/dense and multi-field setups, and choosing a lexical retrieval method. Use when someone asks 'dense and sparse in one search?', 'how to combine multiple fields for retrieval?', 'payloads or sparse vectors for lexical?', 'BM25 on a full-text index or sparse vectors?', 'which sparse embedding model to use?', 'BM25 vs SPLADE?', or 'how to migrate BM25 sparse vectors to a text index?'" --- # Different Searches in One Query API Request @@ -19,24 +19,38 @@ If second, use [named vectors](https://skills.qdrant.tech/md/documentation/manag Use when: pure vector search misses exact term or keyword matches and you need lexical retrieval alongside semantic search. -Most likely you need a sparse vector for exact text search alongside the dense one. Qdrant uses sparse vectors for lexical searches, as [payload filtering doesn't provide any ranking score](https://skills.qdrant.tech/md/documentation/search/text-search/?s=filtering-versus-querying). +Most likely you need a lexical retriever alongside the dense one. [Payload filtering doesn't provide any ranking score](https://skills.qdrant.tech/md/documentation/search/text-search/?s=filtering-versus-querying), so rank text with BM25 or a sparse vector. -### Choose a Sparse Vector for Text -- **BM25** statistical representations, built into Qdrant core (computed server-side). Good baseline, works out-of-domain, usually for long texts. Can be used for non-English content with per-language configuration. More in [Text Search Guide](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=bm25) +### Choose a Lexical Retriever for Text +- **BM25 on a full-text index** (Qdrant 1.20 or newer, recommended default). Create a `text` payload index with `scoring: true` on the field and query it with a `text` query, `using` set to the field name. Works on text already stored in the payload, so there are no embeddings to generate or store, existing data is indexed without re-upload, and the average document length is computed from your data. Text processing (tokenizer, stemming, stopwords, ASCII folding) comes from the index settings and defaults to no stemming and no stopwords, so configure the language. `k` and `b` are set per query. Scoring also enables phrase matching and costs extra memory and disk for document lengths and token positions. [BM25 on a full-text index](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=bm25) [BM25 scoring index](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=bm25-scoring) +- **BM25 sparse vector model**, built into Qdrant core (computed server-side by the `qdrant/bm25` inference model or client-side with FastEmbed). Use it on Qdrant 1.19 or older, when the text isn't stored in the payload, or when you need the sparse vector itself, for example for Edge compatibility. Good baseline, works out-of-domain, usually for long texts. Can be used for non-English content with per-language configuration. More in [Sparse Vector BM25](https://skills.qdrant.tech/md/documentation/search/text-search/sparse-vector-search/?s=bm25) - **BM42** learned sparse, based on BM25, but better for small chunks of text & with meaning understanding. Works only on English. Requires fine-tuning for domain-specific retrieval. Requires FastEmbed (Python/REST only, not available in all SDKs). Not maintained. - **miniCOIL** learned sparse, BM25 with additional understanding of words meaning in context. Works only on English. Requires fine-tuning for domain-specific retrieval. Requires FastEmbed. Usage shown in [FastEmbed miniCOIL documentation](https://skills.qdrant.tech/md/documentation/fastembed/fastembed-minicoil/). - **SPLADE++** learned sparse with term expansion. Heavier inference and resources usage but better performance due to term expansion. Requires fine-tuning for domain-specific retrieval. Provided in Qdrant Cloud Inference and FastEmbed versions work only on English. To use with FastEmbed, check [FastEmbed SPLADE documentation](https://skills.qdrant.tech/md/documentation/fastembed/fastembed-splade/). - **External learned sparse embeddings**, for example BAAI/bge-m3. -What to remember when using sparse vectors for lexical search: +What to remember when using any lexical retriever: - tokenization and stemming affect exact matches, especially on custom codes, terms, etc. -- On Qdrant 1.18 and older, IDF statistics are computed over the whole shard, which distorts scoring in multi-tenant collections where tenants have different vocabularies. On 1.19 or newer, scope them to one tenant with the `idf` search parameter, and index the tenant field. See [Per-tenant IDF statistics](https://skills.qdrant.tech/md/documentation/manage-data/multitenancy/?s=per-tenant-idf-statistics). - -What to remember when using Qdrant BM25 and miniCOIL (based on BM25): -- Calibrate `avg_len` per field: Qdrant doesn't compute it, and the default assumes document-length text, so short fields (titles, tags) need a much smaller value or BM25 overweights term frequency [BM25](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=bm25). -- BM25 might be not good for small chunks of text, as BM25 algorithm was initially created for search on long documents; consider adjusting document statistics in sparse vectors (TF & IDF, k, b). +- On Qdrant 1.18 and older, IDF statistics are computed over the whole shard, which distorts scoring in multi-tenant collections where tenants have different vocabularies. On 1.19 or newer, scope them to one tenant with the `idf` search parameter, and index the tenant field. On 1.20 or newer this also applies to BM25 `text` queries on a full-text index. See [Per-tenant scoring statistics](https://skills.qdrant.tech/md/documentation/manage-data/multitenancy/?s=per-tenant-scoring-statistics). +- BM25 might be not good for small chunks of text, as BM25 algorithm was initially created for search on long documents; consider adjusting document statistics (TF & IDF, k, b). +- Statistics are per shard, and deleted or updated points can skew them slightly until the optimizer rebuilds segments, so scores can shift without a data change [Corpus statistics](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=corpus-statistics). + +What to remember when using BM25 on a full-text index: +- Set stemmer and stopwords for the data's language on the index. Without them, BM25 matches only exact lowercased tokens. For language-neutral content (names, codes) use the `multilingual` tokenizer and leave stemming and stopwords off [Text processing](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=text-processing). +- A `text` query returns points that contain at least one query term, and repeated query terms count once. Add a full-text filter to require or exclude terms. +- If the text field is inside an array of objects (reviews, passages), create the index with `nested` so each element is scored as its own document and the point gets its best element's score [Score elements separately](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=score-elements-of-an-array-separately). +- Changing `scoring` on an existing index rebuilds it from the stored payload without re-upload, but can take a while on large collections [BM25 scoring](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=bm25-scoring). + +What to remember when using the BM25 sparse vector model and miniCOIL (based on BM25): +- Calibrate `avg_len` per field: Qdrant doesn't compute it for sparse vectors, and the default (256) assumes document-length text, so short fields (titles, tags) need a much smaller value or BM25 overweights term frequency. This is one reason to prefer a full-text index on 1.20 or newer [Sparse vector BM25](https://skills.qdrant.tech/md/documentation/search/text-search/sparse-vector-search/?s=configuring-bm25-parameters). - Qdrant BM25 vectors are configured per language, so consider customizing stop words, stemming & tokenization when users documents mix several languages or carefully configure vectors per point when they are monolingual. To disable text processing entirely for language-neutral content: on Qdrant 1.19 or newer, use `stemmer: {"type": "none"}` plus an empty `stopwords` set explicitly (by default, BM25 applies English stemming and stopword removal); on 1.18 or older, use `language: none` (deprecated as of 1.19). +### Migrate from Sparse Vector BM25 to a Full-Text Index + +Use when: the user already runs sparse vector BM25 on Qdrant 1.20 or newer and the text is stored in the payload. + +- Create a `text` index with `scoring: true` on the payload field, mirror the sparse model's text processing (the sparse model defaults to English stemming and stopwords, a full-text index doesn't, so set the `english` stemmer and stopwords explicitly, and match tokenizer and ASCII folding), then switch queries from the sparse vector to `text` queries. Drop `avg_len`. Scores will differ, so re-evaluate [Migrate from sparse vector BM25](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=migrate-from-sparse-vector-bm25). + More on [Sparse Vectors for Text Search](https://skills.qdrant.tech/md/course/essentials/day-3/sparse-retrieval-demo/) ## Need to Combine Multiple Representations of the Same Item @@ -47,7 +61,7 @@ Use multiple named vector prefetches, each prefetch covers one representation. A representation only earns its own prefetch if it carries signal independent of the others — e.g. title vocabulary the body never repeats, or an abstract treated as a single semantic unit vs. individual chunks. Don't add a prefetch per field reflexively; verify each candidate contributes content the other vectors don't. -When a representation's signal is mostly lexical — keyword-driven titles, codes, tags, or other short fields — prefer a sparse named vector (e.g. BM25) over an additional dense embedding. Server-side BM25 in Qdrant avoids the inference cost of another dense model and stores far less per point. Skip this when the field carries paraphrase or conceptual signal that exact-term matching would miss. +When a representation's signal is mostly lexical — keyword-driven titles, codes, tags, or other short fields — prefer BM25 over an additional dense embedding. On Qdrant 1.20 or newer, create a BM25 full-text index (`scoring: true`) per payload field and run one `text` query prefetch per field, with no extra named vector. On older versions, use a sparse named vector with BM25. Server-side BM25 in Qdrant avoids the inference cost of another dense model and stores far less per point. Skip this when the field carries paraphrase or conceptual signal that exact-term matching would miss. - End-to-end worked example fusing title, abstract, chunk, and sparse-title named vectors with RRF and document-level grouping in one Query API call: [Multi-Representation Search tutorial](https://skills.qdrant.tech/md/documentation/tutorials-search-engineering/multi-representation-search/) - If you have groups and subgroups of representations (document -> chunk, image -> patch), you could use [searching in groups](https://skills.qdrant.tech/md/documentation/search/search/?s=search-groups). To not store identical payloads several times, check [Lookup in Groups](https://skills.qdrant.tech/md/documentation/search/search/#lookup-in-groups). Index the grouping payload field (e.g. `document_id`) as a keyword payload index before grouping. @@ -67,4 +81,6 @@ There are ways to make multivector retrieval cheaper (MUVERA, pooling), you can ## What NOT to Do - Choose any search method (for example, BM25) without evaluation of its quality & resources used. - Use any search method (for example, BM25) without paying attention to the specifics of their configuration and applicability to the use case. +- On Qdrant 1.20 or newer, default to sparse vector BM25 with a guessed `avg_len` when the text is already in the payload; a full-text index with `scoring: true` computes it from the data. +- Run BM25 on a full-text index without configuring stemmer and stopwords for non-English data, or without `scoring: true` (a plain full-text index only filters). diff --git a/skills/qdrant-sizing/SKILL.md b/skills/qdrant-sizing/SKILL.md index e0dcc3d..974e4b3 100644 --- a/skills/qdrant-sizing/SKILL.md +++ b/skills/qdrant-sizing/SKILL.md @@ -24,7 +24,9 @@ Use when: someone asks how much RAM or disk they need, how much data should be k - Read [Calculating RAM and Disk Size](https://skills.qdrant.tech/md/documentation/capacity-planning/?s=calculating-ram-and-disk-size) and compute each component with its formula. Do not estimate from memory: the formulas and defaults change between versions. - Size each component separately: dense vectors, quantized vectors, HNSW indexes, sparse vectors and their indexes, payload, payload indexes, and the ID tracker. Every component scales with `base = points × replication_factor`. - For multiple named vectors or payload fields per point, size each one separately, then sum them. -- Count quantized vectors on top of the originals: Qdrant stores the compressed copy alongside the original vectors, not instead of them. +- Count quantized vectors on top of the originals: Qdrant stores the compressed copy alongside the original vectors, not instead of them. A TurboQuant vector datatype is different: it replaces the originals, at 2 bytes per dimension for `turbo16` (1.20+), 1 for `turbo8` (1.20+), and 0.5 for `turbo4` (1.19+), against 4 for `float32`. Dense vectors can have 1 to 65,536 dimensions. +- A full-text index with `scoring: true` (BM25, 1.20+) stores document lengths and token positions, so it needs more RAM and disk than a filter-only full-text index. +- A per-element payload index (the `nested` option for arrays of objects, 1.20+) also keeps an element-to-point map in RAM regardless of its memory tier: budget about 4 bytes per point for each such index. - Count payload indexes only for fields used for filtering; index only fields that are frequently filtered on. ### Decide what needs to be loaded in RAM @@ -35,6 +37,8 @@ On Qdrant 1.19 or newer, set this per structure with `memory: pinned`, `cached`, **Recommendations:** - Keep HNSW, inverted indexes for sparse vectors, and payload indexes in RAM for faster search. +- On Qdrant 1.20 or newer the ID tracker has its own memory tier (`pinned` by default, so it counts toward RAM). `cached` lets the OS evict it, and `cold` needs no up-front RAM. +- With inline storage on 1.20 or newer, the original vectors live only in the HNSW index file, and `hnsw_config.memory` sets their tier, not `vectors.memory`. Quantized vectors follow it too unless their `memory` is set explicitly. - Pin quantized vectors in RAM if they fit comfortably in the available memory, and move the original vectors to `cold` for rescoring. This is the main way quantization reduces RAM. - If your use case involves splitting vectors into multiple collections or subgroups based on payload values (for example, serving searches for multiple users, each with their own subset of vectors), store vectors in the `cold` memory tier and size RAM from the active subset of points, not the full collection [Subgroup-oriented configuration](https://skills.qdrant.tech/md/documentation/capacity-planning/?s=subgroup-oriented-configuration). - With vectors on disk, the amount of RAM drives latency: keeping half as many vectors in RAM roughly doubles search latency [Storage-focused configuration](https://skills.qdrant.tech/md/documentation/capacity-planning/?s=storage-focused-configuration). From 1b14d5fd7477afa0cd1c466d0f3d105efaca3412 Mon Sep 17 00:00:00 2001 From: Abdon Pijpelink Date: Thu, 22 Oct 2026 14:18:50 +0200 Subject: [PATCH 2/5] Say "with `scoring` enabled" instead of `scoring: true` --- skills/qdrant-model-migration/SKILL.md | 4 ++-- skills/qdrant-search-quality/diagnosis/SKILL.md | 2 +- .../hybrid-search/search-types/SKILL.md | 10 +++++----- skills/qdrant-sizing/SKILL.md | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/skills/qdrant-model-migration/SKILL.md b/skills/qdrant-model-migration/SKILL.md index 9cb4148..d67f7e2 100644 --- a/skills/qdrant-model-migration/SKILL.md +++ b/skills/qdrant-model-migration/SKILL.md @@ -18,7 +18,7 @@ You MUST re-embed if: changing model provider (OpenAI to Cohere), changing archi You do NOT need to re-embed existing dense vectors if: -- Adding BM25 for hybrid search on v1.20+: no vectors to generate. Create a `text` payload index with `scoring: true` on the payload field; Qdrant builds it from the stored payload [BM25 scoring](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=bm25-scoring) +- Adding BM25 for hybrid search on v1.20+: no vectors to generate. Create a `text` payload index with `scoring` enabled on the payload field; Qdrant builds it from the stored payload [BM25 scoring](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=bm25-scoring) - Adding sparse vectors for hybrid search: generate only the sparse vectors. On v1.18+, add the sparse field to the existing collection and backfill it with `UpdateVectors`. On v1.17 or earlier, copy the dense vectors into the new collection instead of recomputing them [Update vectors](https://skills.qdrant.tech/md/documentation/manage-data/points/?s=update-vectors) - Using Matryoshka models: use the `dimensions` parameter to output lower-dimensional embeddings (some recall loss, good for 100M+ datasets) - Changing quantization (binary to scalar): Qdrant re-quantizes automatically [Quantization](https://skills.qdrant.tech/md/documentation/manage-data/quantization/) @@ -74,7 +74,7 @@ If you anticipate future model migrations, define both vector fields upfront at Use when: adding sparse/BM25 vectors to an existing dense-only collection. Most common migration pattern. -- If the cluster is v1.20 or later and the text is in the payload, prefer BM25 on a full-text index over sparse vectors: create a `text` index with `scoring: true` (plus the language's stemmer and stopwords) on the text field and add a `text` query prefetch next to the dense one. It needs no schema change, no vector backfill, and no `avg_len`, and works on a collection of any age. Re-indexing a large collection can take a while [BM25 on a full-text index](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=bm25) +- If the cluster is v1.20 or later and the text is in the payload, prefer BM25 on a full-text index over sparse vectors: create a `text` index with `scoring` enabled (plus the language's stemmer and stopwords) on the text field and add a `text` query prefetch next to the dense one. It needs no schema change, no vector backfill, and no `avg_len`, and works on a collection of any age. Re-indexing a large collection can take a while [BM25 on a full-text index](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=bm25) - Use the sparse vector options below on v1.19 or earlier, or when the text isn't stored in the payload. - If the cluster is v1.18 or later, add the sparse vector field directly, even if the dense vector is unnamed [Update vector schema](https://skills.qdrant.tech/md/documentation/manage-data/collections/?s=update-vector-schema) diff --git a/skills/qdrant-search-quality/diagnosis/SKILL.md b/skills/qdrant-search-quality/diagnosis/SKILL.md index 99468d8..2c10f69 100644 --- a/skills/qdrant-search-quality/diagnosis/SKILL.md +++ b/skills/qdrant-search-quality/diagnosis/SKILL.md @@ -31,7 +31,7 @@ Use when: results are irrelevant or missing expected matches and you need to iso - Check if filters are too restrictive (then you might need to use ACORN) - If duplicate results from chunked documents, use Grouping API to deduplicate [Grouping](https://skills.qdrant.tech/md/documentation/search/search/?s=grouping-api) -Payload filtering and ranked text search are different things. Metadata (dates, categories, tags) goes in payload for filtering. To rank text content, on Qdrant 1.20 or newer use BM25 on a full-text index with `scoring: true` (queried with a `text` query); on older versions, or for learned sparse models, use sparse vectors. A full-text index without `scoring` only filters. +Payload filtering and ranked text search are different things. Metadata (dates, categories, tags) goes in payload for filtering. To rank text content, on Qdrant 1.20 or newer use BM25 on a full-text index with `scoring` enabled (queried with a `text` query); on older versions, or for learned sparse models, use sparse vectors. A full-text index without `scoring` only filters. ## Approximate Search Worse Than Exact diff --git a/skills/qdrant-search-quality/search-strategies/hybrid-search/search-types/SKILL.md b/skills/qdrant-search-quality/search-strategies/hybrid-search/search-types/SKILL.md index 911f192..f4e2931 100644 --- a/skills/qdrant-search-quality/search-strategies/hybrid-search/search-types/SKILL.md +++ b/skills/qdrant-search-quality/search-strategies/hybrid-search/search-types/SKILL.md @@ -22,7 +22,7 @@ Use when: pure vector search misses exact term or keyword matches and you need l Most likely you need a lexical retriever alongside the dense one. [Payload filtering doesn't provide any ranking score](https://skills.qdrant.tech/md/documentation/search/text-search/?s=filtering-versus-querying), so rank text with BM25 or a sparse vector. ### Choose a Lexical Retriever for Text -- **BM25 on a full-text index** (Qdrant 1.20 or newer, recommended default). Create a `text` payload index with `scoring: true` on the field and query it with a `text` query, `using` set to the field name. Works on text already stored in the payload, so there are no embeddings to generate or store, existing data is indexed without re-upload, and the average document length is computed from your data. Text processing (tokenizer, stemming, stopwords, ASCII folding) comes from the index settings and defaults to no stemming and no stopwords, so configure the language. `k` and `b` are set per query. Scoring also enables phrase matching and costs extra memory and disk for document lengths and token positions. [BM25 on a full-text index](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=bm25) [BM25 scoring index](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=bm25-scoring) +- **BM25 on a full-text index** (Qdrant 1.20 or newer, recommended default). Create a `text` payload index with `scoring` enabled on the field and query it with a `text` query, `using` set to the field name. Works on text already stored in the payload, so there are no embeddings to generate or store, existing data is indexed without re-upload, and the average document length is computed from your data. Text processing (tokenizer, stemming, stopwords, ASCII folding) comes from the index settings and defaults to no stemming and no stopwords, so configure the language. `k` and `b` are set per query. Scoring also enables phrase matching and costs extra memory and disk for document lengths and token positions. [BM25 on a full-text index](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=bm25) [BM25 scoring index](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=bm25-scoring) - **BM25 sparse vector model**, built into Qdrant core (computed server-side by the `qdrant/bm25` inference model or client-side with FastEmbed). Use it on Qdrant 1.19 or older, when the text isn't stored in the payload, or when you need the sparse vector itself, for example for Edge compatibility. Good baseline, works out-of-domain, usually for long texts. Can be used for non-English content with per-language configuration. More in [Sparse Vector BM25](https://skills.qdrant.tech/md/documentation/search/text-search/sparse-vector-search/?s=bm25) - **BM42** learned sparse, based on BM25, but better for small chunks of text & with meaning understanding. Works only on English. Requires fine-tuning for domain-specific retrieval. Requires FastEmbed (Python/REST only, not available in all SDKs). Not maintained. - **miniCOIL** learned sparse, BM25 with additional understanding of words meaning in context. Works only on English. Requires fine-tuning for domain-specific retrieval. Requires FastEmbed. Usage shown in [FastEmbed miniCOIL documentation](https://skills.qdrant.tech/md/documentation/fastembed/fastembed-minicoil/). @@ -49,7 +49,7 @@ What to remember when using the BM25 sparse vector model and miniCOIL (based on Use when: the user already runs sparse vector BM25 on Qdrant 1.20 or newer and the text is stored in the payload. -- Create a `text` index with `scoring: true` on the payload field, mirror the sparse model's text processing (the sparse model defaults to English stemming and stopwords, a full-text index doesn't, so set the `english` stemmer and stopwords explicitly, and match tokenizer and ASCII folding), then switch queries from the sparse vector to `text` queries. Drop `avg_len`. Scores will differ, so re-evaluate [Migrate from sparse vector BM25](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=migrate-from-sparse-vector-bm25). +- Create a `text` index with `scoring` enabled on the payload field, mirror the sparse model's text processing (the sparse model defaults to English stemming and stopwords, a full-text index doesn't, so set the `english` stemmer and stopwords explicitly, and match tokenizer and ASCII folding), then switch queries from the sparse vector to `text` queries. Drop `avg_len`. Scores will differ, so re-evaluate [Migrate from sparse vector BM25](https://skills.qdrant.tech/md/documentation/search/text-search/full-text-search/?s=migrate-from-sparse-vector-bm25). More on [Sparse Vectors for Text Search](https://skills.qdrant.tech/md/course/essentials/day-3/sparse-retrieval-demo/) @@ -61,7 +61,7 @@ Use multiple named vector prefetches, each prefetch covers one representation. A representation only earns its own prefetch if it carries signal independent of the others — e.g. title vocabulary the body never repeats, or an abstract treated as a single semantic unit vs. individual chunks. Don't add a prefetch per field reflexively; verify each candidate contributes content the other vectors don't. -When a representation's signal is mostly lexical — keyword-driven titles, codes, tags, or other short fields — prefer BM25 over an additional dense embedding. On Qdrant 1.20 or newer, create a BM25 full-text index (`scoring: true`) per payload field and run one `text` query prefetch per field, with no extra named vector. On older versions, use a sparse named vector with BM25. Server-side BM25 in Qdrant avoids the inference cost of another dense model and stores far less per point. Skip this when the field carries paraphrase or conceptual signal that exact-term matching would miss. +When a representation's signal is mostly lexical — keyword-driven titles, codes, tags, or other short fields — prefer BM25 over an additional dense embedding. On Qdrant 1.20 or newer, create a BM25 full-text index (with `scoring` enabled) per payload field and run one `text` query prefetch per field, with no extra named vector. On older versions, use a sparse named vector with BM25. Server-side BM25 in Qdrant avoids the inference cost of another dense model and stores far less per point. Skip this when the field carries paraphrase or conceptual signal that exact-term matching would miss. - End-to-end worked example fusing title, abstract, chunk, and sparse-title named vectors with RRF and document-level grouping in one Query API call: [Multi-Representation Search tutorial](https://skills.qdrant.tech/md/documentation/tutorials-search-engineering/multi-representation-search/) - If you have groups and subgroups of representations (document -> chunk, image -> patch), you could use [searching in groups](https://skills.qdrant.tech/md/documentation/search/search/?s=search-groups). To not store identical payloads several times, check [Lookup in Groups](https://skills.qdrant.tech/md/documentation/search/search/#lookup-in-groups). Index the grouping payload field (e.g. `document_id`) as a keyword payload index before grouping. @@ -81,6 +81,6 @@ There are ways to make multivector retrieval cheaper (MUVERA, pooling), you can ## What NOT to Do - Choose any search method (for example, BM25) without evaluation of its quality & resources used. - Use any search method (for example, BM25) without paying attention to the specifics of their configuration and applicability to the use case. -- On Qdrant 1.20 or newer, default to sparse vector BM25 with a guessed `avg_len` when the text is already in the payload; a full-text index with `scoring: true` computes it from the data. -- Run BM25 on a full-text index without configuring stemmer and stopwords for non-English data, or without `scoring: true` (a plain full-text index only filters). +- On Qdrant 1.20 or newer, default to sparse vector BM25 with a guessed `avg_len` when the text is already in the payload; a full-text index with `scoring` enabled computes it from the data. +- Run BM25 on a full-text index without configuring stemmer and stopwords for non-English data, or without `scoring` enabled (a plain full-text index only filters). diff --git a/skills/qdrant-sizing/SKILL.md b/skills/qdrant-sizing/SKILL.md index 974e4b3..bc60aa9 100644 --- a/skills/qdrant-sizing/SKILL.md +++ b/skills/qdrant-sizing/SKILL.md @@ -25,7 +25,7 @@ Use when: someone asks how much RAM or disk they need, how much data should be k - Size each component separately: dense vectors, quantized vectors, HNSW indexes, sparse vectors and their indexes, payload, payload indexes, and the ID tracker. Every component scales with `base = points × replication_factor`. - For multiple named vectors or payload fields per point, size each one separately, then sum them. - Count quantized vectors on top of the originals: Qdrant stores the compressed copy alongside the original vectors, not instead of them. A TurboQuant vector datatype is different: it replaces the originals, at 2 bytes per dimension for `turbo16` (1.20+), 1 for `turbo8` (1.20+), and 0.5 for `turbo4` (1.19+), against 4 for `float32`. Dense vectors can have 1 to 65,536 dimensions. -- A full-text index with `scoring: true` (BM25, 1.20+) stores document lengths and token positions, so it needs more RAM and disk than a filter-only full-text index. +- A full-text index with `scoring` enabled (BM25, 1.20+) stores document lengths and token positions, so it needs more RAM and disk than a filter-only full-text index. - A per-element payload index (the `nested` option for arrays of objects, 1.20+) also keeps an element-to-point map in RAM regardless of its memory tier: budget about 4 bytes per point for each such index. - Count payload indexes only for fields used for filtering; index only fields that are frequently filtered on. From 6fd0b430237afdc5456747ccdefa440c9a4b600a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Istv=C3=A1n=20Zolt=C3=A1n=20Szab=C3=B3?= Date: Thu, 22 Oct 2026 14:38:39 +0200 Subject: [PATCH 3/5] Fix broken link. --- .../memory-usage-optimization/SKILL.md | 2 +- skills/qdrant-sizing/SKILL.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/skills/qdrant-performance-optimization/memory-usage-optimization/SKILL.md b/skills/qdrant-performance-optimization/memory-usage-optimization/SKILL.md index bf03d39..f57d467 100644 --- a/skills/qdrant-performance-optimization/memory-usage-optimization/SKILL.md +++ b/skills/qdrant-performance-optimization/memory-usage-optimization/SKILL.md @@ -41,7 +41,7 @@ There are some scenarios, however, when it can be a good option: - Deployments with low latency disks - local NVMe or similar. - Multi-tenant deployments, where only a subset of tenants is frequently accessed, so that only a fraction of data & index is loaded in RAM at a time. -- For deployments with [inline storage](https://skills.qdrant.tech/md/documentation/ops-optimization/optimize/?s=inline-storage-in-hnsw-index) enabled. On Qdrant 1.20 or newer, inline storage keeps the original vectors only in the HNSW index file, so `hnsw_config.memory` decides their tier and `vectors.memory` is ignored for those segments. Quantized vectors follow `hnsw_config.memory` unless quantization has its own explicit `memory`, so set it explicitly (for example `pinned`) when the HNSW index is `cold` or `cached` [Where Qdrant stores the vectors](https://skills.qdrant.tech/md/documentation/ops-optimization/optimize/?s=where-qdrant-stores-the-vectors). +- For deployments with [inline storage](https://skills.qdrant.tech/md/documentation/ops-optimization/optimize/?s=inline-storage-in-hnsw-index) enabled. On Qdrant 1.20 or newer, inline storage keeps the original vectors only in the HNSW index file, so `hnsw_config.memory` decides their tier and `vectors.memory` is ignored for those segments. Unless quantization has its own explicit `memory`, quantized vectors are `cold` when the HNSW index is `cold` and `pinned` in any other tier, so set it explicitly (for example `pinned`) if the index is `cold` but you want the quantized vectors in RAM [Inline storage](https://skills.qdrant.tech/md/documentation/ops-configuration/memory-tiers/?s=inline-storage). ## How to minimize memory footprint diff --git a/skills/qdrant-sizing/SKILL.md b/skills/qdrant-sizing/SKILL.md index 974e4b3..bfc78b7 100644 --- a/skills/qdrant-sizing/SKILL.md +++ b/skills/qdrant-sizing/SKILL.md @@ -38,7 +38,7 @@ On Qdrant 1.19 or newer, set this per structure with `memory: pinned`, `cached`, - Keep HNSW, inverted indexes for sparse vectors, and payload indexes in RAM for faster search. - On Qdrant 1.20 or newer the ID tracker has its own memory tier (`pinned` by default, so it counts toward RAM). `cached` lets the OS evict it, and `cold` needs no up-front RAM. -- With inline storage on 1.20 or newer, the original vectors live only in the HNSW index file, and `hnsw_config.memory` sets their tier, not `vectors.memory`. Quantized vectors follow it too unless their `memory` is set explicitly. +- With inline storage on 1.20 or newer, the original vectors live only in the HNSW index file, and `hnsw_config.memory` sets their tier, not `vectors.memory`. Unless their `memory` is set explicitly, quantized vectors are `cold` when the index is `cold` and `pinned` otherwise [Inline storage](https://skills.qdrant.tech/md/documentation/ops-configuration/memory-tiers/?s=inline-storage). - Pin quantized vectors in RAM if they fit comfortably in the available memory, and move the original vectors to `cold` for rescoring. This is the main way quantization reduces RAM. - If your use case involves splitting vectors into multiple collections or subgroups based on payload values (for example, serving searches for multiple users, each with their own subset of vectors), store vectors in the `cold` memory tier and size RAM from the active subset of points, not the full collection [Subgroup-oriented configuration](https://skills.qdrant.tech/md/documentation/capacity-planning/?s=subgroup-oriented-configuration). - With vectors on disk, the amount of RAM drives latency: keeping half as many vectors in RAM roughly doubles search latency [Storage-focused configuration](https://skills.qdrant.tech/md/documentation/capacity-planning/?s=storage-focused-configuration). From c224164a9c5e6aaec3436381236f3564d422db13 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Istv=C3=A1n=20Zolt=C3=A1n=20Szab=C3=B3?= Date: Thu, 22 Oct 2026 14:42:50 +0200 Subject: [PATCH 4/5] Update skills/qdrant-performance-optimization/search-speed-optimization/SKILL.md Co-authored-by: Abdon Pijpelink --- .../search-speed-optimization/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/qdrant-performance-optimization/search-speed-optimization/SKILL.md b/skills/qdrant-performance-optimization/search-speed-optimization/SKILL.md index f528fa2..07ba8b0 100644 --- a/skills/qdrant-performance-optimization/search-speed-optimization/SKILL.md +++ b/skills/qdrant-performance-optimization/search-speed-optimization/SKILL.md @@ -49,7 +49,7 @@ Use when: filtered search is significantly slower than unfiltered. Most common S - Create payload index on the filtered field [Payload index](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=payload-index) - Use `is_tenant=true` for primary filtering condition: [Tenant index](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=tenant-index) - Try ACORN algorithm for complex filters: [ACORN](https://skills.qdrant.tech/md/documentation/search/search/?s=acorn-search-algorithm) -- Avoid using `nested` filtering conditions as a primary filter. It might force qdrant to read raw payload values instead of using index. On Qdrant 1.20 or newer, create per-element payload indexes (`nested` parameter set to the array key) on every field the nested filter uses; if all its conditions are match, range, geo, full-text, `is_empty`, or `is_null` on per-element-indexed fields of the same array, Qdrant answers it from the indexes alone. Note `is_empty` semantics differ on these indexes, and budget ~4 bytes per point per index in RAM [Per-element index](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=per-element-index-for-arrays-of-objects) [Nested object filter](https://skills.qdrant.tech/md/documentation/search/filtering/?s=nested-object-filter) +- `nested` filter conditions check each remaining candidate's payload element by element, which is slow when many candidates remain. On Qdrant 1.20 or newer, create per-element indexes (the `nested` parameter of the payload index) on every field the nested filter uses, so Qdrant answers it from the indexes without reading payload [Per-element index](https://skills.qdrant.tech/md/documentation/manage-data/indexing/?s=per-element-index-for-arrays-of-objects). - Substring match (`match` with `substring`, 1.20+) is expensive: it checks every distinct value of the field, and Qdrant rejects it whenever strict mode is enabled (the Qdrant Cloud default). Prefer prefix match (keyword index with `prefix: true`) or full-text match where the use case allows. A keyword index, especially with `prefix`, speeds it up; a full-text index doesn't help [Substring match](https://skills.qdrant.tech/md/documentation/search/filtering/?s=substring-match) - If payload index was added after HNSW build, trigger re-index to create filterable subgraph links From 88b77185a1d84763e76f0bb9e172ce154a757368 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Istv=C3=A1n=20Zolt=C3=A1n=20Szab=C3=B3?= Date: Thu, 22 Oct 2026 14:46:06 +0200 Subject: [PATCH 5/5] Apply suggestion from @szabosteve --- skills/qdrant-sizing/SKILL.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/skills/qdrant-sizing/SKILL.md b/skills/qdrant-sizing/SKILL.md index da82edc..f8300dc 100644 --- a/skills/qdrant-sizing/SKILL.md +++ b/skills/qdrant-sizing/SKILL.md @@ -37,8 +37,6 @@ On Qdrant 1.19 or newer, set this per structure with `memory: pinned`, `cached`, **Recommendations:** - Keep HNSW, inverted indexes for sparse vectors, and payload indexes in RAM for faster search. -- On Qdrant 1.20 or newer the ID tracker has its own memory tier (`pinned` by default, so it counts toward RAM). `cached` lets the OS evict it, and `cold` needs no up-front RAM. -- With inline storage on 1.20 or newer, the original vectors live only in the HNSW index file, and `hnsw_config.memory` sets their tier, not `vectors.memory`. Unless their `memory` is set explicitly, quantized vectors are `cold` when the index is `cold` and `pinned` otherwise [Inline storage](https://skills.qdrant.tech/md/documentation/ops-configuration/memory-tiers/?s=inline-storage). - Pin quantized vectors in RAM if they fit comfortably in the available memory, and move the original vectors to `cold` for rescoring. This is the main way quantization reduces RAM. - If your use case involves splitting vectors into multiple collections or subgroups based on payload values (for example, serving searches for multiple users, each with their own subset of vectors), store vectors in the `cold` memory tier and size RAM from the active subset of points, not the full collection [Subgroup-oriented configuration](https://skills.qdrant.tech/md/documentation/capacity-planning/?s=subgroup-oriented-configuration). - With vectors on disk, the amount of RAM drives latency: keeping half as many vectors in RAM roughly doubles search latency [Storage-focused configuration](https://skills.qdrant.tech/md/documentation/capacity-planning/?s=storage-focused-configuration).