Most ecommerce_hybrid_search documents have empty or blank descriptions?

[jpountz]

After running the data preparation scripts on the first 10 categories (it would fail due to lack of RAM when concatenating all parquet files if I included all categories on my machine with 32GB of RAM, so I trimmed to the first 10 categories as suggested by the script), I got a dataset where 98.7% of documents have an empty or blank description.

This makes the benchmark less realistic/interesting in my mind, because titles alone tend to contain rather specific terms, that are matched by few queries, while descriptions tend to contain more common terms that are more likely to appear in queries. For instance, a query on "car" retrieves few documents despite the presence of an "Automotive" category.

I wonder if this also makes the comparison with Elasticsearch less fair due to the fact that Lucene's BM25 uses the number of docs with at least one term as the corpus size, while Vespa's BM25 seems to use the total number of docs in the index, so produced scores are not comparable?

[radu-gheorghe]

This could be made better, though I think the actual system test will not run out of RAM because it has a lot more of it.

We did measure the results overlap of the first results and it was roughly 0.8 for lexical search (depending on whether you have filters or not - see section 6.3.4 of the report). We thought this is enough to conclude that performance in these situations is comparable.

[jpountz]

I think it's fine to accept minor differences in tokenization, but if the two systems do not compute scores the same way then the comparison cannot be fair, even if they retrieve hits that are 80% similar (which is quite low by the way)?

[radu-gheorghe]

I was thinking the compute load is the same, even if we consider different numbers: docs with at least one term or total docs (thanks for bringing this difference up, I didn't notice it!).

Or are you saying that ES/Lucene does additional compute to fetch the number of docs with at least one term in the field? I thought these term statistics (docs with at least one term or total docs) take a similar amount of compute to fetch in Lucene. Do they not?

Either way, let's keep this open, I can check if on the system test machine we have descriptions for most documents - if not, maybe fixing that would bring that 80% value higher.

[jpountz]

I agree that the cost of computing scores is roughly the same. However the ability to prune hits may be completely different. Lucene's DisjunctionMaxQuery evaluates all clauses in parallel, and the best-case scenario is when one clause produces much higher scores than other clauses because these other clauses will prune hits based on scores produced by the higher-scoring clause. So the fact that Vespa and Lucene don't produce comparable scores for the same matches is a problem for the fairness of this comparison, even if fixing it may not change performance numbers much.

[radu-gheorghe]

Got it. Thanks for expanding the explanation 🙏