Skip to content

add: jina-embeddings-v5-omni results#513

Merged
Samoed merged 4 commits into
embeddings-benchmark:mainfrom
florian-hoenicke:add-jina-v5-omni-results
May 11, 2026
Merged

add: jina-embeddings-v5-omni results#513
Samoed merged 4 commits into
embeddings-benchmark:mainfrom
florian-hoenicke:add-jina-v5-omni-results

Conversation

@florian-hoenicke
Copy link
Copy Markdown
Contributor

@florian-hoenicke florian-hoenicke commented May 4, 2026 •
edited
Loading

Hi,

This PR adds MTEB results for jinaai/jina-embeddings-v5-omni-nano and jinaai/jina-embeddings-v5-omni-small. These text-benchmark JSONs reuse the corresponding v5 text results after parity verification of the omni text path.

Checklist

Note: after the vllm import repro, this PR now depends on the follow-up revision pin fix in embeddings-benchmark/mteb#4653.

Thanks!

@florian-hoenicke @cursoragent
add: jina-embeddings-v5-omni results
d9ebc7b 
Co-authored-by: Cursor <cursoragent@cursor.com>
@florian-hoenicke florian-hoenicke mentioned this pull request May 4, 2026
Merged
3 tasks
@KennethEnevoldsen KennethEnevoldsen added the waiting for review of implementation This PR is waiting for an implementation review before merging the results. label May 4, 2026
@KennethEnevoldsen
Copy link
Copy Markdown
Contributor

KennethEnevoldsen commented May 4, 2026

Great PR!

Note: the results CI depends on the model implementation PR being merged first.

Yes that is required, can I also ask you to add the checklist and fill it out - we especially need the last check-box.

@florian-hoenicke
Copy link
Copy Markdown
Contributor Author

florian-hoenicke commented May 5, 2026

Thanks, added the checklist to the PR body and filled the no-eval-training declaration. I left the public-availability box unchecked because the HF repos are intentionally still private right now.

@Samoed
Copy link
Copy Markdown
Member

Samoed commented May 5, 2026

When you will make your models public?

@Samoed
Copy link
Copy Markdown
Member

Samoed commented May 5, 2026 •
edited
Loading

@florian-hoenicke FYI we have benchmarks:

  • image (ViDoRe(v1&v2), ViDoRe(v3.1), MIEB(Multilingual))
  • audio (MAEB(beta))
  • video in progress, but we have a lot of tasks mteb.get_tasks(modalities=["video"], exclude_beta=False)

@florian-hoenicke @cursoragent
fix: pin jina omni results to whitespace parity revisions
48504fb 
Co-authored-by: Cursor <cursoragent@cursor.com>
@florian-hoenicke
Copy link
Copy Markdown
Contributor Author

florian-hoenicke commented May 6, 2026

Updated the submitted omni result folders to the fixed private HF revisions that preserve text whitespace: nano 36db6194..., small 8b4f2c44.... This keeps the result metadata aligned with the implementation PR after the reproducibility corner-case fix.

@KennethEnevoldsen
Copy link
Copy Markdown
Contributor

KennethEnevoldsen commented May 6, 2026

great to see @florian-hoenicke. We don't generally merge results before the model is public either as an API or repo, but I think we would be fine with doing it just before the planned release to allow you to use it in the release.

@florian-hoenicke @cursoragent
fix: pin jina omni results to latest private revisions
1636b61 
Co-authored-by: Cursor <cursoragent@cursor.com>
@florian-hoenicke
Copy link
Copy Markdown
Contributor Author

florian-hoenicke commented May 6, 2026

Updated the results folders to the latest private HF revisions after the new HF-side fix: nano 6f88a89e..., small 43affca6.... Cross-check passed: MTEB pin = results metadata = HF latest SHA, and both repos remain private.

@florian-hoenicke
Copy link
Copy Markdown
Contributor Author

florian-hoenicke commented May 9, 2026

Update: the two submitted HF model repos are now public and ungated:

  • jinaai/jina-embeddings-v5-omni-nano: Hub API reports private: false, gated: false, disabled: false
  • jinaai/jina-embeddings-v5-omni-small: Hub API reports private: false, gated: false, disabled: false

I updated the checklist accordingly. The remaining failing checks are still expected until the implementation PR lands first: embeddings-benchmark/mteb#4604. That PR is mergeable and I pushed a follow-up there (605cb143) to address the open review feedback.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 10, 2026

Model Results Comparison

Reference models: intfloat/multilingual-e5-large, google/gemini-embedding-001
New models evaluated: jinaai/jina-embeddings-v5-omni-nano, jinaai/jina-embeddings-v5-omni-small
Tasks: AILACasedocs, AILAStatutes, AfriSentiClassification, AlloProfClusteringS2S.v2, AlloprofReranking, AmazonCounterfactualClassification, AppsRetrieval, ArXivHierarchicalClusteringP2P, ArXivHierarchicalClusteringS2S, ArguAna, ArmenianParaphrasePC, AskUbuntuDupQuestions, BIOSSES, BUCC.v2, Banking77Classification, BelebeleRetrieval, BibleNLPBitextMining, BigPatentClustering.v2, BiorxivClusteringP2P.v2, BornholmBitextMining, BrazilianToxicTweetsClassification, BulgarianStoreReviewSentimentClassfication, CEDRClassification, CLSClusteringP2P.v2, CQADupstackAndroidRetrieval, CQADupstackEnglishRetrieval, CQADupstackGamingRetrieval, CQADupstackGisRetrieval, CQADupstackMathematicaRetrieval, CQADupstackPhysicsRetrieval, CQADupstackProgrammersRetrieval, CQADupstackStatsRetrieval, CQADupstackTexRetrieval, CQADupstackUnixRetrieval, CQADupstackWebmastersRetrieval, CQADupstackWordpressRetrieval, CSFDSKMovieReviewSentimentClassification, CTKFactsNLI, CUREv1, CataloniaTweetClassification, ChatDoctorRetrieval, ClimateFEVER, ClimateFEVERHardNegatives, Core17InstructionRetrieval, CovidRetrieval, CyrillicTurkicLangClassification, CzechProductReviewSentimentClassification, DBPedia, DBpediaClassification, DS1000Retrieval, DalajClassification, DiaBlaBitextMining, EstonianValenceClassification, FEVER, FEVERHardNegatives, FaroeseSTS, FiQA2018, FilipinoShopeeReviewsClassification, FinParaSTS, FinQARetrieval, FinanceBenchRetrieval, FinancialPhrasebankClassification, FloresBitextMining, FreshStackRetrieval, GermanSTSBenchmark, GreekLegalCodeClassification, GujaratiNewsClassification, HALClusteringS2S.v2, HC3FinanceRetrieval, HagridRetrieval, HotpotQA, HotpotQAHardNegatives, HumanEvalRetrieval, IN22GenBitextMining, ImdbClassification, IndicCrosslingualSTS, IndicGenBenchFloresBitextMining, IndicLangClassification, IndonesianIdClickbaitClassification, IsiZuluNewsClassification, ItaCaseholdClassification, JSICK, KorHateSpeechMLClassification, KorSarcasmClassification, KurdishSentimentClassification, LEMBNarrativeQARetrieval, LEMBNeedleRetrieval, LEMBPasskeyRetrieval, LEMBQMSumRetrieval, LEMBSummScreenFDRetrieval, LEMBWikimQARetrieval, LanguageClassification, LegalBenchCorporateLobbying, LegalQuAD, LegalSummarization, MBPPRetrieval, MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLRetrievalHardNegatives.v2, MKQARetrieval, MLQARetrieval, MLSUMClusteringP2P, MLSUMClusteringS2S, MSMARCO, MTOPDomainClassification, MTOPIntentClassification, MacedonianTweetSentimentClassification, MalteseNewsClassification, MasakhaNEWSClassification, MasakhaNEWSClusteringS2S, MassiveIntentClassification, MassiveScenarioClassification, MedrxivClusteringP2P.v2, MedrxivClusteringS2S.v2, MindSmallReranking, MintakaRetrieval, MrTidyRetrieval, MultiEURLEXMultilabelClassification, MultiHateClassification, MultiLongDocReranking, MultiLongDocRetrieval, MultilingualSentimentClassification, NFCorpus, NQ, NTREXBitextMining, NepaliNewsClassification, News21InstructionRetrieval, NollySentiBitextMining, NordicLangClassification, NorwegianCourtsBitextMining, NusaParagraphEmotionClassification, NusaTranslationBitextMining, NusaX-senti, NusaXBitextMining, OdiaNewsClassification, OpusparcusPC, PAC, PawsXPairClassification, PlscClusteringP2P.v2, PoemSentimentClassification, PolEmo2.0-OUT, PpcPC, PunjabiNewsClassification, QuoraRetrieval, RTE3, Robust04InstructionRetrieval, RomaniBibleClustering, RuBQReranking, SCIDOCS, SIB200Classification, SIB200ClusteringS2S, SICK-R, STS12, STS13, STS14, STS15, STS17, STS22, STS22.v2, STSB, STSBenchmark, STSBenchmarkMultilingualSTS, STSES, ScalaClassification, SciFact, SemRel24STS, SentimentAnalysisHindi, SinhalaNewsClassification, SiswatiNewsClassification, SlovakMovieReviewSentimentClassification, SpanishNewsClassification.v2, SpanishPassageRetrievalS2P, SpanishPassageRetrievalS2S, SpanishSentimentClassification.v2, SpartQA, SprintDuplicateQuestions, StackExchangeClustering.v2, StackExchangeClusteringP2P.v2, StackOverflowQA, StatcanDialogueDatasetRetrieval, SummEvalSummarization.v2, SwahiliNewsClassification, SwednClusteringP2P, SwissJudgementClassification, T2Reranking, TERRa, TRECCOVID, Tatoeba, TempReasonL1, Touche2020, Touche2020Retrieval.v3, ToxicConversationsClassification, TswanaNewsClassification, TweetSentimentExtractionClassification, TweetTopicSingleClassification, TwentyNewsgroupsClustering.v2, TwitterHjerneRetrieval, TwitterSemEval2015, TwitterURLCorpus, VoyageMMarcoReranking, WebFAQBitextMiningQAs, WebFAQBitextMiningQuestions, WebFAQRetrieval, WebLINXCandidatesReranking, WikiCitiesClustering, WikiClusteringP2P.v2, WikiSQLRetrieval, WikipediaRerankingMultilingual, WikipediaRetrievalMultilingual, WinoGrande, WisesightSentimentClassification.v2, WongnaiReviewsClassification, XNLI, XPQARetrieval, XQuADRetrieval, indonli

Results for jinaai/jina-embeddings-v5-omni-nano

task_name google/gemini-embedding-001 jinaai/jina-embeddings-v5-omni-nano intfloat/multilingual-e5-large Max result Model with max result In Training Data
AILACasedocs 0.4833 0.3968 0.2643 0.6560 Octen/Octen-Embedding-8B-INT8 False
AILAStatutes 0.4877 0.5152 0.2084 0.9451 Octen/Octen-Embedding-8B-INT8 False
AfriSentiClassification 0.5356 0.3684 0.455 0.5688 tencent/KaLM-Embedding-Gemma3-12B-2511 False
AlloProfClusteringS2S.v2 0.5636 0.5644 0.3328 0.6110 microsoft/harrier-oss-v1-27b False
AlloprofReranking 0.8177 0.7967 0.6944 0.8540 Octen/Octen-Embedding-8B False
AmazonCounterfactualClassification 0.8820 0.9372 0.6965 0.9696 GeoGPT-Research-Project/GeoEmbedding False
AppsRetrieval 0.9375 0.5839 0.3255 0.9862 google/gemini-embedding-2-preview False
ArXivHierarchicalClusteringP2P 0.6492 0.6502 0.5569 0.6869 NovaSearch/jasper_en_vision_language_v1 False
ArXivHierarchicalClusteringS2S 0.6384 0.6108 0.5367 0.6548 Qwen/Qwen3-Embedding-8B False
ArguAna 0.8644 0.6570 0.5436 0.8979 voyageai/voyage-3-m-exp False
ArmenianParaphrasePC 0.9689 0.9354 0.9493 0.9703 tencent/KaLM-Embedding-Gemma3-12B-2511 False
AskUbuntuDupQuestions 0.6424 0.6573 0.5924 0.7528 IEITYuan/Yuan-embedding-2.0-en False
BIOSSES 0.8897 0.8745 0.8457 0.9692 Gameselo/STS-multilingual-mpnet-base-v2 False
BUCC.v2 0.9899 0.9862 0.9878 0.9905 codefuse-ai/F2LLM-v2-8B False
Banking77Classification 0.9427 0.9020 0.7492 0.9427 google/gemini-embedding-001 False
BelebeleRetrieval 0.9073 0.7531 0.7791 0.9380 clips/e5-base-trm-nl False
BibleNLPBitextMining 0.2072 0.1250 0.1665 0.9899 deepvk/USER-bge-m3 False
BigPatentClustering.v2 0.3806 0.4352 0.3147 0.4578 BidirLM/BidirLM-0.6B-Embedding False
BiorxivClusteringP2P.v2 0.5386 0.4643 0.372 0.8417 codefuse-ai/F2LLM-4B False
BornholmBitextMining 0.5169 0.7712 0.4416 0.7798 jinaai/jina-embeddings-v5-text-small False
BrazilianToxicTweetsClassification 0.2802 0.2074 0.2123 0.3813 microsoft/harrier-oss-v1-27b False
BulgarianStoreReviewSentimentClassfication 0.7813 0.7401 0.6385 0.8159 microsoft/harrier-oss-v1-27b False
CEDRClassification 0.5742 0.6527 0.4484 0.7301 sergeyzh/BERTA False
CLSClusteringP2P.v2 0.4268 0.4889 0.4037 0.7572 Qwen/Qwen3-Embedding-8B False
CQADupstackAndroidRetrieval nan 0.5304 0.4904 0.7426 voyageai/voyage-3-m-exp False
CQADupstackEnglishRetrieval nan 0.5031 0.4581 0.6998 voyageai/voyage-3-m-exp False
CQADupstackGamingRetrieval 0.7068 0.6160 0.587 0.8161 IEITYuan/Yuan-embedding-2.0-en False
CQADupstackGisRetrieval nan 0.4289 0.3695 0.6340 voyageai/voyage-3-m-exp False
CQADupstackMathematicaRetrieval nan 0.3332 0.2818 0.6948 voyageai/voyage-3-m-exp False
CQADupstackPhysicsRetrieval nan 0.5016 0.4366 0.7371 voyageai/voyage-3-m-exp False
CQADupstackProgrammersRetrieval nan 0.4606 0.416 0.6587 voyageai/voyage-3-m-exp False
CQADupstackStatsRetrieval nan 0.3880 0.3238 0.6242 voyageai/voyage-3-m-exp False
CQADupstackTexRetrieval nan 0.3331 0.2836 0.6295 voyageai/voyage-3-m-exp False
CQADupstackUnixRetrieval 0.5369 0.4740 0.3988 0.7198 voyageai/voyage-3-m-exp False
CQADupstackWebmastersRetrieval nan 0.4407 0.3988 0.6835 voyageai/voyage-3-m-exp False
CQADupstackWordpressRetrieval nan 0.3484 0.3164 0.5862 voyageai/voyage-3-m-exp False
CSFDSKMovieReviewSentimentClassification 0.4938 0.4056 0.3484 0.6790 microsoft/harrier-oss-v1-27b False
CTKFactsNLI 0.8759 0.8161 0.7984 0.8993 omarelshehy/arabic-english-sts-matryoshka False
CUREv1 0.5957 0.5279 0.5162 0.6782 voyageai/voyage-4-large (embed_dim=2048) False
CataloniaTweetClassification 0.5451 0.6580 0.504 0.7983 microsoft/harrier-oss-v1-27b False
ChatDoctorRetrieval 0.7352 0.6939 0.5687 0.7722 voyageai/voyage-4-large (embed_dim=2048) False
ClimateFEVER nan 0.3960 0.2573 0.5693 voyageai/voyage-3-m-exp False
ClimateFEVERHardNegatives 0.3106 0.4003 0.26 0.5905 IEITYuan/Yuan-embedding-2.0-en False
Core17InstructionRetrieval 0.0769 0.0178 -0.0162 0.1461 nvidia/llama-embed-nemotron-8b False
CovidRetrieval 0.7913 0.7803 0.7561 0.9606 TencentBAC/Conan-embedding-v2 False
CyrillicTurkicLangClassification 0.9530 0.9791 0.4085 0.9944 microsoft/harrier-oss-v1-27b False
CzechProductReviewSentimentClassification 0.6816 0.6272 0.5714 0.7667 Bytedance/Seed1.6-embedding-1215 False
DBPedia nan 0.4526 0.413 0.5350 nvidia/NV-Embed-v2 False
DBpediaClassification 0.9476 0.9775 0.8828 0.9926 Qwen/Qwen3-Embedding-8B False
DS1000Retrieval 0.6870 0.5494 nan 0.7149 google/gemini-embedding-2-preview False
DalajClassification 0.5047 0.5043 0.5001 0.6586 microsoft/harrier-oss-v1-27b False
DiaBlaBitextMining 0.8723 0.8385 0.8483 0.8882 codefuse-ai/F2LLM-v2-14B False
EstonianValenceClassification 0.5352 0.5099 0.4289 0.6764 microsoft/harrier-oss-v1-27b False
FEVER nan 0.8951 0.8279 0.9628 voyageai/voyage-3-m-exp False
FEVERHardNegatives 0.8898 0.8982 0.8379 0.9453 ByteDance-Seed/Seed1.5-Embedding False
FaroeseSTS 0.8612 0.7140 0.7239 0.9739 Gameselo/STS-multilingual-mpnet-base-v2 False
FiQA2018 0.6178 0.4785 0.4381 0.8206 ai-sage/Giga-Embeddings-instruct False
FilipinoShopeeReviewsClassification 0.4845 0.4052 0.3527 0.5279 microsoft/harrier-oss-v1-27b False
FinParaSTS 0.2860 0.3505 0.2492 0.3505 jinaai/jina-embeddings-v5-text-nano False
FinQARetrieval 0.6464 0.5784 nan 0.8897 voyageai/voyage-4-large (embed_dim=2048) False
FinanceBenchRetrieval 0.9157 0.7882 nan 0.9459 Octen/Octen-Embedding-8B False
FinancialPhrasebankClassification 0.8864 0.4861 0.8394 0.9519 microsoft/harrier-oss-v1-0.6b False
FloresBitextMining 0.8371 0.5344 0.8108 0.9087 SamilPwC-AXNode-GenAI/PwC-Embedding_expr False
FreshStackRetrieval 0.3979 0.3831 0.2519 0.5776 Octen/Octen-Embedding-8B False
GermanSTSBenchmark 0.8809 0.9073 0.8408 0.9541 Gameselo/STS-multilingual-mpnet-base-v2 False
GreekLegalCodeClassification 0.4376 0.5896 0.3713 0.8052 Bytedance/Seed1.6-embedding-1215 False
GujaratiNewsClassification 0.9205 0.9101 0.7674 0.9343 Bytedance/Seed1.6-embedding-1215 False
HALClusteringS2S.v2 0.3200 0.2921 0.2261 0.3299 BidirLM/BidirLM-Omni-2.5B-Embedding False
HC3FinanceRetrieval 0.7758 0.5784 nan 0.8242 nvidia/NV-Embed-v2 False
HagridRetrieval 0.9931 0.9878 0.9891 0.9931 google/gemini-embedding-001 False
HotpotQA nan 0.6907 0.7122 0.8696 voyageai/voyage-3-m-exp False
HotpotQAHardNegatives 0.8701 0.6928 0.7055 0.8701 google/gemini-embedding-001 False
HumanEvalRetrieval 0.9910 0.9212 nan 1.0000 google/gemini-embedding-2-preview False
IN22GenBitextMining 0.9375 0.7170 0.7675 0.9375 google/gemini-embedding-001 False
ImdbClassification 0.9498 0.9474 0.8867 0.9737 Qwen/Qwen3-Embedding-8B False
IndicCrosslingualSTS 0.6287 0.4143 0.4387 0.8477 Gameselo/STS-multilingual-mpnet-base-v2 False
IndicGenBenchFloresBitextMining 0.9677 0.8624 0.8875 0.9881 Sailesh97/Hinvec False
IndicLangClassification 0.8769 0.9760 0.2025 0.9930 Bytedance/Seed1.6-embedding-1215 False
IndonesianIdClickbaitClassification 0.6700 0.5428 0.6122 0.7560 nvidia/llama-embed-nemotron-8b False
IsiZuluNewsClassification 0.4053 0.1775 0.3241 0.4257 microsoft/harrier-oss-v1-27b False
ItaCaseholdClassification 0.7330 0.8742 0.6679 0.9439 bigscience/sgpt-bloom-7b1-msmarco False
JSICK 0.8499 0.8152 0.7981 0.8963 Octen/Octen-Embedding-8B False
KorHateSpeechMLClassification 0.1769 0.5884 0.1049 0.7625 Bytedance/Seed1.6-embedding-1215 False
KorSarcasmClassification 0.6051 0.5370 0.5679 0.8190 ICT-TIME-and-Querit/BOOM_4B_v1 False
KurdishSentimentClassification 0.8639 0.9331 0.7708 0.9403 Bytedance/Seed1.6-embedding-1215 False
LEMBNarrativeQARetrieval nan 0.5217 0.2422 0.7690 lightonai/GTE-ModernColBERT-v1 False
LEMBNeedleRetrieval nan 0.5975 0.28 0.9325 lightonai/GTE-ModernColBERT-v1 False
LEMBPasskeyRetrieval 0.3850 0.8150 0.3825 1.0000 sentence-transformers/static-similarity-mrl-multilingual-v1 False
LEMBQMSumRetrieval nan 0.3183 0.2426 0.8323 mteb/baseline-bm25s False
LEMBSummScreenFDRetrieval nan 0.8180 0.7112 0.9784 mteb/baseline-bm25s False
LEMBWikimQARetrieval nan 0.7487 0.568 0.9988 lightonai/GTE-ModernColBERT-v1 False
LanguageClassification nan 0.6580 0.8761 0.9948 intfloat/multilingual-e5-large-instruct False
LegalBenchCorporateLobbying 0.9598 0.9452 0.8972 0.9696 voyageai/voyage-3-large False
LegalQuAD 0.6553 0.5777 0.4317 0.7675 mteb/baseline-bm25s False
LegalSummarization 0.7122 0.6577 0.621 0.7921 voyageai/voyage-3.5 False
MBPPRetrieval 0.9416 0.8670 nan 0.9608 voyageai/voyage-4-large (embed_dim=2048) False
MIRACLReranking 0.6409 0.6446 0.6935 0.7071 Cohere/Cohere-embed-multilingual-v3.0 False
MIRACLRetrieval nan 0.7707 nan 0.8214 BAAI/bge-m3 False
MIRACLRetrievalHardNegatives 0.7042 0.6580 0.5923 0.7305 nvidia/llama-embed-nemotron-8b False
MIRACLRetrievalHardNegatives.v2 0.5597 0.6665 0.5333 0.8003 Qwen/Qwen3-Embedding-4B False
MKQARetrieval nan 0.0994 nan 0.4634 codefuse-ai/F2LLM-v2-14B False
MLQARetrieval 0.8416 0.7703 0.7566 0.8416 google/gemini-embedding-001 False
MLSUMClusteringP2P 0.5465 0.4921 0.4631 0.7870 codefuse-ai/F2LLM-v2-14B False
MLSUMClusteringS2S 0.5377 0.4906 0.4681 0.7857 codefuse-ai/F2LLM-v2-14B False
MSMARCO nan 0.4164 0.437 0.4812 TencentBAC/Conan-embedding-v2 False
MTOPDomainClassification 0.9751 0.9725 0.9097 0.9995 voyageai/voyage-3-m-exp False
MTOPIntentClassification nan 0.9122 nan 0.9491 codefuse-ai/F2LLM-v2-14B False
MacedonianTweetSentimentClassification 0.7183 0.6491 0.6192 0.7547 Qwen/Qwen3-Embedding-4B False
MalteseNewsClassification 0.3738 0.5624 0.2395 0.6938 Bytedance/Seed1.6-embedding-1215 False
MasakhaNEWSClassification 0.8355 0.8272 0.7754 0.9009 Bytedance/Seed1.6-embedding-1215 False
MasakhaNEWSClusteringS2S 0.5745 0.5228 0.3804 0.7365 Bytedance/Seed1.6-embedding-1215 False
MassiveIntentClassification 0.8192 0.8406 0.6025 0.9194 voyageai/voyage-3-m-exp False
MassiveScenarioClassification 0.8868 0.9061 0.7003 0.9930 voyageai/voyage-3-m-exp False
MedrxivClusteringP2P.v2 0.4716 0.4256 0.3431 0.7199 codefuse-ai/F2LLM-4B False
MedrxivClusteringS2S.v2 0.4501 0.4033 0.3152 0.7023 codefuse-ai/F2LLM-4B False
MindSmallReranking 0.3295 0.3272 0.3024 0.3437 Kingsoft-LLM/QZhou-Embedding False
MintakaRetrieval 0.6179 0.4566 0.3423 0.6425 codefuse-ai/F2LLM-v2-14B False
MrTidyRetrieval nan 0.7183 nan 0.7977 BAAI/bge-m3 False
MultiEURLEXMultilabelClassification 0.0528 0.0545 0.0516 0.0968 Bytedance/Seed1.6-embedding-1215 False
MultiHateClassification 0.7247 0.5700 0.6357 0.8621 microsoft/harrier-oss-v1-27b False
MultiLongDocReranking nan 0.4352 nan 0.9243 codefuse-ai/F2LLM-v2-1.7B False
MultiLongDocRetrieval nan 0.2874 nan 0.3547 Alibaba-NLP/gte-multilingual-base False
MultilingualSentimentClassification nan 0.6522 nan 0.7793 intfloat/multilingual-e5-large-instruct False
NFCorpus nan 0.3869 0.3398 0.5575 TencentBAC/Conan-embedding-v2 False
NQ nan 0.6338 0.6403 0.8248 voyageai/voyage-3-m-exp False
NTREXBitextMining 0.9364 0.7282 0.914 0.9592 microsoft/harrier-oss-v1-27b False
NepaliNewsClassification 0.9814 0.9857 0.8847 0.9953 jinaai/jina-embeddings-v5-text-small False
News21InstructionRetrieval 0.1026 0.0128 -0.0006 0.1145 google/embeddinggemma-300m False
NollySentiBitextMining 0.6871 0.3710 0.675 0.8376 microsoft/harrier-oss-v1-27b False
NordicLangClassification 0.8597 0.8817 0.8015 0.9578 microsoft/harrier-oss-v1-27b False
NorwegianCourtsBitextMining 0.9342 0.9481 0.9404 0.9481 jinaai/jina-embeddings-v5-text-nano False
NusaParagraphEmotionClassification 0.5638 0.7337 0.4166 0.8374 Bytedance/Seed1.6-embedding-1215 False
NusaTranslationBitextMining 0.7752 0.6959 0.672 0.9222 Qwen/Qwen3-Embedding-8B False
NusaX-senti 0.8031 0.7611 0.7055 0.8482 Bytedance/Seed1.6-embedding-1215 False
NusaXBitextMining 0.8252 0.6581 0.7267 0.9056 Bytedance/Seed1.6-embedding-1215 False
OdiaNewsClassification 0.9184 0.9454 0.8001 0.9779 microsoft/harrier-oss-v1-27b False
OpusparcusPC 0.9662 0.9400 0.9451 0.9698 microsoft/harrier-oss-v1-27b False
PAC 0.7168 0.8630 0.7033 0.8811 Bytedance/Seed1.6-embedding-1215 False
PawsXPairClassification 0.5999 0.6223 0.5507 0.7557 Bytedance/Seed1.6-embedding-1215 False
PlscClusteringP2P.v2 0.7431 0.7437 0.7161 0.7542 tencent/KaLM-Embedding-Gemma3-12B-2511 False
PoemSentimentClassification 0.5966 0.7561 0.5067 0.8642 Bytedance/Seed1.6-embedding-1215 False
PolEmo2.0-OUT 0.7753 0.6690 0.3648 0.8063 microsoft/harrier-oss-v1-27b False
PpcPC 0.9550 0.9387 0.9116 0.9576 microsoft/harrier-oss-v1-27b False
PunjabiNewsClassification 0.8261 0.8045 0.807 0.8879 Bytedance/Seed1.6-embedding-1215 False
QuoraRetrieval nan 0.8887 0.8926 0.9235 TencentBAC/Conan-embedding-v2 False
RTE3 0.8955 0.8942 0.8752 0.9173 Bytedance/Seed1.6-embedding-1215 False
Robust04InstructionRetrieval -0.0241 -0.0292 -0.0748 0.1244 Qwen/Qwen3-Embedding-4B False
RomaniBibleClustering 0.4322 0.4060 0.4092 0.4658 microsoft/harrier-oss-v1-27b False
RuBQReranking 0.7384 0.7368 0.756 0.8051 ai-sage/Giga-Embeddings-instruct False
SCIDOCS 0.2515 0.2260 0.1745 0.5986 IEITYuan/Yuan-embedding-2.0-en False
SIB200Classification nan 0.4111 nan 0.9680 codefuse-ai/F2LLM-v2-8B False
SIB200ClusteringS2S 0.4174 0.3890 0.3945 0.7929 codefuse-ai/F2LLM-v2-14B False
SICK-R 0.8275 0.9197 0.8023 0.9465 Gameselo/STS-multilingual-mpnet-base-v2 False
STS12 0.8155 0.8534 0.8002 0.9546 Gameselo/STS-multilingual-mpnet-base-v2 False
STS13 0.8989 0.8948 0.8155 0.9776 Gameselo/STS-multilingual-mpnet-base-v2 False
STS14 0.8541 0.8891 0.7772 0.9753 Gameselo/STS-multilingual-mpnet-base-v2 False
STS15 0.9044 0.9280 0.8931 0.9811 Gameselo/STS-multilingual-mpnet-base-v2 False
STS17 0.8858 0.8629 0.8214 0.9342 infgrad/Jasper-Token-Compression-600M False
STS22 0.7176 0.6923 0.6823 0.7219 jinaai/jina-embeddings-v3 False
STS22.v2 0.7169 0.6962 0.643 0.7718 Kingsoft-LLM/QZhou-Embedding False
STSB 0.8550 0.8978 0.8236 0.9199 Gameselo/STS-multilingual-mpnet-base-v2 False
STSBenchmark 0.8908 0.9445 0.8729 0.9504 Kingsoft-LLM/QZhou-Embedding False
STSBenchmarkMultilingualSTS 0.8867 0.9144 0.8507 0.9589 Gameselo/STS-multilingual-mpnet-base-v2 False
STSES 0.8175 0.8007 0.8021 0.8231 google/embeddinggemma-300m False
ScalaClassification 0.5185 0.5011 0.5109 0.9112 microsoft/harrier-oss-v1-27b False
SciFact nan 0.7578 0.702 0.8660 openbmb/MiniCPM-Embedding False
SemRel24STS 0.7314 0.6187 0.6266 0.8112 VPLabs/SearchMap_Preview False
SentimentAnalysisHindi 0.7606 0.5535 0.642 0.8070 microsoft/harrier-oss-v1-27b False
SinhalaNewsClassification 0.8229 0.5169 0.6682 0.8591 microsoft/harrier-oss-v1-27b False
SiswatiNewsClassification 0.6238 0.4850 0.535 0.7837 Lajavaness/bilingual-embedding-small False
SlovakMovieReviewSentimentClassification 0.9035 0.8506 0.7441 0.9616 microsoft/harrier-oss-v1-27b False
SpanishNewsClassification.v2 0.9095 0.6769 0.8862 0.9290 codefuse-ai/F2LLM-v2-14B False
SpanishPassageRetrievalS2P 0.4887 0.5097 0.4196 0.5097 jinaai/jina-embeddings-v5-text-nano False
SpanishPassageRetrievalS2S 0.7715 0.7189 0.7232 0.7973 codefuse-ai/F2LLM-v2-8B False
SpanishSentimentClassification.v2 0.9664 0.8179 0.9241 0.9781 codefuse-ai/F2LLM-v2-8B False
SpartQA 0.1030 0.0628 0.0565 0.8769 microsoft/harrier-oss-v1-27b False
SprintDuplicateQuestions 0.9690 0.9555 0.9314 0.9838 Kingsoft-LLM/QZhou-Embedding False
StackExchangeClustering.v2 0.9207 0.6939 0.4643 0.9207 google/gemini-embedding-001 False
StackExchangeClusteringP2P.v2 0.5091 0.5058 0.3854 0.5510 Kingsoft-LLM/QZhou-Embedding False
StackOverflowQA 0.9671 0.9234 0.8889 0.9749 codefuse-ai/F2LLM-v2-14B False
StatcanDialogueDatasetRetrieval 0.5111 0.3305 0.1063 0.5889 ibm-granite/granite-embedding-311m-multilingual-r2 False
SummEvalSummarization.v2 0.3828 0.3195 0.3141 0.3893 annamodels/LGAI-Embedding-Preview False
SwahiliNewsClassification 0.6605 0.5074 0.5969 0.7066 codefuse-ai/F2LLM-v2-4B False
SwednClusteringP2P 0.4584 0.5603 0.3691 0.6213 Qwen/Qwen3-Embedding-4B False
SwissJudgementClassification 0.5786 0.6049 0.5362 0.7958 microsoft/harrier-oss-v1-27b False
T2Reranking 0.6795 0.6763 0.6632 0.7315 tencent/Youtu-Embedding False
TERRa 0.6392 0.6459 0.5842 0.7957 ai-sage/Giga-Embeddings-instruct False
TRECCOVID 0.8631 0.7760 0.7115 0.9833 IEITYuan/Yuan-embedding-2.0-en False
Tatoeba 0.8197 0.5654 0.7573 0.9659 SamilPwC-AXNode-GenAI/PwC-Embedding_expr False
TempReasonL1 0.0296 0.0124 0.0114 0.4184 microsoft/harrier-oss-v1-27b False
Touche2020 nan 0.3070 0.2313 0.3939 voyageai/voyage-3-m-exp False
Touche2020Retrieval.v3 0.5239 0.6612 0.4959 0.7465 Qwen/Qwen3-Embedding-4B False
ToxicConversationsClassification 0.8875 0.9277 0.6601 0.9759 voyageai/voyage-3-m-exp False
TswanaNewsClassification 0.5337 0.5265 0.47 0.6417 Bytedance/Seed1.6-embedding-1215 False
TweetSentimentExtractionClassification 0.6988 0.7150 0.628 0.8823 voyageai/voyage-3-m-exp False
TweetTopicSingleClassification 0.7111 0.8466 0.6532 0.8631 jinaai/jina-embeddings-v5-text-small False
TwentyNewsgroupsClustering.v2 0.5737 0.5256 0.3921 0.8758 GeoGPT-Research-Project/GeoEmbedding False
TwitterHjerneRetrieval 0.9802 0.7401 0.3522 0.9802 google/gemini-embedding-001 False
TwitterSemEval2015 0.7917 0.7310 0.7528 0.8946 voyageai/voyage-large-2-instruct False
TwitterURLCorpus 0.8705 0.8556 0.8583 0.9571 TencentBAC/Conan-embedding-v2 False
VoyageMMarcoReranking 0.6673 0.6702 0.6821 0.8366 codefuse-ai/F2LLM-v2-14B False
WebFAQBitextMiningQAs nan 0.9713 0.9826 0.9936 sentence-transformers/LaBSE False
WebFAQBitextMiningQuestions nan 0.9795 0.9572 0.9820 jinaai/jina-embeddings-v5-text-small False
WebFAQRetrieval nan 0.7554 0.7611 0.8552 codefuse-ai/F2LLM-v2-14B False
WebLINXCandidatesReranking 0.1097 0.1095 0.0778 0.2658 Querit/Querit False
WikiCitiesClustering 0.9163 0.8873 0.755 0.9500 microsoft/harrier-oss-v1-27b False
WikiClusteringP2P.v2 0.2823 0.3023 0.256 0.3319 microsoft/harrier-oss-v1-27b False
WikiSQLRetrieval 0.8814 0.9765 nan 0.9892 Octen/Octen-Embedding-8B False
WikipediaRerankingMultilingual 0.9224 0.8883 0.8981 0.9308 jinaai/jina-reranker-v3 False
WikipediaRetrievalMultilingual 0.9420 0.8995 0.9111 0.9420 google/gemini-embedding-001 False
WinoGrande 0.6052 0.5341 0.5498 0.9314 microsoft/harrier-oss-v1-27b False
WisesightSentimentClassification.v2 nan 0.2801 nan 0.4169 codefuse-ai/F2LLM-v2-14B False
WongnaiReviewsClassification nan 0.2886 nan 0.3695 google/embeddinggemma-300m False
XNLI 0.8526 0.8137 0.7477 0.9291 Bytedance/Seed1.6-embedding-1215 False
XPQARetrieval 0.6688 0.5716 0.5073 0.6856 codefuse-ai/F2LLM-v2-14B False
XQuADRetrieval nan 0.9486 0.9674 0.9709 telepix/PIXIE-Rune-v1.0 False
indonli 0.6069 0.5956 0.5174 0.6722 Bytedance/Seed1.6-embedding-1215 False
Average 0.6876 0.6352 0.5728 0.7919 nan -

Results for jinaai/jina-embeddings-v5-omni-small

task_name google/gemini-embedding-001 jinaai/jina-embeddings-v5-omni-small intfloat/multilingual-e5-large Max result Model with max result In Training Data
AILACasedocs 0.4833 0.4388 0.2643 0.6560 Octen/Octen-Embedding-8B-INT8 False
AILAStatutes 0.4877 0.5330 0.2084 0.9451 Octen/Octen-Embedding-8B-INT8 False
AfriSentiClassification 0.5356 0.3666 0.455 0.5688 tencent/KaLM-Embedding-Gemma3-12B-2511 False
AlloProfClusteringS2S.v2 0.5636 0.5303 0.3328 0.6110 microsoft/harrier-oss-v1-27b False
AlloprofReranking 0.8177 0.8139 0.6944 0.8540 Octen/Octen-Embedding-8B False
AmazonCounterfactualClassification 0.8820 0.9429 0.6965 0.9696 GeoGPT-Research-Project/GeoEmbedding False
AppsRetrieval 0.9375 0.7328 0.3255 0.9862 google/gemini-embedding-2-preview False
ArXivHierarchicalClusteringP2P 0.6492 0.6579 0.5569 0.6869 NovaSearch/jasper_en_vision_language_v1 False
ArXivHierarchicalClusteringS2S 0.6384 0.6271 0.5367 0.6548 Qwen/Qwen3-Embedding-8B False
ArguAna 0.8644 0.6507 0.5436 0.8979 voyageai/voyage-3-m-exp False
ArmenianParaphrasePC 0.9689 0.9446 0.9493 0.9703 tencent/KaLM-Embedding-Gemma3-12B-2511 False
AskUbuntuDupQuestions 0.6424 0.6608 0.5924 0.7528 IEITYuan/Yuan-embedding-2.0-en False
BIOSSES 0.8897 0.8516 0.8457 0.9692 Gameselo/STS-multilingual-mpnet-base-v2 False
BUCC.v2 0.9899 0.9867 0.9878 0.9905 codefuse-ai/F2LLM-v2-8B False
Banking77Classification 0.9427 0.9146 0.7492 0.9427 google/gemini-embedding-001 False

Note: Content truncated due to GitHub API limits. See the full report in the workflow artifacts.

@Samoed Samoed removed the waiting for review of implementation This PR is waiting for an implementation review before merging the results. label May 10, 2026
@Samoed
Copy link
Copy Markdown
Member

Samoed commented May 10, 2026

Results for jinaai/jina-embeddings-v5-omni-small

task_name google/gemini-embedding-001 jinaai/jina-embeddings-v5-omni-small intfloat/multilingual-e5-large Max result Model with max result In Training Data
AILACasedocs 0.4833 0.4388 0.2643 0.6560 Octen/Octen-Embedding-8B-INT8 False
AILAStatutes 0.4877 0.5330 0.2084 0.9451 Octen/Octen-Embedding-8B-INT8 False
AfriSentiClassification 0.5356 0.3666 0.455 0.5688 tencent/KaLM-Embedding-Gemma3-12B-2511 False
AlloProfClusteringS2S.v2 0.5636 0.5303 0.3328 0.6110 microsoft/harrier-oss-v1-27b False
AlloprofReranking 0.8177 0.8139 0.6944 0.8540 Octen/Octen-Embedding-8B False
AmazonCounterfactualClassification 0.8820 0.9429 0.6965 0.9696 GeoGPT-Research-Project/GeoEmbedding False
AppsRetrieval 0.9375 0.7328 0.3255 0.9862 google/gemini-embedding-2-preview False
ArXivHierarchicalClusteringP2P 0.6492 0.6579 0.5569 0.6869 NovaSearch/jasper_en_vision_language_v1 False
ArXivHierarchicalClusteringS2S 0.6384 0.6271 0.5367 0.6548 Qwen/Qwen3-Embedding-8B False
ArguAna 0.8644 0.6507 0.5436 0.8979 voyageai/voyage-3-m-exp False
ArmenianParaphrasePC 0.9689 0.9446 0.9493 0.9703 tencent/KaLM-Embedding-Gemma3-12B-2511 False
AskUbuntuDupQuestions 0.6424 0.6608 0.5924 0.7528 IEITYuan/Yuan-embedding-2.0-en False
BIOSSES 0.8897 0.8516 0.8457 0.9692 Gameselo/STS-multilingual-mpnet-base-v2 False
BUCC.v2 0.9899 0.9867 0.9878 0.9905 codefuse-ai/F2LLM-v2-8B False
Banking77Classification 0.9427 0.9146 0.7492 0.9427 google/gemini-embedding-001 False
BelebeleRetrieval 0.9073 0.7751 0.7791 0.9380 clips/e5-base-trm-nl False
BibleNLPBitextMining 0.2072 0.1356 0.1665 0.9899 deepvk/USER-bge-m3 False
BigPatentClustering.v2 0.3806 0.4259 0.3147 0.4578 BidirLM/BidirLM-0.6B-Embedding False
BiorxivClusteringP2P.v2 0.5386 0.4765 0.372 0.8417 codefuse-ai/F2LLM-4B False
BornholmBitextMining 0.5169 0.7798 0.4416 0.7798 jinaai/jina-embeddings-v5-text-small False
BrazilianToxicTweetsClassification 0.2802 0.2126 0.2123 0.3813 microsoft/harrier-oss-v1-27b False
BulgarianStoreReviewSentimentClassfication 0.7813 0.7484 0.6385 0.8159 microsoft/harrier-oss-v1-27b False
CEDRClassification 0.5742 0.6501 0.4484 0.7301 sergeyzh/BERTA False
CLSClusteringP2P.v2 0.4268 0.5071 0.4037 0.7572 Qwen/Qwen3-Embedding-8B False
CQADupstackAndroidRetrieval nan 0.5426 0.4904 0.7426 voyageai/voyage-3-m-exp False
CQADupstackEnglishRetrieval nan 0.5165 0.4581 0.6998 voyageai/voyage-3-m-exp False
CQADupstackGamingRetrieval 0.7068 0.6216 0.587 0.8161 IEITYuan/Yuan-embedding-2.0-en False
CQADupstackGisRetrieval nan 0.4459 0.3695 0.6340 voyageai/voyage-3-m-exp False
CQADupstackMathematicaRetrieval nan 0.3633 0.2818 0.6948 voyageai/voyage-3-m-exp False
CQADupstackPhysicsRetrieval nan 0.5341 0.4366 0.7371 voyageai/voyage-3-m-exp False
CQADupstackProgrammersRetrieval nan 0.4833 0.416 0.6587 voyageai/voyage-3-m-exp False
CQADupstackStatsRetrieval nan 0.4274 0.3238 0.6242 voyageai/voyage-3-m-exp False
CQADupstackTexRetrieval nan 0.3487 0.2836 0.6295 voyageai/voyage-3-m-exp False
CQADupstackUnixRetrieval 0.5369 0.4961 0.3988 0.7198 voyageai/voyage-3-m-exp False
CQADupstackWebmastersRetrieval nan 0.4600 0.3988 0.6835 voyageai/voyage-3-m-exp False
CQADupstackWordpressRetrieval nan 0.3671 0.3164 0.5862 voyageai/voyage-3-m-exp False
CSFDSKMovieReviewSentimentClassification 0.4938 0.4665 0.3484 0.6790 microsoft/harrier-oss-v1-27b False
CTKFactsNLI 0.8759 0.8269 0.7984 0.8993 omarelshehy/arabic-english-sts-matryoshka False
CUREv1 0.5957 0.5363 0.5162 0.6782 voyageai/voyage-4-large (embed_dim=2048) False
CataloniaTweetClassification 0.5451 0.6594 0.504 0.7983 microsoft/harrier-oss-v1-27b False
ChatDoctorRetrieval 0.7352 0.7106 0.5687 0.7722 voyageai/voyage-4-large (embed_dim=2048) False
ClimateFEVER nan 0.4150 0.2573 0.5693 voyageai/voyage-3-m-exp False
ClimateFEVERHardNegatives 0.3106 0.4175 0.26 0.5905 IEITYuan/Yuan-embedding-2.0-en False
Core17InstructionRetrieval 0.0769 0.0240 -0.0162 0.1461 nvidia/llama-embed-nemotron-8b False
CovidRetrieval 0.7913 0.8010 0.7561 0.9606 TencentBAC/Conan-embedding-v2 False
CyrillicTurkicLangClassification 0.9530 0.9760 0.4085 0.9944 microsoft/harrier-oss-v1-27b False
CzechProductReviewSentimentClassification 0.6816 0.6363 0.5714 0.7667 Bytedance/Seed1.6-embedding-1215 False
DBPedia nan 0.4438 0.413 0.5350 nvidia/NV-Embed-v2 False
DBpediaClassification 0.9476 0.9827 0.8828 0.9926 Qwen/Qwen3-Embedding-8B False
DS1000Retrieval 0.6870 0.6136 nan 0.7149 google/gemini-embedding-2-preview False
DalajClassification 0.5047 0.4988 0.5001 0.6586 microsoft/harrier-oss-v1-27b False
DiaBlaBitextMining 0.8723 0.8483 0.8483 0.8882 codefuse-ai/F2LLM-v2-14B False
EstonianValenceClassification 0.5352 0.5605 0.4289 0.6764 microsoft/harrier-oss-v1-27b False
FEVER nan 0.8999 0.8279 0.9628 voyageai/voyage-3-m-exp False
FEVERHardNegatives 0.8898 0.9046 0.8379 0.9453 ByteDance-Seed/Seed1.5-Embedding False
FaroeseSTS 0.8612 0.7690 0.7239 0.9739 Gameselo/STS-multilingual-mpnet-base-v2 False
FiQA2018 0.6178 0.4963 0.4381 0.8206 ai-sage/Giga-Embeddings-instruct False
FilipinoShopeeReviewsClassification 0.4845 0.4192 0.3527 0.5279 microsoft/harrier-oss-v1-27b False
FinParaSTS 0.2860 0.3306 0.2492 0.3505 jinaai/jina-embeddings-v5-text-nano False
FinQARetrieval 0.6464 0.5567 nan 0.8897 voyageai/voyage-4-large (embed_dim=2048) False
FinanceBenchRetrieval 0.9157 0.8055 nan 0.9459 Octen/Octen-Embedding-8B False
FinancialPhrasebankClassification 0.8864 0.6341 0.8394 0.9519 microsoft/harrier-oss-v1-0.6b False
FloresBitextMining 0.8371 0.5799 0.8108 0.9087 SamilPwC-AXNode-GenAI/PwC-Embedding_expr False
FreshStackRetrieval 0.3979 0.3922 0.2519 0.5776 Octen/Octen-Embedding-8B False
GermanSTSBenchmark 0.8809 0.9097 0.8408 0.9541 Gameselo/STS-multilingual-mpnet-base-v2 False
GreekLegalCodeClassification 0.4376 0.6496 0.3713 0.8052 Bytedance/Seed1.6-embedding-1215 False
GujaratiNewsClassification 0.9205 0.9297 0.7674 0.9343 Bytedance/Seed1.6-embedding-1215 False
HALClusteringS2S.v2 0.3200 0.3169 0.2261 0.3299 BidirLM/BidirLM-Omni-2.5B-Embedding False
HC3FinanceRetrieval 0.7758 0.6206 nan 0.8242 nvidia/NV-Embed-v2 False
HagridRetrieval 0.9931 0.9877 0.9891 0.9931 google/gemini-embedding-001 False
HotpotQA nan 0.6976 0.7122 0.8696 voyageai/voyage-3-m-exp False
HotpotQAHardNegatives 0.8701 0.6994 0.7055 0.8701 google/gemini-embedding-001 False
HumanEvalRetrieval 0.9910 0.9608 nan 1.0000 google/gemini-embedding-2-preview False
IN22GenBitextMining 0.9375 0.7486 0.7675 0.9375 google/gemini-embedding-001 False
ImdbClassification 0.9498 0.9555 0.8867 0.9737 Qwen/Qwen3-Embedding-8B False
IndicCrosslingualSTS 0.6287 0.4780 0.4387 0.8477 Gameselo/STS-multilingual-mpnet-base-v2 False
IndicGenBenchFloresBitextMining 0.9677 0.8847 0.8875 0.9881 Sailesh97/Hinvec False
IndicLangClassification 0.8769 0.9796 0.2025 0.9930 Bytedance/Seed1.6-embedding-1215 False
IndonesianIdClickbaitClassification 0.6700 0.5774 0.6122 0.7560 nvidia/llama-embed-nemotron-8b False
IsiZuluNewsClassification 0.4053 0.1991 0.3241 0.4257 microsoft/harrier-oss-v1-27b False
ItaCaseholdClassification 0.7330 0.9095 0.6679 0.9439 bigscience/sgpt-bloom-7b1-msmarco False
JSICK 0.8499 0.8131 0.7981 0.8963 Octen/Octen-Embedding-8B False
KorHateSpeechMLClassification 0.1769 0.6008 0.1049 0.7625 Bytedance/Seed1.6-embedding-1215 False
KorSarcasmClassification 0.6051 0.5523 0.5679 0.8190 ICT-TIME-and-Querit/BOOM_4B_v1 False
KurdishSentimentClassification 0.8639 0.9341 0.7708 0.9403 Bytedance/Seed1.6-embedding-1215 False
LEMBNarrativeQARetrieval nan 0.5295 0.2422 0.7690 lightonai/GTE-ModernColBERT-v1 False
LEMBNeedleRetrieval nan 0.4450 0.28 0.9325 lightonai/GTE-ModernColBERT-v1 False
LEMBPasskeyRetrieval 0.3850 0.8050 0.3825 1.0000 mteb/baseline-bm25s False
LEMBQMSumRetrieval nan 0.4380 0.2426 0.8323 mteb/baseline-bm25s False
LEMBSummScreenFDRetrieval nan 0.9688 0.7112 0.9784 mteb/baseline-bm25s False
LEMBWikimQARetrieval nan 0.7968 0.568 0.9988 lightonai/GTE-ModernColBERT-v1 False
LanguageClassification nan 0.6733 0.8761 0.9948 intfloat/multilingual-e5-large-instruct False
LegalBenchCorporateLobbying 0.9598 0.9400 0.8972 0.9696 voyageai/voyage-3-large False
LegalQuAD 0.6553 0.6361 0.4317 0.7675 mteb/baseline-bm25s False
LegalSummarization 0.7122 0.6494 0.621 0.7921 voyageai/voyage-3.5 False
MBPPRetrieval 0.9416 0.9054 nan 0.9608 voyageai/voyage-4-large (embed_dim=2048) False
MIRACLReranking 0.6409 0.6505 0.6935 0.7071 Cohere/Cohere-embed-multilingual-v3.0 False
MIRACLRetrieval nan 0.7827 nan 0.8214 BAAI/bge-m3 False
MIRACLRetrievalHardNegatives 0.7042 0.6659 0.5923 0.7305 nvidia/llama-embed-nemotron-8b False
MIRACLRetrievalHardNegatives.v2 0.5597 0.6760 0.5333 0.8003 Qwen/Qwen3-Embedding-4B False
MKQARetrieval nan 0.1014 nan 0.4634 codefuse-ai/F2LLM-v2-14B False
MLQARetrieval 0.8416 0.7759 0.7566 0.8416 google/gemini-embedding-001 False
MLSUMClusteringP2P 0.5465 0.4891 0.4631 0.7870 codefuse-ai/F2LLM-v2-14B False
MLSUMClusteringS2S 0.5377 0.4872 0.4681 0.7857 codefuse-ai/F2LLM-v2-14B False
MSMARCO nan 0.4209 0.437 0.4812 TencentBAC/Conan-embedding-v2 False
MTOPDomainClassification 0.9751 0.9769 0.9097 0.9995 voyageai/voyage-3-m-exp False
MTOPIntentClassification nan 0.9272 nan 0.9491 codefuse-ai/F2LLM-v2-14B False
MacedonianTweetSentimentClassification 0.7183 0.6767 0.6192 0.7547 Qwen/Qwen3-Embedding-4B False
MalteseNewsClassification 0.3738 0.5696 0.2395 0.6938 Bytedance/Seed1.6-embedding-1215 False
MasakhaNEWSClassification 0.8355 0.8687 0.7754 0.9009 Bytedance/Seed1.6-embedding-1215 False
MasakhaNEWSClusteringS2S 0.5745 0.5324 0.3804 0.7365 Bytedance/Seed1.6-embedding-1215 False
MassiveIntentClassification 0.8192 0.8532 0.6025 0.9194 voyageai/voyage-3-m-exp False
MassiveScenarioClassification 0.8868 0.9120 0.7003 0.9930 voyageai/voyage-3-m-exp False
MedrxivClusteringP2P.v2 0.4716 0.4351 0.3431 0.7199 codefuse-ai/F2LLM-4B False
MedrxivClusteringS2S.v2 0.4501 0.4143 0.3152 0.7023 codefuse-ai/F2LLM-4B False
MindSmallReranking 0.3295 0.3269 0.3024 0.3437 Kingsoft-LLM/QZhou-Embedding False
MintakaRetrieval 0.6179 0.4265 0.3423 0.6425 codefuse-ai/F2LLM-v2-14B False
MrTidyRetrieval nan 0.7458 nan 0.7977 BAAI/bge-m3 False
MultiEURLEXMultilabelClassification 0.0528 0.0656 0.0516 0.0968 Bytedance/Seed1.6-embedding-1215 False
MultiHateClassification 0.7247 0.5835 0.6357 0.8621 microsoft/harrier-oss-v1-27b False
MultiLongDocReranking nan 0.3885 nan 0.9243 codefuse-ai/F2LLM-v2-1.7B False
MultiLongDocRetrieval nan 0.2497 nan 0.3547 Alibaba-NLP/gte-multilingual-base False
MultilingualSentimentClassification nan 0.6744 nan 0.7793 intfloat/multilingual-e5-large-instruct False
NFCorpus nan 0.3981 0.3398 0.5575 TencentBAC/Conan-embedding-v2 False
NQ nan 0.6404 0.6403 0.8248 voyageai/voyage-3-m-exp False
NTREXBitextMining 0.9364 0.7655 0.914 0.9592 microsoft/harrier-oss-v1-27b False
NepaliNewsClassification 0.9814 0.9953 0.8847 0.9953 jinaai/jina-embeddings-v5-text-small False
News21InstructionRetrieval 0.1026 0.0324 -0.0006 0.1145 google/embeddinggemma-300m False
NollySentiBitextMining 0.6871 0.3882 0.675 0.8376 microsoft/harrier-oss-v1-27b False
NordicLangClassification 0.8597 0.8812 0.8015 0.9578 microsoft/harrier-oss-v1-27b False
NorwegianCourtsBitextMining 0.9342 0.9408 0.9404 0.9481 jinaai/jina-embeddings-v5-text-nano False
NusaParagraphEmotionClassification 0.5638 0.7584 0.4166 0.8374 Bytedance/Seed1.6-embedding-1215 False
NusaTranslationBitextMining 0.7752 0.7010 0.672 0.9222 Qwen/Qwen3-Embedding-8B False
NusaX-senti 0.8031 0.7973 0.7055 0.8482 Bytedance/Seed1.6-embedding-1215 False
NusaXBitextMining 0.8252 0.6922 0.7267 0.9056 Bytedance/Seed1.6-embedding-1215 False
OdiaNewsClassification 0.9184 0.9585 0.8001 0.9779 microsoft/harrier-oss-v1-27b False
OpusparcusPC 0.9662 0.9412 0.9451 0.9698 microsoft/harrier-oss-v1-27b False
PAC 0.7168 0.8699 0.7033 0.8811 Bytedance/Seed1.6-embedding-1215 False
PawsXPairClassification 0.5999 0.6565 0.5507 0.7557 Bytedance/Seed1.6-embedding-1215 False
PlscClusteringP2P.v2 0.7431 0.7442 0.7161 0.7542 tencent/KaLM-Embedding-Gemma3-12B-2511 False
PoemSentimentClassification 0.5966 0.8192 0.5067 0.8642 Bytedance/Seed1.6-embedding-1215 False
PolEmo2.0-OUT 0.7753 0.7387 0.3648 0.8063 microsoft/harrier-oss-v1-27b False
PpcPC 0.9550 0.9335 0.9116 0.9576 microsoft/harrier-oss-v1-27b False
PunjabiNewsClassification 0.8261 0.8860 0.807 0.8879 Bytedance/Seed1.6-embedding-1215 False
QuoraRetrieval nan 0.8909 0.8926 0.9235 TencentBAC/Conan-embedding-v2 False
RTE3 0.8955 0.8970 0.8752 0.9173 Bytedance/Seed1.6-embedding-1215 False
Robust04InstructionRetrieval -0.0241 -0.0161 -0.0748 0.1244 Qwen/Qwen3-Embedding-4B False
RomaniBibleClustering 0.4322 0.4039 0.4092 0.4658 microsoft/harrier-oss-v1-27b False
RuBQReranking 0.7384 0.7489 0.756 0.8051 ai-sage/Giga-Embeddings-instruct False
SCIDOCS 0.2515 0.2304 0.1745 0.5986 IEITYuan/Yuan-embedding-2.0-en False
SIB200Classification nan 0.5033 nan 0.9680 codefuse-ai/F2LLM-v2-8B False
SIB200ClusteringS2S 0.4174 0.4052 0.3945 0.7929 codefuse-ai/F2LLM-v2-14B False
SICK-R 0.8275 0.9162 0.8023 0.9465 Gameselo/STS-multilingual-mpnet-base-v2 False
STS12 0.8155 0.8511 0.8002 0.9546 Gameselo/STS-multilingual-mpnet-base-v2 False
STS13 0.8989 0.8870 0.8155 0.9776 Gameselo/STS-multilingual-mpnet-base-v2 False
STS14 0.8541 0.8850 0.7772 0.9753 Gameselo/STS-multilingual-mpnet-base-v2 False
STS15 0.9044 0.9258 0.8931 0.9811 Gameselo/STS-multilingual-mpnet-base-v2 False
STS17 0.8858 0.8718 0.8214 0.9342 infgrad/Jasper-Token-Compression-600M False
STS22 0.7176 0.7039 0.6823 0.7219 jinaai/jina-embeddings-v3 False
STS22.v2 0.7169 0.7097 0.643 0.7718 Kingsoft-LLM/QZhou-Embedding False
STSB 0.8550 0.8992 0.8236 0.9199 Gameselo/STS-multilingual-mpnet-base-v2 False
STSBenchmark 0.8908 0.9483 0.8729 0.9504 Kingsoft-LLM/QZhou-Embedding False
STSBenchmarkMultilingualSTS 0.8867 0.9187 0.8507 0.9589 Gameselo/STS-multilingual-mpnet-base-v2 False
STSES 0.8175 0.8123 0.8021 0.8231 google/embeddinggemma-300m False
ScalaClassification 0.5185 0.5023 0.5109 0.9112 microsoft/harrier-oss-v1-27b False
SciFact nan 0.7653 0.702 0.8660 openbmb/MiniCPM-Embedding False
SemRel24STS 0.7314 0.6093 0.6266 0.8112 VPLabs/SearchMap_Preview False
SentimentAnalysisHindi 0.7606 0.4429 0.642 0.8070 microsoft/harrier-oss-v1-27b False
SinhalaNewsClassification 0.8229 0.5252 0.6682 0.8591 microsoft/harrier-oss-v1-27b False
SiswatiNewsClassification 0.6238 0.4662 0.535 0.7837 Lajavaness/bilingual-embedding-small False
SlovakMovieReviewSentimentClassification 0.9035 0.8769 0.7441 0.9616 microsoft/harrier-oss-v1-27b False
SpanishNewsClassification.v2 0.9095 0.7347 0.8862 0.9290 codefuse-ai/F2LLM-v2-14B False
SpanishPassageRetrievalS2P 0.4887 0.4947 0.4196 0.5097 jinaai/jina-embeddings-v5-text-nano False
SpanishPassageRetrievalS2S 0.7715 0.7542 0.7232 0.7973 codefuse-ai/F2LLM-v2-8B False
SpanishSentimentClassification.v2 0.9664 0.8708 0.9241 0.9781 codefuse-ai/F2LLM-v2-8B False
SpartQA 0.1030 0.1145 0.0565 0.8769 microsoft/harrier-oss-v1-27b False
SprintDuplicateQuestions 0.9690 0.9656 0.9314 0.9838 Kingsoft-LLM/QZhou-Embedding False
StackExchangeClustering.v2 0.9207 0.7013 0.4643 0.9207 google/gemini-embedding-001 False
StackExchangeClusteringP2P.v2 0.5091 0.5195 0.3854 0.5510 Kingsoft-LLM/QZhou-Embedding False
StackOverflowQA 0.9671 0.9339 0.8889 0.9749 codefuse-ai/F2LLM-v2-14B False
StatcanDialogueDatasetRetrieval 0.5111 0.4626 0.1063 0.5889 ibm-granite/granite-embedding-311m-multilingual-r2 False
SummEvalSummarization.v2 0.3828 0.3177 0.3141 0.3893 annamodels/LGAI-Embedding-Preview False
SwahiliNewsClassification 0.6605 0.5461 0.5969 0.7066 codefuse-ai/F2LLM-v2-4B False
SwednClusteringP2P 0.4584 0.5760 0.3691 0.6213 Qwen/Qwen3-Embedding-4B False
SwissJudgementClassification 0.5786 0.6612 0.5362 0.7958 microsoft/harrier-oss-v1-27b False
T2Reranking 0.6795 0.6804 0.6632 0.7315 tencent/Youtu-Embedding False
TERRa 0.6392 0.6731 0.5842 0.7957 ai-sage/Giga-Embeddings-instruct False
TRECCOVID 0.8631 0.7849 0.7115 0.9833 IEITYuan/Yuan-embedding-2.0-en False
Tatoeba 0.8197 0.6113 0.7573 0.9659 SamilPwC-AXNode-GenAI/PwC-Embedding_expr False
TempReasonL1 0.0296 0.0093 0.0114 0.4184 microsoft/harrier-oss-v1-27b False
Touche2020 nan 0.2989 0.2313 0.3939 voyageai/voyage-3-m-exp False
Touche2020Retrieval.v3 0.5239 0.7059 0.4959 0.7465 Qwen/Qwen3-Embedding-4B False
ToxicConversationsClassification 0.8875 0.9367 0.6601 0.9759 voyageai/voyage-3-m-exp False
TswanaNewsClassification 0.5337 0.5378 0.47 0.6417 Bytedance/Seed1.6-embedding-1215 False
TweetSentimentExtractionClassification 0.6988 0.7207 0.628 0.8823 voyageai/voyage-3-m-exp False
TweetTopicSingleClassification 0.7111 0.8631 0.6532 0.8631 jinaai/jina-embeddings-v5-text-small False
TwentyNewsgroupsClustering.v2 0.5737 0.5434 0.3921 0.8758 GeoGPT-Research-Project/GeoEmbedding False
TwitterHjerneRetrieval 0.9802 0.7155 0.3522 0.9802 google/gemini-embedding-001 False
TwitterSemEval2015 0.7917 0.7231 0.7528 0.8946 voyageai/voyage-large-2-instruct False
TwitterURLCorpus 0.8705 0.8599 0.8583 0.9571 TencentBAC/Conan-embedding-v2 False
VoyageMMarcoReranking 0.6673 0.6884 0.6821 0.8366 codefuse-ai/F2LLM-v2-14B False
WebFAQBitextMiningQAs nan 0.9872 0.9826 0.9936 sentence-transformers/LaBSE False
WebFAQBitextMiningQuestions nan 0.9820 0.9572 0.9820 jinaai/jina-embeddings-v5-text-small False
WebFAQRetrieval nan 0.7629 0.7611 0.8552 codefuse-ai/F2LLM-v2-14B False
WebLINXCandidatesReranking 0.1097 0.1133 0.0778 0.2658 Querit/Querit False
WikiCitiesClustering 0.9163 0.8976 0.755 0.9500 microsoft/harrier-oss-v1-27b False
WikiClusteringP2P.v2 0.2823 0.3081 0.256 0.3319 microsoft/harrier-oss-v1-27b False
WikiSQLRetrieval 0.8814 0.9370 nan 0.9892 Octen/Octen-Embedding-8B False
WikipediaRerankingMultilingual 0.9224 0.8946 0.8981 0.9308 jinaai/jina-reranker-v3 False
WikipediaRetrievalMultilingual 0.9420 0.9058 0.9111 0.9420 google/gemini-embedding-001 False
WinoGrande 0.6052 0.5865 0.5498 0.9314 microsoft/harrier-oss-v1-27b False
WisesightSentimentClassification.v2 nan 0.3264 nan 0.4169 codefuse-ai/F2LLM-v2-14B False
WongnaiReviewsClassification nan 0.3331 nan 0.3695 google/embeddinggemma-300m False
XNLI 0.8526 0.8242 0.7477 0.9291 Bytedance/Seed1.6-embedding-1215 False
XPQARetrieval 0.6688 0.5917 0.5073 0.6856 codefuse-ai/F2LLM-v2-14B False
XQuADRetrieval nan 0.9459 0.9674 0.9709 telepix/PIXIE-Rune-v1.0 False
indonli 0.6069 0.5997 0.5174 0.6722 Bytedance/Seed1.6-embedding-1215 False
Average 0.6876 0.6505 0.5728 0.7919 nan -

@Samoed
Copy link
Copy Markdown
Member

Samoed commented May 10, 2026

@florian-hoenicke I tried to run your model, but got an error that vllm is required. Is this expected?

@Samoed Samoed mentioned this pull request May 10, 2026
Open
75 tasks
@florian-hoenicke
Copy link
Copy Markdown
Contributor Author

florian-hoenicke commented May 11, 2026

Update: the implementation PR has now merged: embeddings-benchmark/mteb#4604

The latest checks on this results PR are green (compare-results, test, and check-file-size). The older failed compare-results entry was from before the model registry landed in mteb/main.

Please let me know if anything else is needed before merging the results.

@florian-hoenicke
Copy link
Copy Markdown
Contributor Author

florian-hoenicke commented May 11, 2026

vllm should not be required for the normal MTEB / sentence-transformers / transformers loading path. It is only optional for vLLM serving.

I checked the current HF remote code for both submitted repos with transformers.dynamic_module_utils.check_imports:

  • jinaai/jina-embeddings-v5-omni-nano: modeling_jina_embeddings_v5_omni.py and custom_st.py do not expose vllm as a required import; check_imports passes.
  • jinaai/jina-embeddings-v5-omni-small: same.

The vLLM-specific files are present in the repo for serving support, but the model code guards vLLM registration behind importlib.util.find_spec("vllm") / optional imports. If you still hit a hard vllm requirement, could you share the exact command and traceback? I can patch it if there is another import path making it mandatory.

@Samoed
Copy link
Copy Markdown
Member

Samoed commented May 11, 2026 

mteb run -m jinaai/jina-embeddings-v5-omni-small -t TERRa HotpotQA                                                                                                                                                                                                              
[05/11/26 10:43:52] INFO     Running with parameters: Namespace(model='jinaai/jina-embeddings-v5-omni-small', task_types=None, categories=None, tasks=['TERRa', 'HotpotQA'], languages=None, benchmarks=None, device=None, output_folder='results', verbosity=2,   build_cli.py:50
                             co2_tracker=None, eval_splits=None, model_revision=None, batch_size=None, overwrite=False, overwrite_strategy='only-missing', save_predictions=False, prediction_folder=None, func=<function run at 0x35a09a140>)                                    
jinaai/jina-embeddings-v5-omni-small custom_st
jinaai/jina-embeddings-v5-omni-small modeling_jina_embeddings_v5_omni
[transformers] Encountered exception while importing vllm: No module named 'vllm'
Traceback (most recent call last):
  File "/Users/samoed/Desktop/mteb/.venv/bin/mteb", line 10, in <module>
    sys.exit(main())
             ~~~~^^
  File "/Users/samoed/Desktop/mteb/mteb/cli/build_cli.py", line 496, in main
    args.func(args)
    ~~~~~~~~~^^^^^^
  File "/Users/samoed/Desktop/mteb/mteb/cli/build_cli.py", line 57, in run
    model = mteb.get_model(args.model, args.model_revision, device=device)
  File "/Users/samoed/Desktop/mteb/mteb/models/get_model_meta.py", line 133, in get_model
    model = meta.load_model(device=device, embed_dim=embed_dim, **kwargs)
  File "/Users/samoed/Desktop/mteb/mteb/models/model_meta.py", line 406, in load_model
    model: MTEBModels = loader(
                        ~~~~~~^
        name,
        ^^^^^
        revision=revision,
        ^^^^^^^^^^^^^^^^^^
        **_kwargs,
        ^^^^^^^^^^
    )
    ^
  File "/Users/samoed/Desktop/mteb/mteb/models/sentence_transformer_wrapper.py", line 261, in __init__
    super().__init__(*args, **kwargs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/samoed/Desktop/mteb/mteb/models/sentence_transformer_wrapper.py", line 89, in __init__
    self.model = SentenceTransformer(
                 ~~~~~~~~~~~~~~~~~~~^
        model,
        ^^^^^^
    ...<3 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "/Users/samoed/Desktop/mteb/.venv/lib/python3.14/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
    return func(*args, **kwargs)
  File "/Users/samoed/Desktop/mteb/.venv/lib/python3.14/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
    super().__init__(
    ~~~~~~~~~~~~~~~~^
        model_name_or_path=model_name_or_path,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<13 lines>...
        default_prompt_name=default_prompt_name,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/samoed/Desktop/mteb/.venv/lib/python3.14/site-packages/sentence_transformers/base/model.py", line 198, in __init__
    modules, self.module_kwargs = self._load_modules(
                                  ~~~~~~~~~~~~~~~~~~^
        model_name_or_path,
        ^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
        config_kwargs=config_kwargs,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/samoed/Desktop/mteb/.venv/lib/python3.14/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
    return self._load_config_modules(model_name_or_path, **load_kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samoed/Desktop/mteb/.venv/lib/python3.14/site-packages/sentence_transformers/base/model.py", line 1146, in _load_config_modules
    module = module_class(model_name_or_path, **init_kwargs)
  File "/Users/samoed/.cache/huggingface/modules/transformers_modules/jinaai/jina_hyphen_embeddings_hyphen_v5_hyphen_omni_hyphen_small/43affca6bb810901935abc23dc01ae5d7e0e2ce6/custom_st.py", line 629, in __init__
    self.config = AutoConfig.from_pretrained(
                  ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        model_name_or_path, cache_dir=cache_dir, **config_kwargs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/samoed/Desktop/mteb/.venv/lib/python3.14/site-packages/transformers/models/auto/configuration_auto.py", line 391, in from_pretrained
    config_class = get_class_from_dynamic_module(
        class_ref, pretrained_model_name_or_path, code_revision=code_revision, **kwargs
    )
  File "/Users/samoed/Desktop/mteb/.venv/lib/python3.14/site-packages/transformers/dynamic_module_utils.py", line 617, in get_class_from_dynamic_module
    final_module = get_cached_module_file(
        repo_id,
    ...<7 lines>...
        repo_type=repo_type,
    )
  File "/Users/samoed/Desktop/mteb/.venv/lib/python3.14/site-packages/transformers/dynamic_module_utils.py", line 445, in get_cached_module_file
    modules_needed = check_imports(resolved_module_file)
  File "/Users/samoed/Desktop/mteb/.venv/lib/python3.14/site-packages/transformers/dynamic_module_utils.py", line 258, in check_imports
    raise ImportError(
    ...<2 lines>...
    )
ImportError: This modeling file requires the following packages that were not found in your environment: vllm. Run `pip install vllm`

@florian-hoenicke @cursoragent
fix: pin jina omni results to vllm-free revisions
0715220 
Align submitted result folders with HF revisions that load without requiring optional vllm dependencies.

Co-authored-by: Cursor <cursoragent@cursor.com>
@florian-hoenicke
Copy link
Copy Markdown
Contributor Author

florian-hoenicke commented May 11, 2026

Thanks, this is a real pinned-revision issue. The submitted small results were still pinned to 43affca6..., whose modeling_jina_embeddings_v5_omni.py is detected by Transformers check_imports as requiring vllm.

I pushed a fix here to move the result folders to the current public HF revisions:

  • small: dfdcc361ec47c69a5afcd81e4bd148abb9d0568e
  • nano: 2b230c93c996e091a45b95af4e3315dd07605ee3

I also opened the matching MTEB registry PR: embeddings-benchmark/mteb#4653

Verification:

  • check_imports passes for modeling_jina_embeddings_v5_omni.py and custom_st.py at both new revisions.
  • MTEB test_model_meta.py: 1958 passed.
  • Results test_model_implementation.py passes when pointed at the local MTEB revision bump.

@Samoed
Copy link
Copy Markdown
Member

Samoed commented May 11, 2026

I evaluated it on TERRa and can reproduce results

@Samoed Samoed merged commit 63d7bae into embeddings-benchmark:main May 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Samoed Samoed Samoed approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@florian-hoenicke @KennethEnevoldsen @Samoed