Can't seem to fetch more than 200 papers

[ozefreitas]

Description

Currently trying to make a benchmark of different document retrieval tools. The documents should be related to food sciences/properties/reformulation. When passing a query that finds more than 200 papers, the tool seems to freeze, either if no --limit is given or more than 200 is passed as an argument.
I cannot find anywhere in the docs stating that the tool may not be used without this parameter, neither a predefined limit is defined.

Steps to reproduce

• findpapers search /tmp/test_findpaper.json -q "[food] AND ([sciences] OR [industry] OR [flavoring])" --limit 20 -> Work fine, both search and refine work perfectly

• findpapers search /tmp/test_findpaper.json -q "[food] AND ([sciences] OR [industry] OR [flavoring])" --limit 220 -> Freeze

• findpapers search /tmp/test_findpaper.json -q "[food] AND ([sciences] OR [industry] OR [flavoring])" -> Freeze

In both freeze cases, the last verbose from the tool is: 2025-10-07 09:27:19,218 INFO: (200/492) Fetching arXiv paper: Daily Assistive Modular Robot Design Based on Multi-Objective Black-Box-Optimization.

This message makes me believe that something is wrong behind the scenes.

I would like to get some clearance, as this tool is pivotal in my research.

TY

[ozefreitas]

I am now trying different queries and parameters, and the same happens now at 600 papers for a query that finds 40000 papers.

I also noticed the clear batch method used (normally) limit to 200 papers per batch. However, when adding -d PubMed, the batch is gone, and now the tool fetches each file individually. When setting -d to arXiv, the batch comes back and explodes again...

So I think that something is currently wrong with the arXiv database fetcher