You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
NeMo Retriever 26.5 service-mode ingestion returns abbreviated text values (…[N chars total]) in ServiceIngestResult.dataframe
#2749
NeMo Retriever 26.5 service-mode ingestion returns abbreviated text values (…[N chars total]) in ServiceIngestResult.dataframe
Description:
We are observing apparent text-content abbreviation when using NeMo Retriever 26.5 in service mode.
We reproduced the behavior in a standalone Python test that calls nemo_retriever.create_ingestor() directly, without any application-level normalization, serialization, cleaning, chunking, or RAG processing.
After calling ingestor.ingest(show_progress=True), the returned object is a nemo_retriever.service_ingestor.ServiceIngestResult, with results exposed through a Pandas DataFrame. The text column already contains genuine Python str values ending with patterns such as:
…[1263 chars total]
…[1479 chars total]
…[1543 chars total]
The actual returned string length in these cases is approximately 518–519 characters, while the suffix reports a substantially larger total character count.
We verified this directly on generated_metadata.dataframe before calling DataFrame.to_dict(), JSON serialization, or any downstream application processing.
The same abbreviated values remain after dataframe.to_dict(orient="records"), indicating that to_dict() is not introducing the abbreviation.
Our extraction configuration is:
create_ingestor(
run_mode="service",
base_url=BASE_URL,
)
.files(DOCUMENTS)
.pdf_split_config(
pages_per_chunk=32,
)
.extract(
extract_text=True,
extract_tables=True,
extract_charts=True,
extract_infographics=True,
paddle_output_format="markdown",
text_depth="page",
extract_method="pdfium_hybrid",
)
Captioning and image deduplication are also enabled in the tested pipeline.
Questions:
Is the …[N chars total] representation expected behavior for the text column of ServiceIngestResult.dataframe?
If this is an intentional preview/display representation, where can the complete extracted text be retrieved programmatically?
Is there a configuration option or API that prevents this abbreviation and returns the complete page-level text?
Does the abbreviation occur in the service response itself, or is it introduced by the nemo-retriever Python client while constructing ServiceIngestResult.dataframe?
Is there a known issue in NeMo Retriever 26.5 related to long text values in service-mode results?
Impact:
We use the extracted content for downstream RAG ingestion. An abbreviated text value cannot safely be indexed because content after the preview boundary is unavailable to the downstream pipeline. Filtering these records is also not acceptable because that would result in information loss.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
NeMo Retriever 26.5 service-mode ingestion returns abbreviated text values (…[N chars total]) in ServiceIngestResult.dataframe
Description:
We are observing apparent text-content abbreviation when using NeMo Retriever 26.5 in service mode.
We reproduced the behavior in a standalone Python test that calls nemo_retriever.create_ingestor() directly, without any application-level normalization, serialization, cleaning, chunking, or RAG processing.
After calling ingestor.ingest(show_progress=True), the returned object is a nemo_retriever.service_ingestor.ServiceIngestResult, with results exposed through a Pandas DataFrame. The text column already contains genuine Python str values ending with patterns such as:
…[1263 chars total]
…[1479 chars total]
…[1543 chars total]
The actual returned string length in these cases is approximately 518–519 characters, while the suffix reports a substantially larger total character count.
We verified this directly on generated_metadata.dataframe before calling DataFrame.to_dict(), JSON serialization, or any downstream application processing.
The same abbreviated values remain after dataframe.to_dict(orient="records"), indicating that to_dict() is not introducing the abbreviation.
Our extraction configuration is:
create_ingestor(
run_mode="service",
base_url=BASE_URL,
)
.files(DOCUMENTS)
.pdf_split_config(
pages_per_chunk=32,
)
.extract(
extract_text=True,
extract_tables=True,
extract_charts=True,
extract_infographics=True,
paddle_output_format="markdown",
text_depth="page",
extract_method="pdfium_hybrid",
)
Captioning and image deduplication are also enabled in the tested pipeline.
Questions:
Impact:
We use the extracted content for downstream RAG ingestion. An abbreviated text value cannot safely be indexed because content after the preview boundary is unavailable to the downstream pipeline. Filtering these records is also not acceptable because that would result in information loss.
All reactions