Parent tracker: #2118
Goal
Align the global pipeline, resource, task, and migration concepts with the current Ray Data, Xenna, and Ray Actor Pool APIs documented for 26.07.
Problems found
about/concepts/resource-allocation.mdx imports removed modules: nemo_curator.stages.core and nemo_curator.stages.function_definitions.
about/concepts/auto-balancing.mdx presents dynamic worker balancing as universal and says manual sizing is unnecessary, although behavior is backend-specific.
about/concepts/throughput.mdx uses the nonexistent RaySlurmClient name instead of SlurmRayClient and repeats universal autoscaling claims.
- The Dask-to-Ray migration FAQ says pipelines cannot fan out and stages cannot return multiple tasks, while
ProcessingStage supports list outputs.
- The experimental-executor page links to
/reference/infrastructure/execution-backends instead of /reference/infra/execution-backends.
- Framework-owned task IDs, deterministic lineage, output cardinality,
NoneTask, and FailedTask are buried under resumability even though they apply to every custom stage.
Timing
Correctness fixes are release scope and should land before the 26.07 docs publication tag. A new task-lifecycle concept page may land in the same pass or be split into a clearly tracked follow-up if schedule requires.
Acceptance criteria
Parent tracker: #2118
Goal
Align the global pipeline, resource, task, and migration concepts with the current Ray Data, Xenna, and Ray Actor Pool APIs documented for 26.07.
Problems found
about/concepts/resource-allocation.mdximports removed modules:nemo_curator.stages.coreandnemo_curator.stages.function_definitions.about/concepts/auto-balancing.mdxpresents dynamic worker balancing as universal and says manual sizing is unnecessary, although behavior is backend-specific.about/concepts/throughput.mdxuses the nonexistentRaySlurmClientname instead ofSlurmRayClientand repeats universal autoscaling claims.ProcessingStagesupports list outputs./reference/infrastructure/execution-backendsinstead of/reference/infra/execution-backends.NoneTask, andFailedTaskare buried under resumability even though they apply to every custom stage.Timing
Correctness fixes are release scope and should land before the 26.07 docs publication tag. A new task-lifecycle concept page may land in the same pass or be split into a clearly tracked follow-up if schedule requires.
Acceptance criteria
ProcessingStage,Resources, and decorator paths.RaySlurmClientwithSlurmRayClientand verify all throughput examples.