Skip to content

[Docs] Align global pipeline execution concepts with current backends #2161

Description

@lbliii

Parent tracker: #2118

Goal

Align the global pipeline, resource, task, and migration concepts with the current Ray Data, Xenna, and Ray Actor Pool APIs documented for 26.07.

Problems found

  • about/concepts/resource-allocation.mdx imports removed modules: nemo_curator.stages.core and nemo_curator.stages.function_definitions.
  • about/concepts/auto-balancing.mdx presents dynamic worker balancing as universal and says manual sizing is unnecessary, although behavior is backend-specific.
  • about/concepts/throughput.mdx uses the nonexistent RaySlurmClient name instead of SlurmRayClient and repeats universal autoscaling claims.
  • The Dask-to-Ray migration FAQ says pipelines cannot fan out and stages cannot return multiple tasks, while ProcessingStage supports list outputs.
  • The experimental-executor page links to /reference/infrastructure/execution-backends instead of /reference/infra/execution-backends.
  • Framework-owned task IDs, deterministic lineage, output cardinality, NoneTask, and FailedTask are buried under resumability even though they apply to every custom stage.

Timing

Correctness fixes are release scope and should land before the 26.07 docs publication tag. A new task-lifecycle concept page may land in the same pass or be split into a clearly tracked follow-up if schedule requires.

Acceptance criteria

  • Replace removed imports with current ProcessingStage, Resources, and decorator paths.
  • Scope auto-balancing claims by executor and link to stage worker sizing.
  • Replace RaySlurmClient with SlurmRayClient and verify all throughput examples.
  • Correct the migration FAQ's fan-out, output-cardinality, worker-sizing, and recovery statements.
  • Fix the execution-backends route.
  • Add a backend decision table connecting resource allocation, worker sizing, runtime environments, and executor selection.
  • Add or plan a universal task-lifecycle/lineage concept and keep API/reference pages concise through cross-links.
  • Validate every changed Python import against the source tree and run Fern validation.

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentationneeds-follow-upIssue needs follow-upr1.3.0Pick this label for auto cherry-picking into r1.3.0

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions