Add reusable, safe pagination for ND collection queries in the Orchestrators framework

[gmicol]

Summary

Add a reusable, opt-in offset-pagination component to the Orchestrators framework for Nexus Dashboard collection endpoints that expose max and offset query parameters.

This enhancement will be implemented only after all currently developed Routing Policies modules have been merged into develop.

Until then, Route Maps, Community Lists, and Extended Community Lists will use the existing ACL pagination approach within their respective feature pull requests. No shared pagination abstraction or broader framework refactor should be introduced into those pull requests.

Dependencies and implementation gate

The framework enhancement must not begin until the current Routing Policies module work is merged into develop.

Related pull requests:

• #266 — ACL module
• #286 — Prefix Lists
• #287 — Community Lists and Extended Community Lists
• #288 — Route Maps

The intended sequence is:

1. Complete the individual Routing Policies modules.
2. Use the ACL pagination method temporarily for Route Maps, Community Lists, and Extended Community Lists.
3. Merge all Routing Policies pull requests into develop.
4. Re-evaluate the pagination implementations from the common develop baseline.
5. Implement the shared pagination enhancement in a dedicated framework pull request.
6. Migrate the merged Routing Policies modules to the shared implementation.
7. Evaluate other existing collection modules for follow-up migration.

This sequencing avoids rebasing multiple feature branches around a new framework abstraction and keeps pagination refactoring separate from module delivery.

Background

NDStateMachine initializes its current-state collection using model_orchestrator.query_all(). Collection reconciliation is therefore correct only when query_all() retrieves the complete resource collection.

Several ND Manage APIs use the same pagination contract:

• max and offset query parameters.
• A resource-specific collection wrapper such as:
  • accessControlLists
  • ipv4PrefixLists
  • ipv6PrefixLists
  • communityLists
  • extendedCommunityLists
  • routeMaps
• Optional metadata containing:
  • counts.total
  • counts.remaining
  • links.next
  • links.previous

Pagination behavior is currently implemented independently by individual orchestrators.

The ACL and Prefix List implementations demonstrate different strengths and gaps, while other Routing Policies modules are adopting the ACL method temporarily so they can be completed and merged without waiting for a framework-level change.

The same problem also exists outside Routing Policies. Networks, VRFs, network attachments, VRF attachments, and future ND modules may need equivalent pagination behavior.

Temporary behavior before this enhancement

Before the shared enhancement is implemented:

• ACL will retain its existing pagination implementation.
• Prefix Lists will retain the pagination implementation delivered by its feature branch.
• Route Maps will adopt the ACL pagination method.
• Community Lists will adopt the ACL pagination method.
• Extended Community Lists will adopt the ACL pagination method.
• No shared pagination helper, mixin, or NDBaseOrchestrator change will be added to the feature pull requests.

The ACL method provides an acceptable temporary baseline because it:

• Sends explicit max and offset parameters.
• Stops on empty or short pages.
• Detects pages containing no new resource identities.
• Uses a maximum-page guard.
• Prevents a controller that ignores offset from causing an infinite loop.

This is intentionally a temporary solution. It does not fully use the ND metadata contract and should be replaced after the Routing Policies modules are merged.

Current architectural gaps

Duplicated pagination logic

Each module currently has to implement its own pagination loop, including:

• Page-size configuration.
• Offset advancement.
• Response-wrapper extraction.
• Metadata interpretation.
• Duplicate detection.
• Termination conditions.
• Maximum-page protection.
• Error handling.

This increases module-authoring effort and makes it likely that future modules will copy whichever implementation is closest rather than use one tested framework contract.

Different termination strategies

The ACL implementation primarily uses page length:

• Continue when a page is full.
• Stop when a page is empty or shorter than the requested page size.
• Stop when a page contains no new names.
• Stop after a configured maximum number of pages.

The Prefix List implementation uses API metadata:

• Read counts.total and counts.remaining.
• Advance by the number of rows actually returned.
• Fall back to page-size detection when metadata is absent.

Neither implementation is ideal by itself:

• ACL does not fully use the documented metadata.
• ACL advances by the requested page size rather than the number of received rows.
• ACL may make an additional empty request when the total is an exact multiple of the page size.
• Reaching the ACL page limit currently risks returning partial state unless handled explicitly.
• Prefix Lists do not currently have equivalent repeated-page and maximum-page safeguards.
• Prefix Lists could loop indefinitely if ND ignores offset or returns stale metadata.

Incomplete current-state discovery

If query_all() returns only the first page or terminates incorrectly, NDStateMachine may operate on an incomplete view of controller state.

Possible effects include:

• Existing resources being treated as absent.
• Attempts to recreate resources that already exist.
• Missed updates.
• Missed deletes.
• overridden state leaving undeclared resources behind.
• API conflicts.
• Incorrect idempotency results.
• Incomplete module return data.

Goals

The enhancement should provide:

• One reusable pagination implementation.
• Correct support for the ND max/offset contract.
• Metadata-aware termination.
• Safe fallback when metadata is absent.
• Advancement by the number of records actually received.
• Protection against repeated or non-progressing pages.
• A configurable maximum-page limit.
• Resource-specific identity handling.
• Explicit errors instead of silent partial success.
• A small and consistent API for future module authors.
• Compatibility with existing endpoint query-parameter representations.

Non-goals

This issue should not:

• Block or refactor the current Routing Policies feature pull requests.
• Modify the feature branches before they are merged.
• Refactor NDStateMachine.
• Require every orchestrator to support pagination.
• Cover cursor-based or token-based pagination.
• Migrate every existing module in the first implementation pull request.
• Change public Ansible module arguments or return structures.

Proposed design

Introduce an opt-in pagination component, potentially located at:

plugins/module_utils/orchestrators/pagination.py

The component could be implemented as:

• An NDOffsetPaginationMixin.
• A standalone OffsetPaginator.
• A small paginator plus an orchestrator-facing mixin.

The implementation should remain optional rather than becoming mandatory behavior in NDBaseOrchestrator.

A resource orchestrator should only need to provide:

• Its collection endpoint factory.
• Endpoint-specific configuration such as fabric_name and cluster_name.
• Its response collection key.
• A stable resource identity function.
• An optional item transformation function.

Resource-specific endpoint setup and response normalization should remain in the orchestrator. The shared component should own page sequencing, continuation decisions, progress detection, bounds, and errors.

Endpoint pagination contract

New collection endpoints should preferably expose generic pagination fields through LuceneQueryParams.

The shared paginator must also support existing endpoints that place max and offset directly on endpoint_params.

The implementation should:

• Prefer lucene_params.max and lucene_params.offset when available.
• Fall back to endpoint_params.max and endpoint_params.offset.
• Fail clearly when an endpoint declared as paginated does not support both parameters.
• Preserve endpoint-specific query parameters such as fabric, cluster, filter, and sort on every page.

This provides a migration path without requiring an immediate endpoint refactor.

Recommended pagination behavior

The shared paginator should:

1. Begin with offset=0.

2. Use a configurable page size.

3. Request each page through the orchestrator’s normal _request() method.

4. Validate the response shape.

5. Extract the configured collection wrapper.

6. Preserve controller result ordering.

7. Advance using the number of rows received:

   next_offset = offset + len(page_items)

8. Determine whether another page is required using:

   • counts.remaining;
   • counts.total;
   • links.next;
   • a full-page fallback when usable metadata is absent.

9. Avoid an unnecessary additional request when metadata confirms that the current page is the last page.

10. Detect repeated or non-progressing pages.

11. De-duplicate overlapping results using the complete resource identity.

12. Enforce a configurable maximum number of pages.

13. Raise an explicit pagination error if a complete collection cannot be obtained.

14. Never return silently truncated state after a pagination failure.

The metadata parser should support both meta and metadata, as both names are already used by collection responses handled in the repository.

Resource identity

The paginator must not assume that name alone is universally unique.

Each orchestrator should provide an identity appropriate for its resource.

Examples:

• ACL: (tenantName, name, type)
• Prefix List: (ipVersion, tenantName, name)
• Community List: (tenantName, name, type)
• Extended Community List: (tenantName, name, type)
• Route Map: (tenantName, name)

Overlapping pages should preserve first-seen order while suppressing already collected identities.

If metadata indicates that more records remain but a page yields no new identities, pagination should fail with a no-progress error rather than loop indefinitely or return partial data.

Error handling and observability

Introduce a specific pagination exception, such as NDPaginationError.

The error should include:

• Endpoint path.
• Current offset.
• Requested page size.
• Number of pages fetched.
• Number of unique records collected.
• Relevant response metadata.
• Failure reason.

Failure reasons may include:

• Invalid response shape.
• Invalid collection-wrapper shape.
• Unsupported endpoint pagination parameters.
• Repeated page.
• Empty page while metadata reports remaining records.
• No new resource identities.
• Maximum-page limit reached.

Every page request should continue to use _request() so existing API-call registration and Results observability are preserved.

Implementation plan

Phase 1: Audit the merged baseline

After all Routing Policies modules are merged:

• Compare the ACL and Prefix List implementations.
• Review the ACL-derived implementations in:
  • Route Maps.
  • Community Lists.
  • Extended Community Lists.
• Confirm each API’s:
  • collection wrapper;
  • pagination query parameters;
  • metadata shape;
  • tenant behavior;
  • resource identifier.
• Inventory other existing pagination implementations in:
  • Networks.
  • VRFs.
  • Network attachments.
  • VRF attachments.
  • Other Manage and OneManage collection endpoints.
• Finalize the shared paginator contract from the merged code rather than from feature branches.

Phase 2: Add the framework pagination component

• Add the reusable paginator or mixin.
• Add endpoint pagination configuration supporting both:
  • lucene_params;
  • legacy endpoint_params.
• Add metadata normalization.
• Add identity-aware de-duplication.
• Add no-progress detection.
• Add a maximum-page limit.
• Add explicit pagination errors.
• Keep NDBaseOrchestrator and NDStateMachine behavior unchanged unless a separate design review determines otherwise.

Phase 3: Migrate Routing Policies modules

Migrate the merged implementations together in a dedicated pull request:

• ACL.
• Prefix Lists.
• Community Lists.
• Extended Community Lists.
• Route Maps.

Each module should retain only its resource-specific behavior:

• Endpoint construction.
• Fabric and cluster configuration.
• Collection key.
• Identity extraction.
• Required item transformations.

The local ACL-derived loops and the Prefix List-specific loop should then be removed.

Phase 4: Validate and document the framework pattern

• Add shared paginator unit tests.
• Update module-specific pagination tests.
• Run the affected Routing Policies unit and integration targets.
• Run ansible-test sanity.
• Document the expected pagination pattern for future module authors.
• Add the pagination component to module planning and implementation guidance.

Phase 5: Possible follow-up migrations

After the Routing Policies migration is stable:

• Evaluate Networks and VRFs.
• Evaluate network and VRF attachment managers.
• Evaluate other collection endpoints.
• Create focused follow-up pull requests where the shared offset paginator is compatible.
• Keep cursor-based or otherwise incompatible APIs on resource-specific implementations.

Compatibility and migration notes

• No public Ansible arguments should change.
• Module return shapes should remain unchanged.
• query_all() should continue returning a list suitable for NDStateMachine.
• Existing page sizes should be retained initially unless testing shows a reason to change them.
• Page size and maximum pages should remain configurable by the orchestrator.
• Existing endpoint_params and lucene_params representations should both be supported.
• Scoped item lookups should remain separate from full collection pagination.
• A repeated page, no-progress condition, or page-limit exhaustion should become an explicit error.
• Returning an error instead of partial state is an intentional correctness improvement.
• Migration should occur only from the common post-merge develop baseline.

Test plan

Shared paginator tests

Cover:

• Empty collection.
• Single short page.
• Multiple pages using remaining.
• Multiple pages using total.
• Continuation using links.next.
• Missing metadata with page-length fallback.
• Exact page-size boundary.
• Controller returning fewer rows than requested while more remain.
• Offset advancement by received rows.
• Repeated full page.
• Partially overlapping pages.
• Empty page while metadata reports remaining records.
• String and integer metadata counts.
• Missing or malformed metadata.
• Invalid response shape.
• Invalid collection-wrapper shape.
• Maximum-page limit exhaustion.
• Complete resource identity handling.
• Same resource name in different tenants or address families.

Endpoint configuration tests

Cover:

• Pagination through LuceneQueryParams.
• Pagination through legacy endpoint parameters.
• Preservation of fabric and cluster parameters.
• Preservation of filter and sort parameters.
• Clear failure when pagination parameters are unsupported.

Routing Policies regression tests

Add or update tests for:

• ACL.
• IPv4 and IPv6 Prefix Lists.
• Community Lists.
• Extended Community Lists.
• Route Maps.
• Tenant-qualified identities.
• Correct wrapper extraction.
• Correct response transformations.
• Correct max and offset values on every page.
• Explicit failure for repeated or non-progressing pages.

Acceptance criteria

Prerequisites

• ACL is merged into develop.
• Prefix Lists are merged into develop.
• Community Lists and Extended Community Lists are merged into develop.
• Route Maps are merged into develop.
• Route Maps, Community Lists, and Extended Community Lists have temporary ACL-style pagination before merge.
• The framework implementation begins only after the above modules are available on develop.

Framework behavior

• A reusable, opt-in offset paginator exists.
• It supports both meta and metadata.
• It uses remaining, total, and links.next when available.
• It advances by the number of records received.
• It detects repeated and non-progressing pages.
• It supports complete resource-specific identities.
• It preserves first-seen ordering.
• It has a configurable maximum-page limit.
• It raises instead of returning silently partial state.
• It supports both lucene_params and legacy endpoint_params.
• It preserves endpoint-specific query parameters.

Migration

• ACL uses the shared implementation.
• Prefix Lists use the shared implementation.
• Community Lists use the shared implementation.
• Extended Community Lists use the shared implementation.
• Route Maps use the shared implementation.
• The temporary ACL-derived pagination loops are removed.
• The Prefix List-specific pagination loop is removed.
• Other existing collection modules are inventoried for follow-up migration.

Quality

• Shared paginator behavior has dedicated unit coverage.
• Routing Policies orchestrator tests cover pagination wiring and resource identities.
• Relevant unit and integration targets pass.
• ansible-test sanity passes.
• Framework guidance documents the pagination pattern for future modules.

Alternatives considered

Keep the ACL method in every orchestrator

This provides a consistent temporary implementation for the feature pull requests, but it continues to duplicate logic and does not fully use the ND metadata contract.

It is acceptable only as the pre-merge baseline.

Keep pagination permanently resource-specific

This avoids framework changes but requires every module author to reimplement termination, metadata handling, duplicate detection, bounds, and tests.

This is not recommended.

Add pagination directly to NDBaseOrchestrator

This makes pagination easy to discover but couples every orchestrator to offset-based collection semantics, including resources and actions that do not paginate this way.

I would not recommend it without a broader base-class architecture review.

Add an opt-in paginator or mixin after the modules merge

This provides one tested implementation while keeping resource-specific endpoint configuration, wrapper keys, transformations, and identities explicit.

It is the recommended approach.

Open questions

• Should the shared abstraction be a mixin, a standalone paginator, or a combination of both?
• What should the default page size and maximum-page count be?
• Should contradictory total and remaining metadata fail immediately or only when pagination stops making progress?
• Should overlapping duplicate identities retain the first or latest response representation?
• Should Networks, VRFs, and attachment pagination migrate with the initial framework pull request or through follow-up changes?

[mikewiebe]

nd_manage_links follow-up migration opt-in

Please include nd_manage_links from PR #312 as an explicit follow-up consumer
of the shared offset paginator introduced by this issue.

The module's current NDLinkOrchestrator.query_all() sends one collection GET
and discards pagination metadata. The Manage and OneManage link endpoints expose
max, offset, and collection metadata, so a fabric whose links span multiple
pages can give NDStateMachine an incomplete current-state view. That can cause
false creates, missed updates or deletes, incorrect overridden reconciliation,
incomplete gathers, and incorrect idempotency results.

Please add the following follow-up acceptance criteria after the shared
paginator is available:

• Migrate nd_manage_links Manage and OneManage collection reads to the shared
  paginator.
• Preserve fabric, cluster, filter, sort, and other scope parameters on every
  page request.
• De-duplicate using the module's complete physical-link identity rather than a
  link name or UUID assumption.
• Use response metadata when available and fail explicitly on repeated pages,
  no progress, or incomplete pagination.
• Add multi-page tests for gathered, merged, deleted, and overridden behavior.

This opt-in covers pagination completeness. It does not cover the separate
optimization of avoiding a full-fabric read when a task names only one or a few
links; that query-scoping concern remains recorded in the PR #312 performance
comment.