Add opt-in gathered-state filtering for loopback and local users by deekpand-cisco · Pull Request #391 · CiscoDevNet/ansible-nd

deekpand-cisco · 2026-07-07T16:27:16Z

Note

This is a stacked draft that depends on #312.

PR #312 is based on shrsr/ansible-nd:nd_links, so its source branch is not
available as a base branch in the official CiscoDevNet/ansible-nd
repository. This draft therefore temporarily targets develop.

Once #312 is merged, I will rebase this branch onto develop so the PR diff
contains only the gathered-filtering changes.

Related Issue(s)

Related to #340.

Depends on #312.

Proposed Changes

This PR introduces an opt-in gathered-state filtering framework for modules
using NDStateMachine and enables it for:

cisco.nd.nd_interface_loopback
cisco.nd.nd_local_user

The goal of this draft is to get early feedback on the shared filtering
architecture and module-specific behavior before enabling additional
NDStateMachine-backed modules.

Shared gathered-filtering framework

The shared implementation:

runs filtering only when state: gathered;
requires each model to opt in using supports_gathered_filtering;
treats config as partial filter criteria for supported gathered operations;
uses AND semantics between fields within one filter item;
uses OR semantics between separate filter items;
converts API responses into normalized Ansible configuration before final
matching;
allows models to normalize filter values through
normalize_gathered_filter;
allows models to customize matching through matches_gathered_filter;
removes duplicate matches using the model identifier;
keeps gathered filters out of proposed configuration;
prevents gathered filters from entering create, update, or delete diff
processing;
preserves existing behavior for non-gathered states;
preserves existing behavior for models that have not opted in.

The generic local filter remains the final correctness layer. Endpoint-specific
filtering is used only to reduce the number of API candidates where the
endpoint safely supports it.

Loopback-interface filtering

The loopback module supports gather-all and partial gathered filters across
switches in a fabric.

Gathered loopback queries use the interface-list endpoint and its Lucene
filter query parameter:

every query includes interfaceType:loopback and policyType:loopback;
switch_ip is resolved to a switch serial number and limits the query to
that switch;
omitting switch_ip queries all switches in the fabric;
interface_name is translated into an exact-match Lucene term;
separate filter items are queried independently to preserve OR semantics;
an unfiltered loopback query for a switch supersedes narrower interface-name
queries for that switch;
pagination uses max and offset until the API reports no remaining
results;
responses from multiple switches, filter items, and pages are combined and
deduplicated.

Lucene expressions are built only from structured module input. Raw Lucene
expressions are not accepted from users.

The module continues to enforce its resource-ownership boundary:

interfaceType must be loopback;
policyType must be loopback;
system-managed underlayLoopback interfaces are excluded;
non-standard ipfmLoopback and userDefined policies are excluded.

Nested criteria such as IP address, description, administrative state, VRF,
IPv6 address, extra configuration, and route-map tag are evaluated by the
generic final local filter.

Interface names supplied as filters are normalized to lowercase to match the
existing model and Nexus Dashboard API behavior.

Local-user filtering

The local-user endpoint does not expose documented server-side Lucene
filtering. Gather-all retrieves the available local users, and optional
login_id criteria are evaluated through the shared final local-matching
layer.

Only login_id is supported as a gathered-state filter for local users.
Other configuration fields are rejected with a validation error.

Multiple login-ID filter items use OR semantics. Login-ID matching is
case-sensitive.

Local-user passwords are not returned by Nexus Dashboard and remain excluded
from gathered output. Gathered output for an existing user can be reused as
configuration, but recreating a deleted user still requires a new password.

Argument validation and gathered output

For both modules:

config can be omitted for gather-all operations;
config remains required for write states through required_if;
the underlying Pydantic resource identifiers remain required for complete
resources;
gathered output is pruned to the module argument specification so it can be
reused as config;
gathered operations remain read-only and return changed: false.

For loopback gathered operations, switch_ip, interface_name, and nested
configuration fields can be supplied as partial filtering criteria.

For local-user gathered operations, each supplied filter item must contain
login_id. Other local-user fields are not supported as gathered filters.

Compatibility and rollout

Gathered filtering is disabled by default in the base model.
Existing modules are unaffected until they explicitly opt in.
Lucene filtering is disabled by default in the base orchestrator.
Existing orchestrators are unaffected until they explicitly opt in.
Custom modules using specialized runners are unaffected.
Non-gathered state behavior remains unchanged.
Gathered filters do not enter write-state processing.
Gathered operations do not modify Nexus Dashboard resources.

Test Notes

Completed validation:

Unit coverage includes Lucene expression construction and escaping, query
planning, pagination, deduplication, local matching, gathered/write-state
isolation, local-user login-ID filtering, validation, and password secrecy.

Integration coverage was added for gather-all, exact and multiple filters,
no-match behavior, invalid filters, pagination-related behavior, password
secrecy, and gathered-output reuse.

Cisco Nexus Dashboard Version

Cisco Nexus Dashboard 4.2 lab environment.

The exact patch/build version will be added before this draft is marked ready
for final review.

Related ND API Resource Category

The loopback-interface module uses the Manage API, while the local-user module
uses the Infra AAA API.

Checklist

Latest commit is rebased from develop with merge conflicts resolved
New or updated documentation has been added
Assigned the proper reviewers

The rebase/conflict item remains unchecked because this is currently a stacked
draft depending on #312. The branch will be rebased onto develop after #312
is merged.

Adds the nd_links module for ND 4.2 with manage / one_manage link scopes, template models, endpoints, orchestrator, strategies and fabric inventory support, plus integration tests.

…asking secrets

Add server-side Lucene candidate filtering for gathered loopback interfaces with pagination, deduplication, and final local matching. Support gather-all and login-ID filtering for local users while rejecting unsupported gathered criteria. Add unit and integration coverage for filtering, validation, secrecy, and reusable output.

sivakasi-cisco · 2026-07-20T13:30:56Z

+    Returned objects remain in API shape for NDConfigCollection.from_api_response().
+    """
+
+    if not filters:


Should we still deduplicate the results when no filters are provided? Right now this returns before the deduplication logic runs.

Removed the early return so deduplication now runs unconditionally. The to_config() function is only called when active filters exist to avoid unnecessary work.

sivakasi-cisco · 2026-07-20T13:31:38Z

+                query_kwargs["gathered_filters"] = raw_config
+
+            response_data = self.model_orchestrator.query_all(**query_kwargs)
+


Can we validate the gathered filters before calling query_all()? Otherwise, an invalid filter may make unnecessary API calls before it fails.

Added a validate_gathered_filters() function as a pre-flight check in the state machine, called before query_all(). Invalid filters now fail fast without making API calls.

sivakasi-cisco · 2026-07-20T13:40:28Z

+            meta = result.get("meta") or {}
+            counts = meta.get("counts") or {}
+            try:
+                remaining = int(counts.get("remaining", 0))


If remaining is missing or invalid, this treats it as zero and stops pagination. Could this return incomplete gathered results when the first page is full? Or, may be continue based on the page size or fail instead of returning partial data?

- Remove early return in filter_gathered_response() so deduplication runs unconditionally regardless of whether filters are provided - Add validate_gathered_filters() pre-flight check called before query_all() to reject invalid filters without wasted API calls

shrsr added 2 commits June 30, 2026 00:36

[minor_change] Add ND 4.2 Links module (nd_links)

f7c6641

Adds the nd_links module for ND 4.2 with manage / one_manage link scopes, template models, endpoints, orchestrator, strategies and fabric inventory support, plus integration tests.

[minor_change] Added gathered all to the framework and a method for m…

f4083fe

…asking secrets

deekpand-cisco changed the title ~~[minor_change] Add opt-in gathered-state filtering for loopback and local users~~ Add opt-in gathered-state filtering for loopback and local users Jul 7, 2026

allenrobel assigned deekpand-cisco Jul 7, 2026

deekpand-cisco force-pushed the feature/pr312-gathered-module-filtering branch from 7a31104 to 3a3e41b Compare July 9, 2026 08:38

sivakasi-cisco added the NaC-CapSet1 label Jul 14, 2026

sivakasi-cisco reviewed Jul 20, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add opt-in gathered-state filtering for loopback and local users#391

Add opt-in gathered-state filtering for loopback and local users#391
deekpand-cisco wants to merge 4 commits into
CiscoDevNet:developfrom
deekpand-cisco:feature/pr312-gathered-module-filtering

deekpand-cisco commented Jul 7, 2026 •

edited

Loading

Uh oh!

sivakasi-cisco Jul 20, 2026

Uh oh!

deekpand-cisco Jul 23, 2026

Uh oh!

sivakasi-cisco Jul 20, 2026

Uh oh!

deekpand-cisco Jul 23, 2026

Uh oh!

sivakasi-cisco Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		query_kwargs["gathered_filters"] = raw_config

		response_data = self.model_orchestrator.query_all(**query_kwargs)

Uh oh!

Conversation

deekpand-cisco commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issue(s)

Proposed Changes

Shared gathered-filtering framework

Loopback-interface filtering

Local-user filtering

Argument validation and gathered output

Compatibility and rollout

Test Notes

Cisco Nexus Dashboard Version

Related ND API Resource Category

Checklist

Uh oh!

sivakasi-cisco Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

deekpand-cisco Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

sivakasi-cisco Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

deekpand-cisco Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

sivakasi-cisco Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

deekpand-cisco commented Jul 7, 2026 •

edited

Loading