Fixes #39323 - Optimize grouped authorization filter handling

[pablomh]

Summary

• Group granular filters by effective taxonomy scope before building the scoped search expression — filters with identical (or reordered-equivalent) taxonomy predicates collapse into a single clause
• Derive filter granularity from the resource class once, instead of inspecting each filter individually — eliminates N+1 association lookups through Filter#granular? → resource_type → filterings.first.permission.resource_type
• Normalize taxonomy grouping keys by sorting IDs, so semantically equivalent scopes group even if assembled in different order
• Memoize the normalized grouping key per unique taxonomy_search value

Context

With many roles and filters (e.g., 400 roles × 33 filters = 13,200 filters), the authorization path was spending 5-7 seconds on per-filter Ruby-side work before even building the scoped search query. The repeated Filter#granular? calls triggered lazy association loading for each filter, and identical taxonomy clauses were duplicated across all filters instead of being grouped.

Community thread: https://community.theforeman.org/t/46388

Results

Test setup: 161 organizations, 1,814 locations, 400 roles with 33 filters each, non-admin user assigned to all roles.

Endpoint	Before	After
/api/hosts	6.7s	0.23s
/api/hosts?per_page	6.6s	0.22s
/api/job_templates	5.2s	0.36s
/api/job_templates?search	5.1s	0.25s

Authorization overhead dropped from 5-7 seconds to 57-75ms.

Test plan

• Existing authorizer tests pass
• New tests for grouped filter conditions, reordered taxonomy ID grouping
• Tested on foremanctl deployment with production-scale data

🤖 Generated with Claude Code

[adamruzicka]

Some at-a-glance styled notes, more coming later

[adamruzicka]

If condition is expected to be taxonomy_search that is generated by us, then it would be better to modify the code that generates it to make sure it will always be sorted.

[pablomh]

Agreed — addressed in c2e36f3. Added .sort in Filter#build_taxonomy_search_string_from_ids so new filters produce canonical taxonomy_search strings going forward.

For existing rows that may have unsorted IDs, I kept a lightweight normalize_taxonomy_search in the Authorizer grouping path so legacy data still collapses correctly without requiring a migration.

[adamruzicka]

I kept a lightweight normalize_taxonomy_search in the Authorizer grouping path so legacy data still collapses correctly without requiring a migration.

While this reads like the intent is for this to be temporary, things like that have a tendency to stick around. How about adding a migration to recalculate (or just reorder) all the taxonomy searches to put everyone on the happy path?

[pablomh]

Agreed — replaced the runtime normalization with a data migration in f7d9bdd.

The initial approach kept normalization at runtime because we wanted to avoid the risk of a migration touching every filters.taxonomy_search row in production — a bad regex could silently mangle search strings and broaden or restrict permissions. We opted for a read-only normalizer that could not corrupt data.

Now that the write path is canonical (.sort in Filter#build_taxonomy_search_string_from_ids), a one-time migration is the cleaner solution.

The migration (CanonicalizeFilterTaxonomySearch) sorts the IDs inside each persisted taxonomy_search value. It strictly matches only the three formats that Filter#build_taxonomy_search writes today:

• (organization_id ^ (...))
• (location_id ^ (...))
• ((organization_id ^ (...)) AND (location_id ^ (...)))

Rows with unrecognized formats are skipped and counted (logged via say), so any unexpected legacy data is preserved rather than silently rewritten. The migration uses a local model (MigrationFilter < ApplicationRecord) to avoid coupling to the app Filter class.

With that in place:

• normalize_taxonomy_search is removed from Authorizer
• Grouping is now a plain filters.group_by(&:taxonomy_search).values
• Added a test in filter_test.rb proving Filter writes sorted IDs on save

[adamruzicka]

Test setup: 161 organizations, 1,814 locations, 400 roles with 33 filters each, non-admin user assigned to all roles.

How are the organizations and locations assigned? I would expect this patch to bring biggest improvements for situations where many orgs/locs are assigned to the roles themselves, right? This wouldn't really address a case where a user has a single role with 1 org/loc but the user itself is in all the orgs/locs

Also, test failures are related

[pablomh]

In our reproduction, all 400 roles are assigned the same 36 organizations and 36 locations (out of 161/1,814 total). This is the best case for the grouping optimization — all 13,200 filters share the same taxonomy scope and collapse into a single group.

This wouldn't really address a case where a user has a single role with 1 org/loc but the user itself is in all the orgs/locs

Correct — the grouping optimization helps when multiple filters share the same taxonomy scope. A single role with unique taxonomy assignments wouldn't benefit from grouping.

However, the other two improvements in this PR still help regardless of taxonomy diversity:

1. granular_for_resource? class method — avoids N+1 association loading through Filter#granular? → filterings.first.permission.resource_type for every filter
2. Memoized taxonomy key normalization — reduces per-filter processing overhead even with diverse scopes

We could extend the test setup with varied per-role taxonomy assignments to measure the diverse case. Would that help?