Skip to content

Fixes #39336 - Batch taxonomy subtree lookups to avoid N+1 queries#10990

Open
adamruzicka wants to merge 4 commits into
theforeman:developfrom
adamruzicka:batch-subtrees
Open

Fixes #39336 - Batch taxonomy subtree lookups to avoid N+1 queries#10990
adamruzicka wants to merge 4 commits into
theforeman:developfrom
adamruzicka:batch-subtrees

Conversation

@adamruzicka
Copy link
Copy Markdown
Contributor

Heavily inspired by https://community.theforeman.org/t/bug-orgs-and-locs-job-templates-are-broken-in-3-18-probably-3-17-as-well/46388/15

When a user belongs to many organizations, authorization performed 1+N SQL queries per taxonomy type to resolve subtree IDs (one subtree_ids pluck per organization). This patch replaces the per-taxonomy loop with a single batched query using OR conditions on the ancestry column.

Alternatives would be using recursive CTEs to let postgres walk the subtrees with a single query, but CTEs are still somewhat uncommon in rails lands.

The problem grows purely with the number of taxonomies assigned to the user. Here I have a user with 337 organizations and 1801 locations.

Before

# time curl --silent -q -u $username:$password -k https://$(hostname -f)/api/v2/job_templates | jq '.total'
79

real	0m12.970s
user	0m0.016s
sys	0m0.012s

After

# time curl --silent -q -u $username:$password -k https://$(hostname -f)/api/v2/job_templates | jq '.total'
79

real	0m4.527s
user	0m0.015s
sys	0m0.014s

When a user belongs to many organizations, authorization performed
1+N SQL queries per taxonomy type to resolve subtree IDs (one
subtree_ids pluck per organization). Replace the per-taxonomy loop
with a single batched query using OR conditions on the ancestry
column.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
adamruzicka and others added 2 commits May 20, 2026 13:19
Add ArgumentError guard and test to prevent silent data loss when
mixed Organization/Location objects are passed to batch_subtree_ids.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Avoid loading all taxonomy records into memory when finding
taxonomies that ignore 'user'. Use a LIKE pre-filter in SQL
to narrow the set, then verify with the exact ignore? check.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@adamruzicka
Copy link
Copy Markdown
Contributor Author

adamruzicka commented May 20, 2026

With the ignored taxonomies commit:

# time curl --silent -q -u $username:$password -k https://$(hostname -f)/api/v2/job_templates | jq '.total'
79

real    0m1.884s
user    0m0.013s
sys     0m0.012s

@adamruzicka
Copy link
Copy Markdown
Contributor Author

# time curl --silent -q -u $username:$password -k https://$(hostname -f)/api/v2/hosts | jq '.total'
1

real    0m25.790s
user    0m0.015s
sys     0m0.014s
# time curl --silent -q -u $username:$password -k https://$(hostname -f)/api/v2/hosts | jq '.total'
1

real    0m4.581s
user    0m0.016s
sys     0m0.011s

The single-query pluck(:id) returns IDs in arbitrary database
order, causing intermittent test failures in filter_test.rb
where expected strings embed literal ID sequences.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant