Skip to content

Replace per-request settings DB polling with cache generation counter#10995

Open
pablomh wants to merge 1 commit into
theforeman:developfrom
pablomh:fix/settings-generation-counter
Open

Replace per-request settings DB polling with cache generation counter#10995
pablomh wants to merge 1 commit into
theforeman:developfrom
pablomh:fix/settings-generation-counter

Conversation

@pablomh
Copy link
Copy Markdown
Contributor

@pablomh pablomh commented May 22, 2026

Summary

Replace the per-request SELECT * FROM settings WHERE updated_at >= ... with a shared cache generation counter that skips the DB entirely when nothing has changed.

Problem

Every HTTP request triggers before_action :load_settings, which executes a query against the settings table to detect changes. During registration performance testing at 912+ concurrent registrations:

Metric Value
Sequential scans on settings 488K
Sequential scan rate 100%
Avg query time at 152 concurrency 0.9ms
Avg query time at 912 concurrency 20.6ms (23x inflation)
Total DB time in 30s window (912 conc) 17.6s

99.99% of these queries return zero rows — settings almost never change during normal operation.

Design

Two-tier cache with a safety net:

  • Same-process: Setting#after_save updates the in-memory SettingPresenter directly (existing behavior, unchanged)
  • Cross-process: Setting#after_commit bumps a shared Rails.cache generation counter. Other Puma workers check the counter before querying DB — if it matches their last-seen value, skip the query entirely.
  • Safety net: 30-second staleness fallback reloads from DB even if the generation appears current, guarding against lost cache writes during transient Redis outages.

The generation counter is the sole cross-process coherence signal — no split between generation and updated_at watermarks. On generation mismatch, all settings are reloaded (7 rows — negligible cost).

Why after_commit, not after_save

after_save fires before the transaction commits. Another worker could see the bumped generation, query the DB, get the old value (transaction not yet visible), cache it as seen, and serve stale settings indefinitely. after_commit ensures the generation only bumps after the new value is visible to all readers.

Cache backend compatibility

Works with both :redis_cache_store (atomic increment) and :file_store (non-atomic but safe — worst case is one extra DB poll, not stale data). Falls back to DB polling if cache is entirely unavailable.

Changes

  • app/models/setting.rb: Add after_commit :invalidate_settings_generation callback
  • app/registries/setting_registry.rb:
    • load_values: check generation counter before querying DB
    • reload_required?: extracted predicate (ignore_cache → nil check → staleness → generation)
    • increment_generation!: class method to bump the shared counter
    • stale?: 30-second fallback guard
    • Full reload on mismatch (no incremental updated_at filtering)

Test plan

  • Skips DB query when generation is current
  • Queries DB when generation changes
  • Queries DB when generation is nil (cache unavailable)
  • Bypasses generation check with ignore_cache: true
  • Increments generation on save
  • Increments generation on destroy
  • Staleness fallback reloads despite matching generation
  • Generation bumps after commit, not during save (regression test)
  • Deploy on test Satellite, verify settings queries drop from N×requests to ~0

Found via pg_stat_user_tables during registration performance testing at 1368 concurrent registrations with SQL debug logging enabled.

🤖 Generated with Claude Code

…eration counter

Every HTTP request triggered a SELECT on the settings table to detect
changes (before_action :load_settings). At 912+ concurrent registrations,
this generated 488K sequential scans on a 7-row table, with avg query
time inflating from 0.9ms to 20.6ms under contention (23x).

Replace the per-request DB poll with a Rails.cache generation counter:
- Setting#after_commit bumps a shared cache counter (after_commit, not
  after_save, to avoid a race where another process observes the bump
  before the transaction commits)
- SettingRegistry#load_values checks the counter before querying DB
- Generation mismatch triggers a full reload of all settings (single
  coherence mechanism — no split between generation and updated_at)
- 30-second staleness fallback: reload from DB even if generation
  appears current, to guard against lost cache writes during transient
  Redis outages
- Works with both Redis and file-based cache stores
- Falls back to DB polling if cache is unavailable

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@pablomh pablomh force-pushed the fix/settings-generation-counter branch from aa6fd4d to f9bc228 Compare May 22, 2026 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant