Skip to content

Fixes #39208 - Cache global registration script in smart-proxy#935

Open
pablomh wants to merge 1 commit intotheforeman:developfrom
pablomh:registration/cache-global-registration-script
Open

Fixes #39208 - Cache global registration script in smart-proxy#935
pablomh wants to merge 1 commit intotheforeman:developfrom
pablomh:registration/cache-global-registration-script

Conversation

@pablomh
Copy link
Copy Markdown

@pablomh pablomh commented Apr 1, 2026

Problem

GET /register returns the same shell script for all hosts sharing the same registration parameters (org, location, hostgroup, activation keys). Under bulk registration — 100+ hosts hitting the same capsule simultaneously — this endpoint is called once per host, each time proxying to Foreman and waiting for the ERB template to render (~103ms in profiling). With WEBrick's MaxClients=100, all threads can be held waiting for identical Foreman responses.

Solution

Cache the rendered script in memory (5-minute TTL) keyed on a canonical form of the request query string (parameters sorted alphabetically, so requests differing only in parameter order share the same cache entry).

Per-key double-checked locking prevents the thundering herd on a cold cache while allowing concurrent requests for genuinely different keys (e.g. different activation keys) to fetch from Foreman in parallel without blocking each other:

  • Fast path: unsynchronised Concurrent::Map read, no lock acquired
  • Slow path: key-specific Mutex serialises only competing threads; re-checks under lock before fetching from Foreman

The per-key Mutex is evicted from KEY_MUTEXES immediately after the script is written to cache, so KEY_MUTEXES is empty under steady state and bounded to in-flight fetches only.

Only 2xx responses are cached — errors never poison the cache. SCRIPT_CACHE uses Concurrent::Map (already a smart-proxy dependency) for lock-free reads that are safe on all Ruby VMs.

Test plan

  • Cache hit: Foreman called once for repeated identical requests
  • Per-key isolation: different parameter sets cached independently
  • Parameter order independence: requests differing only in param order share the same cache entry
  • TTL expiry: expired entries are not served; Foreman is re-called
  • Mutex eviction: KEY_MUTEXES is empty after a successful cache write
  • Error non-caching: Foreman is called on every request when it errors

Fixes #39208

@pablomh pablomh force-pushed the registration/cache-global-registration-script branch 3 times, most recently from 83ee924 to bb29755 Compare April 1, 2026 23:34
Copy link
Copy Markdown
Member

@ekohl ekohl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Ruby you can pass blocks and that creates a nice abstraction mechanism. In somewhat pseudo-code:

CACHE = {}
MUTEX = Mutex.new

def cache(key, &block)
  if (value = CACHE[key])
    value
  else
    MUTEX.synchronize do
      CACHE[key] = block.call
    end
  end
end 

def get_value(query)
  cache(query) do
    real_call_here(query)
  end
end

You may need to do something like raising an exception to handle errors from Foreman code (in handle_response) so it doesn't get cached.


response = Proxy::Registration::ProxyRequest.new.global_register(request)

if response.code.start_with?('2')
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is HTTP 201 or 202 I wouldn't expect it to be cached. Only HTTP 200.

@pablomh pablomh force-pushed the registration/cache-global-registration-script branch from bb29755 to 527a20b Compare April 2, 2026 10:24
GET /register returns the same shell script for all hosts sharing the
same registration parameters (org, location, hostgroup, activation keys).
Under bulk registration — 100+ hosts hitting the same capsule simultaneously
— this endpoint is called once per host, each time proxying to Foreman and
waiting for the ERB template to render (~103ms in profiling).

Cache the rendered script in memory (5-minute TTL) keyed on a canonical
form of the request query string. The cache key is computed by parsing the
query string, sorting parameters alphabetically, and rebuilding — so
requests that differ only in parameter order share the same cache entry
and the same Foreman response, regardless of how the client ordered them.

The implementation uses three clearly separated layers:

  get '/'             — handles errors only; delegates to registration_script
  registration_script — owns the cache key and the business logic (what
                        to cache and what to do on a miss); raises
                        ScriptFetchError on non-200 so errors never reach
                        the cache write
  cache(key, &block)  — owns the mechanism: per-key double-checked locking
                        via a block abstraction that keeps the caller free
                        of locking concerns

Per-key locking allows concurrent requests for genuinely different keys
(e.g. different activation keys) to fetch from Foreman in parallel, while
serialising only threads competing for the same key. The per-key Mutex is
evicted from KEY_MUTEXES immediately after caching — once a key is hot,
all future requests take the lock-free fast path and KEY_MUTEXES is empty
under steady state.

Only HTTP 200 responses are cached. Non-200 responses raise ScriptFetchError
out of the cache block, which is rescued in get '/' and rendered via
handle_response without poisoning the cache.

Both KEY_MUTEXES and SCRIPT_CACHE use Concurrent::Map (already a
smart-proxy dependency) for lock-free, thread-safe access on all Ruby
VMs without relying on MRI's GIL.

Tests added:
- Cache hit: Foreman called once for repeated identical requests
- Per-key isolation: different parameter sets cached independently
- Parameter order independence: requests differing only in param order
  share the same cache entry
- TTL expiry: expired entries are not served; Foreman is re-called
- Mutex eviction: KEY_MUTEXES is empty after a successful cache write
- Error non-caching: Foreman is called on every request when it errors
- setup clears both SCRIPT_CACHE and KEY_MUTEXES between tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants