Repository navigation
dynamic_modules: add bootstrap secret lifecycle callbacks - #47437
basundhara-c wants to merge 6 commits into
Conversation
453417d to
d480814
Compare
|
/restest |
|
/retest |
df39d2a to
111ea52
Compare
5c61519 to
c2bbc24
Compare
|
Needs a main merge. /wait |
a86ecca to
3ff772e
Compare
|
/retest |
|
PTAL when you have time @agrawroh |
Signed-off-by: Basundhara Chakrabarty <basundhara17061996@gmail.com>
3ff772e to
1f655e5
Compare
Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Basundhara Chakrabarty <basundhara17061996@gmail.com>
|
/retest |
There was a problem hiding this comment.
The secret lifecycle looks like the right thing to add but the core piece is a single replaceable callback that never reports a provider leaving the manager, so removal means the wrong thing and a second extension silently starves the first, and the cluster-hook marshal fixes a worker-thread crash that the per-thread callback registration cannot produce.
I'd like the manager-owned observer and the marshal settled before we go further. The module side otherwise looks fine.
/wait
| * callback is held; a later call replaces the earlier one. Default is a no-op so only | ||
| * implementations that support secret lifecycle observation need override it. | ||
| */ | ||
| virtual void setDynamicTlsCertificateSecretProviderCreatedCallback( |
There was a problem hiding this comment.
This holds one std::function and a later call replaces it, so with two bootstrap extensions the first keeps the subscriptions it already made and never hears about a provider created afterwards, and since removeDynamicSecretProvider emits nothing the observer also cannot learn that a provider died.
The manager is the one place that sees both ends, so I'd rather see the sibling shape here, an addSecretLifecycleCallbacks returning a handle, PURE like dynamicActiveSecretNames above, with the manager owning one update subscription per provider and emitting add-or-update from it and removal from removeDynamicSecretProvider, for all four provider kinds.
There was a problem hiding this comment.
The manager-owned observer with a handle is the shape I hoped for, and the open question about the other secret kinds now sits on the hook doc.
There was a problem hiding this comment.
The observer now lives in the SecretManager as a callback list, so each extension registers its own.
| return absl::OkStatus(); | ||
| })); | ||
| secret_callback_handles_.push_back( | ||
| provider->addRemoveCallback([weak = weak_from_this(), secret_name]() { |
There was a problem hiding this comment.
The remove callback only runs on the delta-xDS branch where the server withdraws the resource and SdsApi deliberately keeps serving the last secret, while a provider leaving the manager with its last listener or cluster fires nothing.
So after this hook fires get_active_resource_names(Secret) still lists the name, and when the secret actually goes away the module hears nothing. Removal should mean the name left the manager, and if the delta withdrawal is worth surfacing it needs its own hook and doc.
There was a problem hiding this comment.
Removal now means the last provider left the manager, which settles the meaning, and the on-demand selector case is a doc note in the new review.
There was a problem hiding this comment.
Removal now fires when the last provider for a name is destroyed.
| // addUpdateCallback fires immediately if the secret is already present, and again on every | ||
| // rotation; addRemoveCallback fires when the resource is explicitly removed. Both run on the main | ||
| // thread. The handles are retained so the subscriptions stay live for the life of the config. | ||
| secret_callback_handles_.push_back( |
There was a problem hiding this comment.
We push two handles per provider creation and never erase them, and nothing tells us when the provider dies, so this grows for the life of the process under remove-and-re-add and on-demand churn. The holders stay safe through still_alive_, and a weak_ptr beside each pair would only hide the growth while the missing removal signal stays, so with the manager-owned observer this is one handle.
There was a problem hiding this comment.
One handle per extension, with each provider's handle erased with the provider, settles the growth.
There was a problem hiding this comment.
Gone. The manager tracks providers and drops them when they are removed.
| // the module hooks may only run on the main thread. Marshal there via post(), copying the name | ||
| // since the view does not outlive the callback, and locking a weak_ptr so a post that outlives | ||
| // the config is dropped rather than dereferencing freed memory. | ||
| main_thread_dispatcher_.post([weak = weak_from_this(), name = std::string(cluster_name)]() { |
There was a problem hiding this comment.
We register these callbacks on main through *tls_, and the cluster manager runs each thread's list on the thread that registered it with the main pass executed inline by runOnAllThreads, so a hook armed on main already fired on main before this change.
The post turns an inline notification into a deferred one, can run a hook after the shutdown handle was reset and after on_shutdown, and leaves the listener sibling inline. I'd drop the marshal and ClusterCallbacksMarshaledToMainThread from this PR unless a crash report survives this trace.
There was a problem hiding this comment.
Thanks for dropping the marshal, though the description still describes it, which the review notes.
There was a problem hiding this comment.
Gone. The observer is registered and called on the main thread.
| bool envoy_dynamic_module_callback_bootstrap_extension_enable_secret_lifecycle( | ||
| envoy_dynamic_module_type_bootstrap_extension_config_envoy_ptr extension_config_envoy_ptr) { | ||
| auto* config = static_cast<DynamicModuleBootstrapExtensionConfig*>(extension_config_envoy_ptr); | ||
| return config->enableSecretLifecycle(); |
There was a problem hiding this comment.
The thread hazard is at the enable entry rather than at delivery: a call from a module thread reads thread_local_data_ out of bounds in a release build, a call from a worker registers on that worker's list which the shutdown reset then erases from main, and this one touches the secret manager's map and callback lists with no synchronization. We guard get_active_resource_names and timer_delete with isMainOrTestThread() plus IS_ENVOY_BUG, could we do the same in all three enable_* entries and return false?
There was a problem hiding this comment.
All three enables now fail closed off the main thread, and main landed the same guards for the cluster and listener entries, so the merge can keep main's copy.
There was a problem hiding this comment.
Done. The enable entry checks the main thread first.
| * envoy_dynamic_module_on_bootstrap_extension_secret_removal). | ||
| * | ||
| * This must be called on the main thread, typically during or after | ||
| * envoy_dynamic_module_on_bootstrap_extension_server_initialized, since the SecretManager is |
There was a problem hiding this comment.
Enabling replays every delivered secret through on_secret_add_or_update before this call returns, since the provider's update callback fires inline on registration, so a module holding a lock or a borrow across the call deadlocks or panics inside its own enable. The Rust doc says so and this header should too. Also the production server creates the SecretManager before any bootstrap extension, so that is not why we wait. The validation server creates it after the extensions and never calls on_server_initialized, which is the case this gate protects.
There was a problem hiding this comment.
The header now warns about the inline replay and names the validation server as the case the gate covers.
There was a problem hiding this comment.
Done. The doc says not to hold a lock or borrow across the call.
|
|
||
| /** | ||
| * envoy_dynamic_module_on_bootstrap_extension_secret_add_or_update is called when a dynamic secret | ||
| * (SDS) becomes active or is updated in the SecretManager. Unlike clusters and listeners, secrets |
There was a problem hiding this comment.
This covers TLS certificate providers only, while get_active_resource_names(Secret) reports certificates, validation contexts, session ticket keys and generic secrets, so a module can enumerate names it will never get events for. Worth saying which kinds fire, that a name can repeat across config sources and warm flags, and that the event means the provider stored the secret rather than that the TLS contexts serve it.
There was a problem hiding this comment.
The doc now says which kinds fire, that a name can repeat and that the event means the provider stored the secret.
There was a problem hiding this comment.
Kept TLS-only and documented it on both hooks.
| init_target_handle->initialize(init_watcher); | ||
| Protobuf::RepeatedPtrField<std::string> removed_resources; | ||
| *removed_resources.Add() = "secret_0"; | ||
| EXPECT_TRUE(secret_context.server_context_.cluster_manager_.subscription_factory_.callbacks_ |
There was a problem hiding this comment.
This only asserts that the delta update returned ok, which it does with no subscriber at all, so the test passes without the removal reaching the module. Could we record the hook through the public function pointer and assert the name?
There was a problem hiding this comment.
The test now records both hooks through the function pointers and asserts the names, the silent delta removal included.
There was a problem hiding this comment.
Done. The test records the hook and asserts the removal reaches it.
| @@ -0,0 +1,8 @@ | |||
| Added dynamic-module bootstrap ABI callbacks for the secret and cluster lifecycle. A bootstrap | |||
There was a problem hiding this comment.
enable_cluster_lifecycle already exists on main and its hooks already ran on the main thread, so this describes the cluster ABI as new and the marshal as a guarantee it does not add. Let's scope the entry to the secret hooks.
There was a problem hiding this comment.
The fragment now covers only the secret hooks.
| (void)listener_name; | ||
| } | ||
|
|
||
| void envoy_dynamic_module_on_bootstrap_extension_secret_add_or_update( |
There was a problem hiding this comment.
These two symbols are optional, so adding them to the nineteen fixtures that exist to fail on a missing required symbol changes nothing there, and the edit here removes the one successfully loading fixture without either hook, which is the module built against an older SDK. I'd keep the bootstrap_no_op.c edit for LoadOK and drop the rest.
There was a problem hiding this comment.
The required-symbol fixtures are untouched again, though after the merge main's optional-hooks module looks like the better home for the new hooks, as the new inline comment says.
There was a problem hiding this comment.
Done. The hooks now live only in the optional-hooks module.
…Manager Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Basundhara Chakrabarty <basundhara17061996@gmail.com>
agrawroh
left a comment
There was a problem hiding this comment.
🚫 Suggested Fixes. The manager-owned observer is the right shape now, and what blocks is the merge with main and a shutdown gap the merge won't carry over on its own.
⚠️ This branch conflicts with main in ten files, so CI never built or tested this head. Could you merge main, write the secret guard in the form main uses for its siblings and add it to their off-main test table, and runbazel run //tools/dynamic_modules/bindgen:fix, since the Rust SDK won't compile against main's checked-in bindings otherwise?⚠️ In init-only mode, or for an enable after shutdown has started, nothing runs this ShutdownExit release, so removal hooks reach the module during server teardown, where a filter-chain lookup from the hook walks a dying or freed listener manager.- ❓ Is TLS-only meant to stay?
- 💡 The on-demand certificate selector drops its provider after a delta removal, so removal can fire then.
- 💡 A TLS context can still reject a stored secret, and a rejected file rotation then repeats this event on every later poll or file event.
- 💡 The replay checks for a held secret, while removal follows the name's sticky delivered state, so an observer that enables after the delivering provider left gets a removal its replay never announced.
- 💡 Main keeps optional bootstrap hooks in the optional-hooks test module, not the no-op one.
- 💡 No test reaches this removal trampoline, and the add trampoline runs only through the integration log.
- 💡 Could we add a weak-stub test row for this, as the cluster and listener enables have?
- 💡 Could both hooks say they're optional, as main's newer bootstrap hooks do, so a module author knows a module without them still loads?
- 💡 Could the fragment also name the two event hooks and the SDK's
enable_secret_lifecycle, since module authors search for the names they call? - 💡 The description still says the cluster hooks are posted to the main thread and names the old changelog file and tests, and it becomes the squash commit. Could you update it to the secret hooks and the new secret manager method?
| secret_lifecycle_enabled_ = true; | ||
| // Replays the delivered secrets inline through onSecretAddOrUpdate. | ||
| secret_lifecycle_callbacks_handle_ = context_.secretManager().addSecretLifecycleCallbacks(*this); | ||
| // Release the handle at shutdown so no secret event reaches the module after it. |
There was a problem hiding this comment.
| /** | ||
| * envoy_dynamic_module_on_bootstrap_extension_secret_add_or_update is called each time a dynamic | ||
| * (SDS) TLS certificate provider stores a secret, on first delivery and on every rotation. Only TLS | ||
| * certificates fire; certificate validation contexts, session ticket keys and generic secrets do |
There was a problem hiding this comment.
❓ Is TLS-only meant to stay? These hooks are named for any secret, and their meaning freezes once they ship, so could the names say TLS certificate, or could they take a kind argument now if other kinds may follow?
There was a problem hiding this comment.
Yes, TLS-only stays. Other secret kinds would get their own hooks, so these names keep their meaning.
| * envoy_dynamic_module_on_bootstrap_extension_secret_removal is called when the last dynamic (SDS) | ||
| * TLS certificate provider for a name leaves the SecretManager, which happens when the last | ||
| * listener or cluster that references it is removed. It fires only for a name that delivered a | ||
| * secret. A delta xDS removal of the resource does not fire it, since the provider keeps serving |
There was a problem hiding this comment.
💡 The on-demand certificate selector drops its provider after a delta removal, so removal can fire then. Could this doc and the SDK's say removal fires when the last provider is destroyed?
| * re-pushed. | ||
| * | ||
| * A name can be held by several providers (one per config source and warming mode), so this can | ||
| * repeat for the same name. The event means the provider stored the secret, not that the TLS |
There was a problem hiding this comment.
💡 A TLS context can still reject a stored secret, and a rejected file rotation then repeats this event on every later poll or file event. Could the doc say both, that the secret may never be served and that the event can repeat?
| tracked_providers_.erase(tracked); | ||
| auto state = names_.find(name); | ||
| ASSERT(state != names_.end()); | ||
| if (--state->second.providers_ > 0) { |
There was a problem hiding this comment.
💡 The replay checks for a held secret, while removal follows the name's sticky delivered state, so an observer that enables after the delivering provider left gets a removal its replay never announced. Could the replay use that delivered state too, with the doc noting a name can leave the active names before its removal?
| @@ -0,0 +1,177 @@ | |||
| #include <stddef.h> | |||
There was a problem hiding this comment.
💡 Main keeps optional bootstrap hooks in the optional-hooks test module, not the no-op one. After the merge, could these hooks move there, with the load checks following them, so the no-op module covers the missing-hook load and these two copies can go?
| /// | ||
| /// This is an FFI function called by Envoy. All pointer arguments must be valid as guaranteed | ||
| /// by the Envoy dynamic module ABI. | ||
| unsafe fn envoy_dynamic_module_on_bootstrap_extension_secret_removal( |
There was a problem hiding this comment.
💡 No test reaches this removal trampoline, and the add trampoline runs only through the integration log. Could we add SDK unit tests for both, as the listener hooks have?
| WEAK_STUB(bool, envoy_dynamic_module_callback_bootstrap_extension_enable_listener_lifecycle, false, | ||
| envoy_dynamic_module_type_bootstrap_extension_config_envoy_ptr) | ||
|
|
||
| WEAK_STUB(bool, envoy_dynamic_module_callback_bootstrap_extension_enable_secret_lifecycle, false, |
There was a problem hiding this comment.
💡 Could we add a weak-stub test row for this, as the cluster and listener enables have?
| * | ||
| * This is only called if the module has opted in to receiving secret lifecycle events via | ||
| * envoy_dynamic_module_callback_bootstrap_extension_enable_secret_lifecycle. The callback is | ||
| * registered on the main thread and invoked on the main thread. |
There was a problem hiding this comment.
💡 Could both hooks say they're optional, as main's newer bootstrap hooks do, so a module author knows a module without them still loads?
| Added dynamic-module bootstrap ABI callbacks for the secret lifecycle. A bootstrap extension opts in | ||
| via ``envoy_dynamic_module_callback_bootstrap_extension_enable_secret_lifecycle`` and is then | ||
| notified when a dynamic TLS certificate secret is delivered or rotated, and when the last provider | ||
| for it is removed. Available through the Rust SDK as ``on_secret_add_or_update`` and |
There was a problem hiding this comment.
💡 Could the fragment also name the two event hooks and the SDK's enable_secret_lifecycle, since module authors search for the names they call?
Signed-off-by: Basundhara Chakrabarty <basundhara17061996@gmail.com>
…lay and docs Signed-off-by: Basundhara Chakrabarty <basundhara17061996@gmail.com>
|
Merged main, regenerated the Rust bindings, and updated the description. |
Signed-off-by: Basundhara Chakrabarty <basundhara17061996@gmail.com>
Commit Message: dynamic_modules: add bootstrap secret lifecycle callbacks
Additional Description:
A bootstrap extension opts in with
enable_secret_lifecycle()and then receiveson_secret_add_or_update(name)when a dynamic TLS certificate secret is delivered or rotated, andon_secret_removal(name)when the last provider for that name is destroyed. SDS secrets are independent xDS resources, so these fire even when no cluster or listener is re-pushed. Enabling replays the names that have already delivered a secret.The
SecretManagerowns the observer through a newaddSecretLifecycleCallbacks()method, which tracks providers per name and reports removal only for names that delivered a secret. Both hooks are optional and run on the main thread. Enabling is refused before the server is initialized, once shutdown has started, and in--mode init_only.Risk Level: Low
Testing:
SecretManagerImpladd, rotate, replay and removal tracking; enable refused before server init and once-only; the hooks reach the module and are optional at load; the off-main-thread and weak-stub tables; SDK trampolines.on_secret_add_or_updatefor a file-based SDS secret on the listener, including a rotation.Docs Changes: ABI and Rust SDK docs for the new hooks and callback.
Release Notes: Added
changelogs/current/new_features/dynamic_modules__bootstrap-secret-lifecycle.rstPlatform Specific Features: N/A
AI Disclosure: I used AI to help write this change and PR description. I understand the code and am responsible for it.