Skip to content

dynamic_modules: add bootstrap secret lifecycle callbacks - #47437

Open
basundhara-c wants to merge 6 commits into
envoyproxy:mainfrom
basundhara-c:bootstrap-secret-lifecycle
Open

basundhara-c wants to merge 6 commits into
envoyproxy:mainfrom
basundhara-c:bootstrap-secret-lifecycle

Conversation

@basundhara-c

@basundhara-c basundhara-c commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Commit Message: dynamic_modules: add bootstrap secret lifecycle callbacks
Additional Description:

A bootstrap extension opts in with enable_secret_lifecycle() and then receives on_secret_add_or_update(name) when a dynamic TLS certificate secret is delivered or rotated, and on_secret_removal(name) when the last provider for that name is destroyed. SDS secrets are independent xDS resources, so these fire even when no cluster or listener is re-pushed. Enabling replays the names that have already delivered a secret.

The SecretManager owns the observer through a new addSecretLifecycleCallbacks() method, which tracks providers per name and reports removal only for names that delivered a secret. Both hooks are optional and run on the main thread. Enabling is refused before the server is initialized, once shutdown has started, and in --mode init_only.

Risk Level: Low
Testing:

  • Unit: SecretManagerImpl add, rotate, replay and removal tracking; enable refused before server init and once-only; the hooks reach the module and are optional at load; the off-main-thread and weak-stub tables; SDK trampolines.
  • Integration: a Rust module observes on_secret_add_or_update for a file-based SDS secret on the listener, including a rotation.

Docs Changes: ABI and Rust SDK docs for the new hooks and callback.
Release Notes: Added changelogs/current/new_features/dynamic_modules__bootstrap-secret-lifecycle.rst
Platform Specific Features: N/A
AI Disclosure: I used AI to help write this change and PR description. I understand the code and am responsible for it.

@repokitteh-read-only

Copy link
Copy Markdown

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #47437 was opened by basundhara-c.

see: more, trace.

@basundhara-c
basundhara-c force-pushed the bootstrap-secret-lifecycle branch 4 times, most recently from 453417d to d480814 Compare September 14, 2026 14:49
@basundhara-c

Copy link
Copy Markdown
Contributor Author

/restest

@basundhara-c

Copy link
Copy Markdown
Contributor Author

/retest

@basundhara-c
basundhara-c marked this pull request as ready for review September 14, 2026 15:05
@basundhara-c
basundhara-c force-pushed the bootstrap-secret-lifecycle branch 3 times, most recently from df39d2a to 111ea52 Compare September 15, 2026 01:15
@agrawroh agrawroh self-assigned this Sep 15, 2026
@basundhara-c
basundhara-c force-pushed the bootstrap-secret-lifecycle branch 2 times, most recently from 5c61519 to c2bbc24 Compare September 15, 2026 04:29
@agrawroh

Copy link
Copy Markdown
Member

Needs a main merge.

/wait

@basundhara-c
basundhara-c force-pushed the bootstrap-secret-lifecycle branch 2 times, most recently from a86ecca to 3ff772e Compare October 6, 2026 20:22
@basundhara-c

Copy link
Copy Markdown
Contributor Author

/retest

@KBaichoo

Copy link
Copy Markdown
Contributor

PTAL when you have time @agrawroh

Signed-off-by: Basundhara Chakrabarty <basundhara17061996@gmail.com>
@basundhara-c
basundhara-c force-pushed the bootstrap-secret-lifecycle branch from 3ff772e to 1f655e5 Compare October 24, 2026 19:11
@basundhara-c basundhara-c changed the title dynamic_modules: add bootstrap secret lifecycle callbacks dynamic_modules: add bootstrap secret lifecycle callbacks Oct 24, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Basundhara Chakrabarty <basundhara17061996@gmail.com>
@basundhara-c

Copy link
Copy Markdown
Contributor Author

/retest

@agrawroh agrawroh left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The secret lifecycle looks like the right thing to add but the core piece is a single replaceable callback that never reports a provider leaving the manager, so removal means the wrong thing and a second extension silently starves the first, and the cluster-hook marshal fixes a worker-thread crash that the per-thread callback registration cannot produce.

I'd like the manager-owned observer and the marshal settled before we go further. The module side otherwise looks fine.

/wait

Comment thread envoy/secret/secret_manager.h Outdated
* callback is held; a later call replaces the earlier one. Default is a no-op so only
* implementations that support secret lifecycle observation need override it.
*/
virtual void setDynamicTlsCertificateSecretProviderCreatedCallback(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This holds one std::function and a later call replaces it, so with two bootstrap extensions the first keeps the subscriptions it already made and never hears about a provider created afterwards, and since removeDynamicSecretProvider emits nothing the observer also cannot learn that a provider died.

The manager is the one place that sees both ends, so I'd rather see the sibling shape here, an addSecretLifecycleCallbacks returning a handle, PURE like dynamicActiveSecretNames above, with the manager owning one update subscription per provider and emitting add-or-update from it and removal from removeDynamicSecretProvider, for all four provider kinds.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The manager-owned observer with a handle is the shape I hoped for, and the open question about the other secret kinds now sits on the hook doc.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The observer now lives in the SecretManager as a callback list, so each extension registers its own.

return absl::OkStatus();
}));
secret_callback_handles_.push_back(
provider->addRemoveCallback([weak = weak_from_this(), secret_name]() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The remove callback only runs on the delta-xDS branch where the server withdraws the resource and SdsApi deliberately keeps serving the last secret, while a provider leaving the manager with its last listener or cluster fires nothing.

So after this hook fires get_active_resource_names(Secret) still lists the name, and when the secret actually goes away the module hears nothing. Removal should mean the name left the manager, and if the delta withdrawal is worth surfacing it needs its own hook and doc.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removal now means the last provider left the manager, which settles the meaning, and the on-demand selector case is a doc note in the new review.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removal now fires when the last provider for a name is destroyed.

// addUpdateCallback fires immediately if the secret is already present, and again on every
// rotation; addRemoveCallback fires when the resource is explicitly removed. Both run on the main
// thread. The handles are retained so the subscriptions stay live for the life of the config.
secret_callback_handles_.push_back(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We push two handles per provider creation and never erase them, and nothing tells us when the provider dies, so this grows for the life of the process under remove-and-re-add and on-demand churn. The holders stay safe through still_alive_, and a weak_ptr beside each pair would only hide the growth while the missing removal signal stays, so with the manager-owned observer this is one handle.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One handle per extension, with each provider's handle erased with the provider, settles the growth.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gone. The manager tracks providers and drops them when they are removed.

// the module hooks may only run on the main thread. Marshal there via post(), copying the name
// since the view does not outlive the callback, and locking a weak_ptr so a post that outlives
// the config is dropped rather than dereferencing freed memory.
main_thread_dispatcher_.post([weak = weak_from_this(), name = std::string(cluster_name)]() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We register these callbacks on main through *tls_, and the cluster manager runs each thread's list on the thread that registered it with the main pass executed inline by runOnAllThreads, so a hook armed on main already fired on main before this change.

The post turns an inline notification into a deferred one, can run a hook after the shutdown handle was reset and after on_shutdown, and leaves the listener sibling inline. I'd drop the marshal and ClusterCallbacksMarshaledToMainThread from this PR unless a crash report survives this trace.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for dropping the marshal, though the description still describes it, which the review notes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gone. The observer is registered and called on the main thread.

bool envoy_dynamic_module_callback_bootstrap_extension_enable_secret_lifecycle(
envoy_dynamic_module_type_bootstrap_extension_config_envoy_ptr extension_config_envoy_ptr) {
auto* config = static_cast<DynamicModuleBootstrapExtensionConfig*>(extension_config_envoy_ptr);
return config->enableSecretLifecycle();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thread hazard is at the enable entry rather than at delivery: a call from a module thread reads thread_local_data_ out of bounds in a release build, a call from a worker registers on that worker's list which the shutdown reset then erases from main, and this one touches the secret manager's map and callback lists with no synchronization. We guard get_active_resource_names and timer_delete with isMainOrTestThread() plus IS_ENVOY_BUG, could we do the same in all three enable_* entries and return false?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three enables now fail closed off the main thread, and main landed the same guards for the cluster and listener entries, so the merge can keep main's copy.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The enable entry checks the main thread first.

* envoy_dynamic_module_on_bootstrap_extension_secret_removal).
*
* This must be called on the main thread, typically during or after
* envoy_dynamic_module_on_bootstrap_extension_server_initialized, since the SecretManager is

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enabling replays every delivered secret through on_secret_add_or_update before this call returns, since the provider's update callback fires inline on registration, so a module holding a lock or a borrow across the call deadlocks or panics inside its own enable. The Rust doc says so and this header should too. Also the production server creates the SecretManager before any bootstrap extension, so that is not why we wait. The validation server creates it after the extensions and never calls on_server_initialized, which is the case this gate protects.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The header now warns about the inline replay and names the validation server as the case the gate covers.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The doc says not to hold a lock or borrow across the call.


/**
* envoy_dynamic_module_on_bootstrap_extension_secret_add_or_update is called when a dynamic secret
* (SDS) becomes active or is updated in the SecretManager. Unlike clusters and listeners, secrets

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This covers TLS certificate providers only, while get_active_resource_names(Secret) reports certificates, validation contexts, session ticket keys and generic secrets, so a module can enumerate names it will never get events for. Worth saying which kinds fire, that a name can repeat across config sources and warm flags, and that the event means the provider stored the secret rather than that the TLS contexts serve it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc now says which kinds fire, that a name can repeat and that the event means the provider stored the secret.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept TLS-only and documented it on both hooks.

init_target_handle->initialize(init_watcher);
Protobuf::RepeatedPtrField<std::string> removed_resources;
*removed_resources.Add() = "secret_0";
EXPECT_TRUE(secret_context.server_context_.cluster_manager_.subscription_factory_.callbacks_

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only asserts that the delta update returned ok, which it does with no subscriber at all, so the test passes without the removal reaching the module. Could we record the hook through the public function pointer and assert the name?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test now records both hooks through the function pointers and asserts the names, the silent delta removal included.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The test records the hook and asserts the removal reaches it.

@@ -0,0 +1,8 @@
Added dynamic-module bootstrap ABI callbacks for the secret and cluster lifecycle. A bootstrap

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enable_cluster_lifecycle already exists on main and its hooks already ran on the main thread, so this describes the cluster ABI as new and the marshal as a guarantee it does not add. Let's scope the entry to the secret hooks.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fragment now covers only the secret hooks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

(void)listener_name;
}

void envoy_dynamic_module_on_bootstrap_extension_secret_add_or_update(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two symbols are optional, so adding them to the nineteen fixtures that exist to fail on a missing required symbol changes nothing there, and the edit here removes the one successfully loading fixture without either hook, which is the module built against an older SDK. I'd keep the bootstrap_no_op.c edit for LoadOK and drop the rest.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The required-symbol fixtures are untouched again, though after the merge main's optional-hooks module looks like the better home for the new hooks, as the new inline comment says.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The hooks now live only in the optional-hooks module.

…Manager

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Basundhara Chakrabarty <basundhara17061996@gmail.com>

@agrawroh agrawroh left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 Suggested Fixes. The manager-owned observer is the right shape now, and what blocks is the merge with main and a shutdown gap the merge won't carry over on its own.

  • ⚠️ This branch conflicts with main in ten files, so CI never built or tested this head. Could you merge main, write the secret guard in the form main uses for its siblings and add it to their off-main test table, and run bazel run //tools/dynamic_modules/bindgen:fix, since the Rust SDK won't compile against main's checked-in bindings otherwise?
  • ⚠️ In init-only mode, or for an enable after shutdown has started, nothing runs this ShutdownExit release, so removal hooks reach the module during server teardown, where a filter-chain lookup from the hook walks a dying or freed listener manager.
  • ❓ Is TLS-only meant to stay?
  • 💡 The on-demand certificate selector drops its provider after a delta removal, so removal can fire then.
  • 💡 A TLS context can still reject a stored secret, and a rejected file rotation then repeats this event on every later poll or file event.
  • 💡 The replay checks for a held secret, while removal follows the name's sticky delivered state, so an observer that enables after the delivering provider left gets a removal its replay never announced.
  • 💡 Main keeps optional bootstrap hooks in the optional-hooks test module, not the no-op one.
  • 💡 No test reaches this removal trampoline, and the add trampoline runs only through the integration log.
  • 💡 Could we add a weak-stub test row for this, as the cluster and listener enables have?
  • 💡 Could both hooks say they're optional, as main's newer bootstrap hooks do, so a module author knows a module without them still loads?
  • 💡 Could the fragment also name the two event hooks and the SDK's enable_secret_lifecycle, since module authors search for the names they call?
  • 💡 The description still says the cluster hooks are posted to the main thread and names the old changelog file and tests, and it becomes the squash commit. Could you update it to the secret hooks and the new secret manager method?

secret_lifecycle_enabled_ = true;
// Replays the delivered secrets inline through onSecretAddOrUpdate.
secret_lifecycle_callbacks_handle_ = context_.secretManager().addSecretLifecycleCallbacks(*this);
// Release the handle at shutdown so no secret event reaches the module after it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ In init-only mode, or for an enable after shutdown has started, nothing runs this ShutdownExit release, so removal hooks reach the module during server teardown, where a filter-chain lookup from the hook walks a dying or freed listener manager. Could this refuse both cases, as main's cluster and listener enables now do, with their doc sentence and test row?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

/**
* envoy_dynamic_module_on_bootstrap_extension_secret_add_or_update is called each time a dynamic
* (SDS) TLS certificate provider stores a secret, on first delivery and on every rotation. Only TLS
* certificates fire; certificate validation contexts, session ticket keys and generic secrets do

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ Is TLS-only meant to stay? These hooks are named for any secret, and their meaning freezes once they ship, so could the names say TLS certificate, or could they take a kind argument now if other kinds may follow?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, TLS-only stays. Other secret kinds would get their own hooks, so these names keep their meaning.

* envoy_dynamic_module_on_bootstrap_extension_secret_removal is called when the last dynamic (SDS)
* TLS certificate provider for a name leaves the SecretManager, which happens when the last
* listener or cluster that references it is removed. It fires only for a name that delivered a
* secret. A delta xDS removal of the resource does not fire it, since the provider keeps serving

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 The on-demand certificate selector drops its provider after a delta removal, so removal can fire then. Could this doc and the SDK's say removal fires when the last provider is destroyed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

* re-pushed.
*
* A name can be held by several providers (one per config source and warming mode), so this can
* repeat for the same name. The event means the provider stored the secret, not that the TLS

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 A TLS context can still reject a stored secret, and a rejected file rotation then repeats this event on every later poll or file event. Could the doc say both, that the secret may never be served and that the event can repeat?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

tracked_providers_.erase(tracked);
auto state = names_.find(name);
ASSERT(state != names_.end());
if (--state->second.providers_ > 0) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 The replay checks for a held secret, while removal follows the name's sticky delivered state, so an observer that enables after the delivering provider left gets a removal its replay never announced. Could the replay use that delivered state too, with the doc noting a name can leave the active names before its removal?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -0,0 +1,177 @@
#include <stddef.h>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Main keeps optional bootstrap hooks in the optional-hooks test module, not the no-op one. After the merge, could these hooks move there, with the load checks following them, so the no-op module covers the missing-hook load and these two copies can go?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

///
/// This is an FFI function called by Envoy. All pointer arguments must be valid as guaranteed
/// by the Envoy dynamic module ABI.
unsafe fn envoy_dynamic_module_on_bootstrap_extension_secret_removal(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 No test reaches this removal trampoline, and the add trampoline runs only through the integration log. Could we add SDK unit tests for both, as the listener hooks have?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

WEAK_STUB(bool, envoy_dynamic_module_callback_bootstrap_extension_enable_listener_lifecycle, false,
envoy_dynamic_module_type_bootstrap_extension_config_envoy_ptr)

WEAK_STUB(bool, envoy_dynamic_module_callback_bootstrap_extension_enable_secret_lifecycle, false,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Could we add a weak-stub test row for this, as the cluster and listener enables have?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

*
* This is only called if the module has opted in to receiving secret lifecycle events via
* envoy_dynamic_module_callback_bootstrap_extension_enable_secret_lifecycle. The callback is
* registered on the main thread and invoked on the main thread.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Could both hooks say they're optional, as main's newer bootstrap hooks do, so a module author knows a module without them still loads?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Added dynamic-module bootstrap ABI callbacks for the secret lifecycle. A bootstrap extension opts in
via ``envoy_dynamic_module_callback_bootstrap_extension_enable_secret_lifecycle`` and is then
notified when a dynamic TLS certificate secret is delivered or rotated, and when the last provider
for it is removed. Available through the Rust SDK as ``on_secret_add_or_update`` and

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Could the fragment also name the two event hooks and the SDK's enable_secret_lifecycle, since module authors search for the names they call?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Signed-off-by: Basundhara Chakrabarty <basundhara17061996@gmail.com>
…lay and docs

Signed-off-by: Basundhara Chakrabarty <basundhara17061996@gmail.com>
@basundhara-c

Copy link
Copy Markdown
Contributor Author

Merged main, regenerated the Rust bindings, and updated the description.

Signed-off-by: Basundhara Chakrabarty <basundhara17061996@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants