Skip to content

fix: prevent SIGABRT crash in protonvpn status from Rust local agent background threads - #19

Open
sicambria wants to merge 1 commit into
ProtonVPN:stablefrom
sicambria:fix/sigabrt-local-agent-crash
Open

fix: prevent SIGABRT crash in protonvpn status from Rust local agent background threads#19
sicambria wants to merge 1 commit into
ProtonVPN:stablefrom
sicambria:fix/sigabrt-local-agent-crash

Conversation

@sicambria

@sicambria sicambria commented Jun 9, 2026

Copy link
Copy Markdown

Fix protonvpn status SIGABRT crash from Rust local agent threads

Problem

protonvpn status crashes with SIGABRT and dumps a ~15 MB core when the VPN is connected. The crash originates in local_agent.abi3.so — the Rust native extension for ProtonVPN's local agent protocol.

Environment

Component Version
OS Linux Mint 22.3 (Zena), kernel 7.0.0-14-generic, x86_64
Python 3.12.3
proton-vpn-cli 1.0.1
proton-vpn-api-core 5.2.4
proton-core 0.7.4
local_agent.abi3.so 4.7 MB (release build, no debug symbols)

How to reproduce

  1. Connect to any server: protonvpn connect <server> (WireGuard protocol).
  2. Run protonvpn status while connected.
  3. Check the exit code (echo $?) and look for a new core dump (coredumpctl list | tail).

On affected systems, status prints the connection details correctly but the process then aborts during interpreter shutdown (non-zero/abort exit code, new core dump entry).

Core dump evidence

Thread Frame
crashing PyObject_Call from local_agent.abi3.soPyEval_RestoreThread detects finalization → PyThread_exit_threadabort()
main Py_FinalizeExPyGC_Collect — normal interpreter shutdown
idle workers 15 idle tokio worker threads from the Rust runtime

Root cause trace

protonvpn status
  → status()  [server.py:239]
    → controller.get_vpn_connector()
      → VPNConnector.initialize_state()
        → _get_current_connection()
          → loads persisted connection from disk
          → Wireguard(server, creds, settings, connection_id=...)
            → _initialize_persisted_connection()
              → _async_start_local_agent_listener()
                → asyncio.run_coroutine_threadsafe(listen(), loop)
                  → Listener.connect(domain, key, cert)
                  → Listener.listen(on_status, on_error)
                    → spawns TOKIO THREADS holding Python callbacks via PyO3

The status command starts the local agent listener solely to read the current state. The Rust Listener spawns a persistent tokio runtime with background threads that hold Python callbacks. When the one-shot CLI exits:

  1. asyncio.run() completes → Python enters Py_FinalizeEx
  2. _Py_IsFinalizing() is set to true
  3. A tokio thread wakes with a new status update
  4. PyO3's Python::with_gil() calls PyEval_RestoreThread
  5. Which checks _Py_IsFinalizing() → calls PyThread_exit_thread()abort()SIGABRT

Fix

Two complementary layers:

Layer 1: Explicit listener stop (server.py)

After the status output is printed, stop the AgentListener if it is running. This cancels the Rust Listener.listen() future while the asyncio loop is still alive, giving tokio time to drain its tasks before Python shutdown.

if connection is not None and hasattr(connection, '_agent_listener'):
    try:
        agent_listener = connection._agent_listener
        if agent_listener.is_running:
            await asyncio.wait_for(agent_listener.stop(), timeout=5.0)
    except Exception:  # pylint: disable=broad-except
        logger.debug("Failed to stop local agent listener.", exc_info=True)

A 5-second asyncio.wait_for timeout prevents hanging if the Rust side is slow to respond. Failures are logged at debug level rather than silently swallowed, so cleanup can never crash the command but is still visible when debugging.

Layer 2: SIGABRT safety net (__init__.py)

An atexit-registered callback installs a SIGABRT signal handler, active only during the interpreter-shutdown window. If any remaining Rust code (e.g. the init_logger callback stored in the native module global state) triggers a signal during Py_FinalizeEx, it logs a warning to stderr and converts the crash to a clean os._exit(1) instead of a core dump.

def _handle_shutdown_abort(_signum, _frame):
    sys.stderr.write(
        "Warning: suppressed a SIGABRT during interpreter shutdown "
        "(likely from local_agent.abi3.so background threads).\n"
    )
    os._exit(1)


def _install_abort_guard():
    signal.signal(signal.SIGABRT, _handle_shutdown_abort)


atexit.register(_install_abort_guard)

Why the order matters

Phase What happens
status() runs LA listener started by Wireguard._initialize_persisted_connection()
listener .stop() called Rust Listener.listen() future cancelled; tokio tasks dropped — threads go idle
asyncio.run() returns Event loop closes; remaining Python tasks cancelled
atexit fires SIGABRT handler installed for the shutdown window
Py_FinalizeEx() runs If init_logger or any residual Rust code fires, SIGABRT is logged and converted to os._exit(1)
Process exits Clean exit, no core dump

Tests

Added unit tests in tests/unit/commands/test_server.py covering the new status cleanup path:

  • test_status_stops_local_agent_listener_when_running
  • test_status_does_not_stop_local_agent_listener_when_not_running
  • test_status_succeeds_even_if_stopping_local_agent_listener_fails

Manual verification

=== 10 consecutive runs of `protonvpn status` ===
Run  1:  exit 0
Run  2:  exit 0
Run  3:  exit 0
Run  4:  exit 0
Run  5:  exit 0
Run  6:  exit 0
Run  7:  exit 0
Run  8:  exit 0
Run  9:  exit 0
Run 10:  exit 0
=== Core dumps: 0 new ===

Files changed

File Change
proton/vpn/cli/commands/server.py Explicit AgentListener.stop() with 5s timeout after status display
proton/vpn/cli/__init__.py atexit SIGABRT guard during interpreter shutdown, logs before exiting
tests/unit/commands/test_server.py New tests for the agent listener cleanup

Known limitation

local_agent.abi3.so stores proton.vpn.logging.getLogger as a global Python callback via init_logger(). This callback cannot be unset from Python. If Rust code emits a log during its own native module cleanup, the SIGABRT handler catches it and logs a warning. A proper fix in proton-vpn-local-agent — checking Py_IsInitialized() before dispatching Python callbacks from tokio threads — would make the signal handler unnecessary.

@sicambria sicambria closed this Jun 9, 2026
@sicambria
sicambria deleted the fix/sigabrt-local-agent-crash branch June 9, 2026 23:27
@sicambria sicambria reopened this Jun 10, 2026
…t background threads

Core dump analysis confirmed the crash path:

  status()
    -> get_vpn_connector()
      -> VPNConnector.initialize_state()
        -> _get_current_connection()
          -> Wireguard(connection_id=...)
            -> _initialize_persisted_connection()
              -> _async_start_local_agent_listener()
                -> Listener.listen(on_status, on_error)
                  -> Rust tokio threads hold PyO3 Python callbacks

  CLI exits -> Py_FinalizeEx -> _Py_IsFinalizing() = true
    -> tokio thread dispatches callback via Python::with_gil()
      -> PyEval_RestoreThread detects finalization
        -> PyThread_exit_thread() -> abort() -> SIGABRT

The status command starts the local agent listener only to read the
current connection state. The listener spawns a persistent tokio
runtime whose background threads outlive the Python interpreter.

Fix (two complementary layers):

1. Explicitly stop() the AgentListener after displaying status, so
   tokio tasks are dropped while the asyncio loop is still alive.
   Failures are logged at debug level rather than swallowed silently.

2. Register an atexit SIGABRT guard as a safety net for any residual
   Rust callbacks (e.g. init_logger) during Py_FinalizeEx. Logs a
   warning to stderr before exiting cleanly so the occurrence is
   still visible.

Verified: 10/10 consecutive invocations, exit code 0, zero core dumps.
@sicambria
sicambria force-pushed the fix/sigabrt-local-agent-crash branch from 3a9784e to 2140650 Compare June 10, 2026 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant