Skip to content

overlay: remove shell spawns in init_system_info() to avoid seccomp failures when running within pressure-vessel - #1983

Open
kakra wants to merge 3 commits into
flightlessmango:masterfrom
kakra:fixes/seccomp-crashes-inside-pressure-vessel
Open

kakra wants to merge 3 commits into
flightlessmango:masterfrom
kakra:fixes/seccomp-crashes-inside-pressure-vessel

Conversation

@kakra

@kakra kakra commented Mar 8, 2026

Copy link
Copy Markdown

This PR removes shell-based system info collection from init_system_info() and replaces it with native code paths.

Motivation

In Steam Runtime / pressure-vessel setups (notably via Wine + Vulkan layer), spawning /bin/sh pipelines from MangoHud can fail under runtime/seccomp constraints and produce noisy logs and host coredumps (e.g. dash).

By removing those shell callouts, MangoHud avoids this failure mode during initialization.

Ref: ValveSoftware/steam-runtime#804


Exact scope of this PR

init_system_info() no longer shells out for basic system info

Replaced shell pipelines with native logic:

  • ram:

    • before: sed on /proc/meminfo
    • now: sysinfo(2) (totalram * mem_unit, converted to KiB)
  • kernel:

    • before: uname -r shell call
    • now: uname(2) via utsname.release
  • cpu:

    • before: sed ... /proc/cpuinfo | sed 's/([^)]*)//g;s/ / /g' | tail -n1
    • now: direct /proc/cpuinfo parsing in C++, plus equivalent cleanup of parenthesized blocks/spacing
  • os:

    • before: sed on /etc/os-release for PRETTY_NAME
    • now: direct /etc/os-release parsing in C++, then quote removal

Environment mutation scope reduced

Because the above shell spawns are gone, LD_PRELOAD handling was narrowed down to only where subprocess spawning is still needed (Wine version probing path).

The disabled OpenGL/glxinfo callout block is guarded so re-enabling it cannot silently miss the required LD_PRELOAD workaround.

No broader refactors

This PR intentionally does not migrate every remaining external callout in the file. Only the init_system_info() shell pipelines are targeted to fix the runtime failure mode above.


Behavioral notes

  • Intended output stays consistent with prior behavior, including CPU-name cleanup semantics.
  • Main expected impact: no /usr/bin/dash coredumps in affected pressure-vessel scenarios.
  • Nice side effect: lower init overhead from avoiding shell/process startup (potentially visible as faster game start, and possibly less overhead if/when initialization paths are re-entered).

Future considerations

This change reduces getenv()/setenv() usage in one hot path, but the broader pattern still exists in other parts of the codebase.

Given MangoHud runs as a Vulkan layer inside the game process, process-global environment mutation is risky:

  • getenv()/setenv() are process-global and generally not a good fit for highly concurrent in-process code.
  • Behavior is timing-sensitive and can become hard to reason about across threads/components.
  • It may work most of the time, but failures are non-local and difficult to debug.

Follow-up work could:

  1. Audit all environment mutation/read patterns in layer/runtime paths.
  2. Minimize or eliminate process-global env writes in runtime code.
  3. Prefer explicit per-call process setup (where external tools are unavoidable), or native APIs that avoid subprocesses.
  4. Isolate remaining external callouts behind strict wrappers with clear threading/lifetime constraints.

@smcv

smcv commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

In Steam Runtime / pressure-vessel setups (notably via Wine + Vulkan layer), spawning /bin/sh pipelines from MangoHud can fail under runtime/seccomp constraints

We don't know what component is applying these seccomp filters: it might be something "larger" than the Steam Linux Runtime / pressure-vessel container, or it might be something that runs inside the container, like Proton. SLR/PV itself does not add any seccomp rules.

@kakra

kakra commented Mar 9, 2026 •

Copy link
Copy Markdown
Author

We don't know what component is applying these seccomp filters: it might be something "larger" than the Steam Linux Runtime / pressure-vessel container, or it might be something that runs inside the container, like Proton. SLR/PV itself does not add any seccomp rules.

Yes, I agree, my statement wasn't precise here. But whatever the root cause of the observed crashes is, it uncovered problematic code paths in MangoHud which should be fixed no matter what happens elsewhere. I deliberately chose to not write "sandbox" here to support the fact that the pressure-vessel container isn't a security boundary. We don't know yet where the seccomp filter originates but MangoHud trips over it. These commits mitigate the crashes (but there are still other potential crash scenarios, with or without pressure-vessel / SRT).

Comment thread src/overlay.cpp Resolved
kakra added a commit to kakra/MangoHud that referenced this pull request Mar 15, 2026
When `glxinfo` is not called, setting up `MANGOHUD_RECURSION` is dead
code: we read, set and unset the variable without any effect.

Dropping this block also avoids unnecessary environment mutation
in-process. `getenv()`/`setenv()` are not thread-safe in general, so
avoiding them here reduces risk in multi-threaded contexts.

v2: Dropping this code completely instead of only disabling it as per
the discussion with @flightlessmango.

Ref: flightlessmango#1983 (comment)
@kakra
kakra force-pushed the fixes/seccomp-crashes-inside-pressure-vessel branch from 8ae04a8 to 64dbedd Compare March 15, 2026 16:47
@kakra

kakra commented Mar 15, 2026

Copy link
Copy Markdown
Author

@flightlessmango Before I invest more time in this potentially to be removed code, is there a branch with your current refactoring progress, or are there plans when it is ready?

@kakra

kakra commented Mar 15, 2026 •

Copy link
Copy Markdown
Author

I just saw that shell.cpp already implements most of my ideas.

@flightlessmango

Copy link
Copy Markdown
Owner

The working branch is server2. It's a structural change shifting MangoHud to a server/client model, so the main areas to look at are the server and client directories. The concept of executing commands has not been implemented yet and the design is not finalized, so that part is still open for exploration or discussion

@kakra

kakra commented Mar 16, 2026

Copy link
Copy Markdown
Author

Thanks, I'll look at it. I think one focus area is that the Vulkan layer avoids mutating states it doesn't control, especially if thread safety is concerned (getenv/setenv), and avoid forking shell pipelines. It looks like a client/server model is a very good way to do that. I'm somewhat excited. :-)

@kakra

kakra commented Apr 22, 2026

Copy link
Copy Markdown
Author

With latest Proton Experimental (v11), I now see these:

# coredumpctl info 395606
           PID: 395606 (sh)
           UID: 1000 (kakra)
           GID: 1000 (kakra)
        Signal: 31 (SYS)
     Timestamp: Wed 2026-04-22 02:13:09 CEST (44s ago)
  Command Line: sh -c -- $'unset LD_PRELOAD; "/home/kakra/.local/share/Steam/steamapps/common/Proton - Experimental/files/lib/wine/x86_64-unix/wine64" --version'
    Executable: /usr/bin/dash
 Control Group: /user.slice/user-1000.slice/user@1000.service/app.slice/app-steam-app359320-395004.scope
          Unit: user@1000.service
     User Unit: app-steam-app359320-395004.scope
         Slice: user-1000.slice
     Owner UID: 1000 (kakra)
       Boot ID: b592954fa09240448c590b45cca33752
    Machine ID: 121b87ca633e8ac0016656680000001b
      Hostname: jupiter
       Storage: /var/lib/systemd/coredump/core.sh.1000.b592954fa09240448c590b45cca33752.395606.1776816789000000.zst (present)
  Size on Disk: 25.8K
       Message: Process 395606 (sh) of user 1000 dumped core.

                Stack trace of thread 395606:
                #0  0x000055b19a630544 __GI___getrlimit64 (/run/host/usr/lib64/libc.so.6 + 0xfe544)
                #1  0x000055b19a687ff0 __pthread_early_init (/run/host/usr/lib64/libc.so.6 + 0x155ff0)
                #2  0x00007fb2d2b776d9 dl_main (ld-linux-x86-64.so.2 + 0x216d9)
                #3  0x00007fb2d2b73ef3 _dl_sysdep_start (ld-linux-x86-64.so.2 + 0x1def3)
                #4  0x00007fb2d2b75554 _dl_start_final (ld-linux-x86-64.so.2 + 0x1f554)
                #5  0x00007fb2d2b74448 _start (ld-linux-x86-64.so.2 + 0x1e448)
                ELF object binary architecture: AMD x86-64

This is caused by the shell spawns which my patches didn't touch yet, notably spawning wine to read the wine version.

When `glxinfo` is not called, setting up `MANGOHUD_RECURSION` is dead
code: we read, set and unset the variable without any effect.

Dropping this block also avoids unnecessary environment mutation
in-process. `getenv()`/`setenv()` are not thread-safe in general, so
avoiding them here reduces risk in multi-threaded contexts.

v2: Dropping this code completely instead of only disabling it as per
the discussion with @flightlessmango.

Ref: flightlessmango#1983 (comment)
Spawning a shell from inside pressure-vessel is fragile and can fail
due to seccomp constraints, causing noisy logs and host coredumps.

Replace the shell pipeline (`sh|sed|tail`) used in `init_system_info()`
with native parsing and Linux APIs where available. This avoids
subprocess creation entirely for the covered paths.

This does not address every shell-based code path yet (the Wine
version query still runs `wine --version`); that is handled
separately.

The primary goal is to prevent `/usr/bin/dash` crashes from the Steam
Runtime when MangoHud runs inside pressure-vessel.

Ref: ValveSoftware/steam-runtime#804
init_system_info() still ran "<dir>/wine --version" through popen() to
get the Wine version of non-Proton prefixes, and unset LD_PRELOAD and
WINELOADERNOEXEC around that call before restoring them. Mutating the
environment is not thread-safe while other threads of the game may call
getenv(), forking a multi-threaded process is fragile, and the spawn
fails under the seccomp filter inside pressure-vessel. Drop the callout
together with the environment workaround.

Wine is now detected by its unix-side ntdll.so being loaded, so Wine
layouts whose /proc/self/exe is not wine-preloader or wine64-preloader
are detected as well. ntdll.so carries the build string ("wine-X.Y",
identical to what "wine --version" prints) in a read-only segment. Find
the module with dl_iterate_phdr() and scan its read-only, non-executable
PT_LOAD segments for that string, without dlopen() and without calling
into Wine.

Proton detection stays path-based and keeps upstream's layouts,
including /files/lib/wine. parse_proton_version() now returns the
version instead of writing the global, and no longer runs dirname() on
the shared wineProcess buffer. If the Proton version file is missing or
empty, fall back to "Proton " plus the scanned Wine build string.
@kakra
kakra force-pushed the fixes/seccomp-crashes-inside-pressure-vessel branch from 64dbedd to 77c647b Compare November 14, 2026 12:54
@kakra

kakra commented Nov 14, 2026

Copy link
Copy Markdown
Author

@flightlessmango Rebased onto v0.8.4 and force-pushed.

Rebase: v0.8.4 added parse_proton_version() and the /files/lib/wine Proton layout in the same spot this PR touches. I kept upstream's layouts and turned parse_proton_version() into a function that returns the version string instead of writing the global, so it no longer calls dirname() on the shared wineProcess buffer.

Squash / new commit: The former "narrow getenv()/setenv() scope to the remaining callout" commit is gone; the callout itself is now removed by "overlay: read Wine version from loaded ntdll.so instead of spawning wine". Instead of running <dir>/wine --version (with LD_PRELOAD/WINELOADERNOEXEC unset and restored around it), MangoHud finds Wine's unix-side ntdll.so via dl_iterate_phdr() and reads the build string (wine-X.Y, the same string wine --version prints) from its read-only PT_LOAD segments. No dlopen(), no calls into Wine, no fork, no environment changes. A loaded ntdll.so is also used as the Wine signal, so layouts whose exe is not wine(64)-preloader are detected too.

With this, init_system_info() spawns no subprocesses anymore, and src/ no longer contains any setenv()/unsetenv() call. Background: while running Elite Dangerous with the ED-UP d3d11 proxy under Proton with MangoHud active, I ran into Steam integration failures (launcher Steam auth rejected, Steam overlay crash). I haven't proven the root cause, but in-process environment mutation from a non-game thread is a plausible contributor, which is one more reason to get rid of it.

Testing: Builds cleanly on v0.8.4, each commit compiles on its own. The scanner was verified against the actual ntdll.so files (i386 and x86_64) of Proton Experimental, Hotfix, 10.0 and 9.0 (returns wine-11.0/wine-10.0/wine-9.0), with libc and libMangoHud as negative controls. Runtime testing in a game is still pending, and I don't run plain (non-Proton) Wine, which uses the same path; feedback from someone who does would be welcome.

I'll look at the server2 branch separately and see how these patches port onto it.

@smcv

smcv commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

We don't know what component is applying these seccomp filters: it might be something "larger" than the Steam Linux Runtime / pressure-vessel container, or it might be something that runs inside the container, like Proton. SLR/PV itself does not add any seccomp rules.

While you're updating this PR, I would suggest changing the title: at the moment it's misleading. pressure-vessel still doesn't create any seccomp rules. We still don't know what component on your system is adding them, but it isn't PV.

@kakra kakra changed the title overlay: remove shell spawns in init_system_info() to avoid pressure-vessel seccomp failures overlay: remove shell spawns in init_system_info() to avoid seccomp failures when running within pressure-vessel Nov 16, 2026
@kakra

kakra commented Nov 16, 2026 •

Copy link
Copy Markdown
Author

@smcv It's a really bad idea anyways to mutate env and fork from a game thread - which is essentially where the layer is running. So whatever seccomp was doing here, it was probably right to do that. Forks and env mutation are now gone in my PR. Title changed.

@smcv

smcv commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

It's a really bad idea anyways to mutate env and fork from a game thread - which is essentially where the layer is running.

I agree, especially modifying environ which is known not to be thread-safe (as documented in setenv(3) and putenv(3)).

Forking subprocesses without modifying environ can sometimes be OK, but is also best avoided where possible, partly because it's documented to be unsafe to call any non-async-signal-safe function in the child of a fork() (and programming with only async-signal-safe functions is difficult) and partly because the subprocess can inherit an unexpected execution environment from its parent (seccomp filters, non-standard environment variables, resource limits and so on).

That's all orthogonal to the use of pressure-vessel, though - it has more to do with the fact that MangoHud is running inside a game process that is not under its control.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants