Skip to content

Add experimental NVENC-backed VA-API encode support - #438

Open
ntoskrnl7 wants to merge 14 commits into
elFarto:masterfrom
ntoskrnl7:nvenc
Open

ntoskrnl7 wants to merge 14 commits into
elFarto:masterfrom
ntoskrnl7:nvenc

Conversation

@ntoskrnl7

Copy link
Copy Markdown

Summary

This PR adds experimental VA-API encode support backed by NVENC while keeping the existing decode-focused behavior intact.

The new encode path exposes VAEntrypointEncSlice only after runtime NVENC capability probing, so the advertised profiles should match the target GPU, driver, and NVENC API support.

What changed

  • Added experimental NVENC-backed encode support for H.264, HEVC/H.265, and AV1.
  • Added runtime capability probing for codec/profile support before advertising encode entrypoints.
  • Added CBR, VBR, and CQP rate-control mapping from VA-API parameters to NVENC.
  • Added support for capability-dependent profiles such as H.264 High10, HEVC Main10, HEVC Main422_10, HEVC Main444/Main444_10, and AV1 10-bit input where supported.
  • Improved direct backend external surface import paths for encode workloads, including CUDA/DMABUF import and fallback copy paths.
  • Added runtime smoke tests for advertised VA-API encode profiles using vainfo and ffmpeg.
  • Updated documentation for encode support, limitations, testing commands, and Chrome usage through an external hotpatch runtime.

Limitations

This is intentionally experimental and aimed at transcoding-style workloads.

  • NVENC public APIs do not expose HEVC 12-bit encode, so HEVC Main12/Main444_12 remain decode-only.
  • VP8, VP9, MPEG-2, VC-1, MPEG-4, JPEG, and MJPEG encode are not implemented.
  • Some client paths may renegotiate pixel formats before frames reach the driver, especially HEVC rext/high-bit-depth cases.
  • Chrome still requires an external runtime such as ntoskrnl7/chrome-vaapi-hotpatch; this PR does not make stock Chrome use the driver by itself.

Testing

Tested with:

  • meson test
  • vainfo --display drm --device /dev/dri/renderD128
  • ffmpeg VA-API encode/decode checks for H.264 and HEVC
  • Runtime matrix through driver-runtime-vaapi-smoke

Reference validation documented in the README:

  • Functional matrix: 51/51 pass
  • Stress suite: 20/20 pass
  • Tested across H.264, HEVC Main10, HEVC rext444, CBR/VBR/CQP, long-run, 4K, and parallel encode cases

@thesword53

Copy link
Copy Markdown
Contributor

The RDD process crashes when Firefox tries to encode a video:

#0  0x00007f8012ca398d in negotiateNvencApiVersion (drv=drv@entry=0x7f7ff4ffb700, reason=reason@entry=0x7f8012cbc44e "encode probe") at ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:2366
#1  0x00007f8012caaf56 in probeEncodeSupport (drv=0x7f7ff4ffb700) at ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:4786
#2  0x00007f8012cb3478 in __vaDriverInit_1_0 (ctx=0x7f80253db5e0) at ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:9685
#3  0x00007f80213cfccc in va_openDriver (dpy=0x7f8037d59510, driver_name=<optimized out>) at ../libva/va/va.c:465
#4  va_new_opendriver (dpy=0x7f8037d59510) at ../libva/va/va.c:701
#5  vaInitialize (dpy=0x7f8037d59510, major_version=0x7f801f1bc4a8, minor_version=0x7f801f1bc4a4) at ../libva/va/va.c:730
#6  vaInitialize (dpy=0x7f8037d59510, major_version=0x7f801f1bc4a8, minor_version=0x7f801f1bc4a4) at ../libva/va/va.c:714
#7  0x00007f802fc0ced7 in ??? () at /usr/lib/firefox/libxul.so
#8  0x00007f802fcaad62 in ??? () at /usr/lib/firefox/libxul.so
#9  0x00007f802fcad4ce in ??? () at /usr/lib/firefox/libxul.so
#10 0x00007f802fcaeffc in ??? () at /usr/lib/firefox/libxul.so
#11 0x00007f802fcaef14 in ??? () at /usr/lib/firefox/libxul.so
#12 0x00007f802fcaa539 in ??? () at /usr/lib/firefox/libxul.so
#13 0x00007f802d5a5135 in ??? () at /usr/lib/firefox/libxul.so
#14 0x00007f802d5a4b1e in ??? () at /usr/lib/firefox/libxul.so
#15 0x00007f802d5a3b9f in ??? () at /usr/lib/firefox/libxul.so
#16 0x00007f802d5a3691 in ??? () at /usr/lib/firefox/libxul.so
#17 0x00007f802fadc74c in ??? () at /usr/lib/firefox/libxul.so
#18 0x00007f802d6786ff in ??? () at /usr/lib/firefox/libxul.so
#19 0x00007f802d676f0b in ??? () at /usr/lib/firefox/libxul.so
#20 0x00007f802c085203 in ??? () at /usr/lib/firefox/libxul.so
#21 0x00007f802bc878f5 in ??? () at /usr/lib/firefox/libxul.so
#22 0x00007f802bc86c87 in ??? () at /usr/lib/firefox/libxul.so
#23 0x00007f802bc8f745 in ??? () at /usr/lib/firefox/libxul.so
#24 0x00007f802bc8e0c3 in ??? () at /usr/lib/firefox/libxul.so
#25 0x00007f802d1e33e1 in ??? () at /usr/lib/firefox/libxul.so
#26 0x00007f802d1e2eeb in ??? () at /usr/lib/firefox/libxul.so
#27 0x00007f8040532f4b in _pt_root (arg=0x7f802120be80) at pthreads/ptthread.c:198
#28 0x0000556f4c8252d9 in ??? ()
#29 0x00007f803fe980a2 in start_thread (arg=<optimized out>) at pthread_create.c:454
#30 0x00007f803ff2080c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Logs from the driver:

[NVD_DRIVER_TRACE]      10895.519130513 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:4759 ensureGlobalCodecFunctionsLoaded Global codec functions ready (cu=0x7f8025324800 cv=0x7f8037df8280 nv=0x7f8037d17900)
[NVD_DRIVER_TRACE]      10895.519163694 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:9571       __vaDriverInit_1_0 Initialising NVIDIA VA-API Driver
[NVD_DRIVER_TRACE]      10895.519167984 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:9580       __vaDriverInit_1_0 Got DRM FD: 1 26
[NVD_DRIVER_TRACE]      10895.519175024 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:9592       __vaDriverInit_1_0 Now have 0 (0 max) instances
[NVD_DRIVER_TRACE]      10895.519179184 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:9637       __vaDriverInit_1_0 Selecting Direct backend
[NVD_DRIVER_TRACE]      10895.545551774 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/direct/nv-driver.c: 364            init_nvdriver Initing nvdriver...
[NVD_DRIVER_TRACE]      10895.545590814 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/direct/nv-driver.c: 382            init_nvdriver NVIDIA kernel driver version: 610.57.04, major version: 610, minor version: 57
[NVD_DRIVER_TRACE]      10895.545596494 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/direct/nv-driver.c: 389            init_nvdriver Got dev info: 2d00 1 2 6
[NVD_DRIVER_TRACE]      10895.546287207 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/direct/nv-driver.c: 358       query_ram_location Detected discrete GPU (IS_ZERO_FB=0, RAM_LOCATION=0x0)
[NVD_DRIVER_TRACE]      10895.546304037 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/direct/direct-export-buf.c:1091      direct_initExporter external import policy probe: gpu_id=0 dmabuf_sharing_supported=no allow_direct_dmabuf=no prefer_direct_dmabuf_env=(null) disable_direct_dmabuf_env=(null) prefer_gpu_copy=no force_gpu_copy_raw=no force_gpu_copy_nvkms=no mode=startup_capability_probe
[NVD_DRIVER_TRACE]      10895.657225343 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:2384 negotiateNvencApiVersion NVENC API negotiation (encode probe): header=13.1 max=13.1 selected=13.1
[NVD_DRIVER_TRACE]      10895.657265784 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:4796       probeEncodeSupport encode capability probe persistent cache hit: /home/thesword/.cache/nvidia-vaapi-driver/encode-probe-v1-devuuid85142072d8576520e4b78d06bf8e4a0d-gpu0-api16777229-lib9bdd6f0f53e2c5af-289560-1785778533.cache
[NVD_DRIVER_TRACE]      10895.657272894 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:9686       __vaDriverInit_1_0 driver init encode support: h264=1 h26410=0 h264444=1 hevc=1 hevc10=1 hevc422=0 hevc444=1 av1=0 av110=0
[NVD_DRIVER_TRACE]      10895.812362542 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:5653           nvCreateConfig created encode config id=2 profile=6 supportedRtMask=0x1 requestedRtMask=0x1 selectedRt=0x1 surfaceFormat=0 chroma=1 bitDepth=8 rc=0x2
[NVD_DRIVER_TRACE]      10895.826303285 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:5653           nvCreateConfig created encode config id=4 profile=7 supportedRtMask=0x5 requestedRtMask=0x5 selectedRt=0x1 surfaceFormat=0 chroma=1 bitDepth=8 rc=0x2
[NVD_DRIVER_TRACE]      10895.840369761 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:5653           nvCreateConfig created encode config id=6 profile=13 supportedRtMask=0x1 requestedRtMask=0x1 selectedRt=0x1 surfaceFormat=0 chroma=1 bitDepth=8 rc=0x2
[NVD_DRIVER_TRACE]      10895.854687311 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:5653           nvCreateConfig created encode config id=8 profile=17 supportedRtMask=0x1 requestedRtMask=0x1 selectedRt=0x1 surfaceFormat=0 chroma=1 bitDepth=8 rc=0x2
[NVD_DRIVER_TRACE]      10895.881931346 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:5653           nvCreateConfig created encode config id=12 profile=18 supportedRtMask=0x100 requestedRtMask=0x100 selectedRt=0x100 surfaceFormat=1 chroma=1 bitDepth=10 rc=0x2
[NVD_DRIVER_TRACE]      10895.902244715 [47621-48161] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:9432              nvTerminate Terminating 0x7f80253db220
[NVD_DRIVER_TRACE]      10895.902660693 [47621-48161] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:9448              nvTerminate Now have 0 (0 max) instances
[NVD_DRIVER_TRACE]      10895.920236902 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:9571       __vaDriverInit_1_0 Initialising NVIDIA VA-API Driver
[NVD_DRIVER_TRACE]      10895.920247262 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:9580       __vaDriverInit_1_0 Got DRM FD: 1 39
[NVD_DRIVER_TRACE]      10895.920255472 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:9592       __vaDriverInit_1_0 Now have 0 (0 max) instances
[NVD_DRIVER_TRACE]      10895.920259902 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:9637       __vaDriverInit_1_0 Selecting Direct backend
[NVD_DRIVER_TRACE]      10895.920270633 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/direct/nv-driver.c: 364            init_nvdriver Initing nvdriver...
[NVD_DRIVER_TRACE]      10895.920285723 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/direct/nv-driver.c: 382            init_nvdriver NVIDIA kernel driver version: 610.57.04, major version: 610, minor version: 57
[NVD_DRIVER_TRACE]      10895.920290673 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/direct/nv-driver.c: 389            init_nvdriver Got dev info: 2d00 1 2 6
[NVD_DRIVER_TRACE]      10895.921761210 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/direct/nv-driver.c: 358       query_ram_location Detected discrete GPU (IS_ZERO_FB=0, RAM_LOCATION=0x0)
[NVD_DRIVER_TRACE]      10895.922068275 [47621-48155] ../nvidia-vaapi-driver-0.0.17/src/direct/direct-export-buf.c:1091      direct_initExporter external import policy probe: gpu_id=0 dmabuf_sharing_supported=no allow_direct_dmabuf=no prefer_direct_dmabuf_env=(null) disable_direct_dmabuf_env=(null) prefer_gpu_copy=no force_gpu_copy_raw=no force_gpu_copy_nvkms=no mode=startup_capability_probe
[NVD_DRIVER_TRACE]      10896.056422727 [47621-48161] ../nvidia-vaapi-driver-0.0.17/src/vabackend.c:9469              nvTerminate Released global CUDA/NVDEC/NVENC function tables after last instance shutdown
[GFX1-]: VideoBridgeParent receives IPC close with reason=AbnormalShutdown

I tested with the following URL: https://webrtc.github.io/test-pages/src/canvas-capture/

@ntoskrnl7

ntoskrnl7 commented Aug 24, 2026

Copy link
Copy Markdown
Author

@thesword53

I attempted to reproduce this on the current PR branch (0baaae3) with:

  • Firefox 147.0.3
  • NVIDIA RTX A2000
  • NVIDIA driver 570.211.01
  • Ubuntu 25.04
  • FFmpeg 6.1.1
  • NVD_BACKEND=direct

I used an H.264-only WebRTC loopback test and repeated 100 create/negotiate/teardown cycles. All 100 cycles completed and I did not observe an RDD crash or a minidump.

With the system FFmpeg build, Firefox selected h264_nvenc, so that did not exercise the VA-API encoder path. I then used a local FFmpeg build with h264_nvenc, V4L2 M2M, and libx264 disabled, leaving h264_vaapi as the only H.264 encoder.

Firefox then selected h264_vaapi, but the encoder failed during initialization:

Using preferred hardware codec h264_vaapi
failed to open h264_vaapi avcodec: Invalid argument
Fail to init encoder. Falling back to SW

The NVIDIA VA-API driver was nevertheless loaded. I observed repeated vaInitialize, probeEncodeSupport cache hits, and working H.264 VA-API decoding. However, no VA-API encode context/session was created (encodeSessionInitialized=0).

I also did not observe the teardown/init overlap shown in the original report. In my runs, nvTerminate finished releasing the global CUDA/NVDEC/NVENC tables before the next __vaDriverInit_1_0 started. Therefore, this is a “not reproduced in my environment” result and does not rule out the reported race.

The NVIDIA driver version differs significantly: my system uses 570.211.01, while the crash report uses 610.57.04.

Could you share the Firefox version, FFmpeg version/encoder list, relevant Firefox preferences, and launch environment used for the crashing run? That would help determine why your Firefox reached a different timing/path.

@thesword53

thesword53 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

I'm using Firefox 154.0. I applied the bellow patch on top of our merge request and I have no more crash:

diff --git a/src/vabackend.c b/src/vabackend.c
index 7619fed..cb453af 100644
--- a/src/vabackend.c
+++ b/src/vabackend.c
@@ -4118,6 +4118,7 @@ static bool destroyEncodeSession(NVContext *nvCtx) {
                 LOG("destroyEncodeSession context_id=%d begin recycleCudaContextAfterLastSession", (int) nvCtx->contextId);
                 recycleCudaContextAfterLastSession(drv, "all_sessions_destroyed");
                 LOG("destroyEncodeSession context_id=%d done recycleCudaContextAfterLastSession", (int) nvCtx->contextId);
+/*
                 LOG(
                     "destroyEncodeSession context_id=%d begin reloadGlobalCodecFunctionsAfterLastSession",
                     (int) nvCtx->contextId
@@ -4127,6 +4128,7 @@ static bool destroyEncodeSession(NVContext *nvCtx) {
                     "destroyEncodeSession context_id=%d done reloadGlobalCodecFunctionsAfterLastSession",
                     (int) nvCtx->contextId
                 );
+*/
             }
             pthread_mutex_unlock(&drv->objectCreationMutex);
         }
@@ -9465,8 +9467,8 @@ static VAStatus nvTerminate( VADriverContextP ctx )
             );
         } else {
             resetProcessTransientState();
-            releaseGlobalCodecFunctions();
-            LOG("Released global CUDA/NVDEC/NVENC function tables after last instance shutdown");
+            // releaseGlobalCodecFunctions();
+            // LOG("Released global CUDA/NVDEC/NVENC function tables after last instance shutdown");
         }
     }
 

The issue is releaseGlobalCodecFunctionsLocked writes to nv, cv, and cu with a mutex (global_codec_loader_mutex) but there is no mutex around those variable when they are read.

Edit: This issue can't be reproduced with ffmpeg alone as it only uses one thread for VAAPI.

@ntoskrnl7

Copy link
Copy Markdown
Author

@thesword53 Thanks, your diagnosis was correct.

I updated the PR so the process-global CUDA/NVDEC/NVENC function tables are no longer released or reloaded during normal VA driver instance teardown. They remain valid until the shared library is unloaded.

I tested the updated branch with Firefox 154 using 100 H.264 WebRTC create/negotiate/teardown cycles and observed no RDD crash or minidump. The local VA-API runtime matrix also passed all 21 encode/decode cases, and both GCC and Clang 15 CI jobs pass with -Werror.

My Firefox test still does not fully reproduce your original NVIDIA 610.57.04 environment, so could you please retest the current PR head (8518275) on your system and confirm whether the crash is resolved?

@thesword53

Copy link
Copy Markdown
Contributor

Thanks for the new patches. Encoding is now working without crash with Firefox.
I also noticed with your patch the driver reports VAProfileHEVCMain444, VAProfileHEVCMain444_10 for VAEntrypointVLD (decoding) on my laptop with a GTX 1060 but this GPU only supports HEVC YUV444 for video encoding (VAEntrypointEncSlice) and not video decoding according to https://developer.nvidia.com/video-encode-decode-support-matrix.

@ntoskrnl7

Copy link
Copy Markdown
Author

@thesword53 Thanks for confirming that the Firefox crash is fixed.

You were also correct about the HEVC 4:4:4 decode profiles. The driver was combining NVDEC and NVENC profiles, then incorrectly advertising VAEntrypointVLD for profiles supported only by NVENC.

I pushed a fix in 3f98128 that keeps the NVDEC profile list separate. VAEntrypointVLD, config attribute queries, and decode config creation now require actual NVDEC support reported by cuvidGetDecoderCaps().

GCC and Clang 15 CI pass, and the local VA-API runtime matrix passes all 21 cases. Could you please retest the current PR on your GTX 1060 and confirm that VAProfileHEVCMain444 and VAProfileHEVCMain444_10 are now only reported for VAEntrypointEncSlice, not VAEntrypointVLD?

@thesword53

Copy link
Copy Markdown
Contributor

@ntoskrnl7 With your last changes I have VAProfileHEVCMain444, VAProfileHEVCMain444_10 reported only with VAEntrypointEncSlice using my GTX 1060. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants