feat: Implement zero-copy hardware-accelerated hybrid FFmpeg export pipeline#443
Open
askadityapandey wants to merge 4 commits intosiddharthvaddem:mainfrom
Open
feat: Implement zero-copy hardware-accelerated hybrid FFmpeg export pipeline#443askadityapandey wants to merge 4 commits intosiddharthvaddem:mainfrom
askadityapandey wants to merge 4 commits intosiddharthvaddem:mainfrom
Conversation
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdded a native FFmpeg export pipeline: FFmpeg discovery/probing and packaging, new IPC handlers and preload bindings for streaming encode sessions, a download script, worker-based OffscreenCanvas compositing, a new FFmpegExporter coordinating decode→render→encode→FFmpeg, and platform-specific VideoFrame handling. No public API removals. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a897240fa0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…, and stale save handles
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@electron-builder.json5`:
- Around line 28-31: The packaging filter currently only keeps "ffmpeg.exe" and
"ffprobe.exe", which omits the POSIX binaries referenced by
electron/ffmpeg/ffmpegManager.ts (it resolves
process.resourcesPath/ffmpeg/ffmpeg on macOS/Linux); update the
electron-builder.json5 file's artifact filter (the block with "from":
"vendor/ffmpeg", "to": "ffmpeg") to include non-.exe names as well (e.g.,
include "ffmpeg" and "ffprobe" or use wildcard patterns like "ffmpeg*" and
"ffprobe*") so darwin/linux builds bundle the expected binaries alongside
Windows .exe files.
In `@electron/preload.ts`:
- Around line 146-169: The FFmpeg IPC methods (ffmpegGetCapabilities,
ffmpegExportStart, ffmpegExportFrame, ffmpegExportFinish, ffmpegExportCancel)
are being exposed on the global bridge (window.electronAPI) and must be
restricted to the trusted editor window; either remove them from the global
preload or gate them by only attaching them when creating the trusted editor
preload, and additionally validate the sender in the corresponding main-process
handlers to reject requests from untrusted windows (check where the
KalturaBrowseDialog and windows creation logic in windows.ts construct/load the
remote Kaltura window and ensure that remote windows do not receive the ffmpeg
APIs or that the main ipc handlers perform origin/window identity checks before
accepting session start/frame/finish/cancel calls).
In `@scripts/download-ffmpeg.mjs`:
- Around line 23-43: SOURCES currently contains unverified third‑party archives;
add a per‑platform SHA‑256 fingerprint field to each SOURCES entry and verify
the downloaded archive before extraction/execution by computing its SHA‑256 (use
Node's createHash) and comparing to the expected value; if the hash mismatches,
abort with a clear error instead of extracting or running the binary. Update the
download/extract/run flow that uses SOURCES (and the code around the existing
download lines) to call a small verifier function (e.g., sha256File) after
download and before extraction or the -version exec, and ensure failures stop
the process.
- Around line 7-8: The download script writes binaries to
vendor/ffmpeg/<platform>/..., but electron/ffmpeg/ffmpegManager.ts expects
vendor/ffmpeg/<binaryName> in dev and process.resourcesPath/ffmpeg/<binaryName>
when packaged; update scripts/download-ffmpeg.mjs to place the downloaded binary
(binaryName) directly under vendor/ffmpeg/ (no platform subdirectory) and also
write/copy the file into the packaged layout (process.resourcesPath/ffmpeg/) or
create the same flat vendor/ffmpeg/<binaryName> path and a duplicate into
resourcesPath/ffmpeg so ffmpegManager.ts can find it, adjusting any
platform-specific filename logic in the script to match the binaryName lookup
used by ffmpegManager.ts.
In `@src/components/video-editor/VideoEditor.tsx`:
- Around line 1514-1517: The current cast of result to access ffmpegResult hides
the exporter contract; update the FFmpegExporter.export() return type to a
discriminated union (e.g., { type: 'blob'; blob: Blob } | { type: 'path'; path:
string; canceled?: boolean }) and change VideoEditor to switch on the
discriminant instead of casting (replace usages of ffmpegResult and the current
result.success check with a safe pattern match on the union variants), updating
any handling of result.success to the new union fields so TypeScript enforces
compatibility between FFmpegExporter and VideoEditor.
- Around line 1471-1512: The current branch uses FFmpegExporter whenever
ffmpegCheck.available is true, but FFmpegExporter lacks webcam support and will
drop webcam overlays; add a guard before creating FFmpegExporter to detect when
webcamVideoPath (or any webcam-related props like
webcamLayoutPreset/webcamSizePreset) is present and, in that case, skip the
FFmpeg path and use the existing VideoExporter fallback instead; ensure
exporterRef.current is set to the VideoExporter instance and call its export()
so webcam exports retain parity until FFmpegExporter implements webcam support.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 8fdb1878-e3a0-4b60-84fa-4498ae5a4fec
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (15)
.gitignoreelectron-builder.json5electron/electron-env.d.tselectron/ffmpeg/ffmpegManager.tselectron/ipc/handlers.tselectron/preload.tsscripts/download-ffmpeg.mjssrc/components/video-editor/VideoEditor.tsxsrc/lib/exporter/ffmpegExporter.tssrc/lib/exporter/frameRenderer.tssrc/lib/exporter/frameRendererWorker.tssrc/lib/exporter/index.tssrc/lib/exporter/types.tssrc/lib/exporter/videoExporter.tssrc/vite-env.d.ts
👮 Files not reviewed due to content moderation or server errors (7)
- src/lib/exporter/frameRenderer.ts
- electron/electron-env.d.ts
- electron/ipc/handlers.ts
- src/lib/exporter/frameRendererWorker.ts
- src/vite-env.d.ts
- src/lib/exporter/ffmpegExporter.ts
- electron/ffmpeg/ffmpegManager.ts
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
src/lib/exporter/ffmpegExporter.ts (1)
119-119: nit: TODO left in codethe webcam support is marked as TODO here, but the code below (lines 210-246) still references
webcamDecoderandAsyncVideoFrameQueuewhich don't exist. see my other comment about the undefined variables — you'll want to either implement this or remove the dead code references.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lib/exporter/ffmpegExporter.ts` at line 119, The TODO for webcam support (webcamSize) left a mismatch: the file references webcamDecoder and AsyncVideoFrameQueue which are undefined; either fully implement webcam support (add/ import an AsyncVideoFrameQueue class, implement webcamDecoder creation and handling, and wire webcamSize into that flow) or remove the dead webcam branches entirely. Locate usages of webcamDecoder and AsyncVideoFrameQueue in ffmpegExporter.ts and either replace them with the existing video decoding pipeline, add the missing imports/implementations, and ensure tests/type checks pass, or delete/guard those blocks and the webcamSize TODO so no undefined symbols remain.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@electron/ipc/handlers.ts`:
- Around line 1058-1066: The code accumulates stderrOutput locally in the
ffmpegProcess.stderr handler but never surfaces it on failure; update the
session type (e.g., add stderrOutput string field) and set session.stderrOutput
instead of a local variable inside the ffmpegProcess.stderr?.on("data", ...)
callback, ensure any cleanup/finish handler that currently throws or logs
"FFmpeg exited with code X" (the export finish/exit handler) includes
session.stderrOutput in the error response/log, and make sure
session.stderrOutput is initialized when creating the session so it is available
for later error reporting.
In `@src/lib/exporter/ffmpegExporter.ts`:
- Around line 259-270: The code declares const timestamp twice in the same scope
(first near the top of the snippet and again before the encoder path), causing a
redeclaration error; remove the unused earlier declaration (the one assigned
from frameIndex * frameDurationUs before webcamFrame dequeue) so only the later
timestamp declaration remains and all uses reference that single variable (look
for the variables frameIndex, frameDurationUs and the two const timestamp
declarations in ffmpegExporter.ts).
- Around line 209-246: The block that references AsyncVideoFrameQueue and
webcamDecoder (creating webcamFrameQueue, webcamDecodePromise, stopWebcamDecode,
webcamDecodeError) uses undefined symbols and will throw ReferenceError; remove
this incomplete webcam handling code entirely or replace it with a safe stub:
eliminate creation/usage of AsyncVideoFrameQueue, webcamDecoder,
webcamFrameQueue, webcamDecodePromise and stopWebcamDecode, and instead ensure
any config.webcamVideoUrl path either no-ops or returns a resolved/null promise
so runtime won't crash; if stubbing, create a minimal local stub (e.g., a
null-safe object or async function) that behaves like the queue/decoder
interfaces used here so other code can continue without importing non-existent
types.
---
Nitpick comments:
In `@src/lib/exporter/ffmpegExporter.ts`:
- Line 119: The TODO for webcam support (webcamSize) left a mismatch: the file
references webcamDecoder and AsyncVideoFrameQueue which are undefined; either
fully implement webcam support (add/ import an AsyncVideoFrameQueue class,
implement webcamDecoder creation and handling, and wire webcamSize into that
flow) or remove the dead webcam branches entirely. Locate usages of
webcamDecoder and AsyncVideoFrameQueue in ffmpegExporter.ts and either replace
them with the existing video decoding pipeline, add the missing
imports/implementations, and ensure tests/type checks pass, or delete/guard
those blocks and the webcamSize TODO so no undefined symbols remain.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 8083e8b8-c0b9-421d-a7ff-1009634f8e12
📒 Files selected for processing (3)
electron-builder.json5electron/ipc/handlers.tssrc/lib/exporter/ffmpegExporter.ts
✅ Files skipped from review due to trivial changes (1)
- electron-builder.json5
…ression, enforce typed binary outputs
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (5)
electron/ipc/handlers.ts (2)
972-982: session Map never cleaned up on unexpected process exitif the main process crashes or restarts, stale sessions persist in memory (not a real leak since it's in-memory), but if FFmpeg somehow exits unexpectedly without going through finish/cancel, the session entry could linger. might be worth adding a timeout or cleanup on the
exitPromiseresolution outside of finish/cancel.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@electron/ipc/handlers.ts` around lines 972 - 982, The ffmpegSessions Map can retain entries if a ChildProcess exits unexpectedly; modify the session creation logic where exitPromise is set (for entries stored in ffmpegSessions) to attach a .then/.finally handler that removes the corresponding Map entry when exitPromise resolves, and also consider setting a per-session watchdog timeout (e.g., using setTimeout at creation) that will remove the session after a configurable grace period; update the cleanup to run whether the process finished normally or via unexpected exit so that finish/cancel are not the only paths that delete the ffmpegSessions entry.
1219-1224: consider awaiting thefs.unlinkin cancel pathyou're fire-and-forgetting the temp file deletion. if the file is still being written by FFmpeg when you SIGKILL it, the unlink might race. probably fine in practice since SIGKILL is synchronous, but awaiting wouldn't hurt:
💡 Proposed tweak
session.finished = true; session.process.kill("SIGKILL"); ffmpegSessions.delete(sessionId); // Clean up temp file -fs.unlink(session.outputPath).catch(() => {}); +await fs.unlink(session.outputPath).catch(() => {});🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@electron/ipc/handlers.ts` around lines 1219 - 1224, In the FFmpeg cancellation path update the fire-and-forget temp-file delete to await the unlink so deletion completes (or errors are handled) after you mark session.finished, kill the process and remove it from ffmpegSessions; specifically locate the block that sets session.finished = true, calls session.process.kill("SIGKILL"), and ffmpegSessions.delete(sessionId) and change the fs.unlink(session.outputPath).catch(() => {}) call to an awaited operation with proper error handling (e.g., await fs.unlink(session.outputPath) inside a try/catch) so any race or unlink failure is observed and handled.src/lib/exporter/ffmpegExporter.ts (3)
375-380:cancel()callscleanup()butcancelFFmpeg()is asyncyou're calling
void this.cancelFFmpeg()(fire-and-forget) then immediately callingthis.cleanup(). if cleanup destroys the decoder/renderer while cancelFFmpeg is still awaiting the IPC response, you might get races. probably fine sincecleanup()doesn't touchsessionId, but the ordering is a bit funky.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lib/exporter/ffmpegExporter.ts` around lines 375 - 380, The cancel() method fires cancelFFmpeg() without awaiting it then immediately calls cleanup(), which risks race conditions with async IPC; modify cancel() to await this.cancelFFmpeg() before calling this.cleanup() (i.e., make cancel() async and await this.cancelFFmpeg()), while still calling this.streamingDecoder?.cancel() immediately; update any callers/tests to handle the async signature change (or alternatively explicitly chain the promise: this.cancelFFmpeg().finally(() => this.cleanup())) so cleanup runs only after cancelFFmpeg completes.
1-10: file header comment slightly misleadingthe comment says "pipes raw RGBA frames to FFmpeg" but you're actually piping hardware-encoded H.264 via WebCodecs VideoEncoder. the whole point is avoiding raw RGBA! might want to update:
📝 Suggested update
/** - * FFmpeg-based video exporter that pipes raw RGBA frames to an FFmpeg child process - * running in the Electron main process with hardware-accelerated encoding. + * FFmpeg-based video exporter that uses WebCodecs VideoEncoder to hardware-encode + * frames as H.264 and pipes the encoded stream to an FFmpeg child process for muxing. * - * This replaces the slow WebCodecs VideoEncoder path on Windows, providing - * 5-20x faster exports by leveraging NVENC/QSV/AMF hardware encoders. + * This uses Chrome's hardware encoder (via WebCodecs) and FFmpeg just for muxing, + * avoiding costly raw RGBA frame transfers over IPC.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lib/exporter/ffmpegExporter.ts` around lines 1 - 10, Update the file header comment in src/lib/exporter/ffmpegExporter.ts to accurately describe behavior: replace the claim that the module "pipes raw RGBA frames to an FFmpeg child process" with a description that it sends hardware-encoded H.264 (via the WebCodecs VideoEncoder) to an FFmpeg child process in the Electron main process, and note that it avoids raw RGBA piping and falls back to libx264 when no GPU encoder is available; reference the module name (ffmpegExporter / FFmpeg-based video exporter) in the revised comment so readers can quickly find the implementation.
280-286: keyframe interval hardcoded to 150 framesat 30fps that's 5 seconds, at 60fps it's 2.5 seconds — both reasonable. but the comment in the PR mentions high-res 60fps exports, so you might want to make this configurable or tie it to framerate. not blocking, just noting.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lib/exporter/ffmpegExporter.ts` around lines 280 - 286, The key-frame interval is hardcoded to 150 in the vidEncoder.encode call which makes keyframes vary by export framerate (frameIndex % 150 === 0); change this to a configurable or framerate-derived value (e.g., compute keyframeInterval = Math.round(framerate * desiredKeyframeSeconds) or read from an export config) and use vidEncoder.encode(exportFrame, { keyFrame: frameIndex % keyframeInterval === 0 }); update any export API/option parsing to accept the new setting and ensure frameIndex, exportFrame and vidEncoder.encode usage remain consistent.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@electron/ipc/handlers.ts`:
- Around line 1109-1135: The renderer currently ignores the backpressure flag
returned by the "ffmpeg-export-frame" ipc handler; update the renderer's
ffmpegExporter logic (where it calls ipcRenderer.invoke("ffmpeg-export-frame",
...), e.g., in the function that enqueues/sends frames in ffmpegExporter.ts) to
read the returned object, inspect the backpressure property, and when
backpressure === true, await a short delay (or yield via a sleep/pause helper)
before attempting to encode/send the next frame so the encode loop respects
FFmpeg stdin backpressure and avoids overwhelming the queue; keep existing
encodeQueueSize logic but add this backpressure-driven backoff to throttle more
tightly when needed.
In `@src/lib/exporter/ffmpegExporter.ts`:
- Around line 358-362: The returned object uses a non-null assertion on
finishResult.path which can be undefined; change the code in ffmpegExporter.ts
to validate finishResult.path before returning (e.g., check finishResult.path
!== undefined/null) and handle the missing path by either returning success:
false with an error message or throwing a clear error so callers never get an
undefined path; reference finishResult.path and the function that returns the
object to locate where to add this guard and error handling.
- Around line 175-200: The VideoEncoder error handler currently only logs errors
and doesn't stop the export; update the VideoEncoder error callback in the
VideoEncoder creation (the error: (e) => { ... } block) to set an internal flag
(e.g., this.encoderError = true or this._encoderError) and store the error
details, then ensure the decode/render loop that sends frames via
window.electronAPI.ffmpegExportFrame(this.sessionId!, buffer) checks
this.encoderError alongside this.cancelled and aborts the export path
(reject/push a failure state, stop/close the encoder, and perform cleanup) so a
hardware encoder failure fails the export instead of continuing to send frames.
- Around line 191-195: The ffmpegExportFrame failure is currently only logged
and the loop continues, which can produce corrupted output; update the send-loop
in ffmpegExporter.ts (around the call to window.electronAPI.ffmpegExportFrame
and the frameResult handling) to track consecutive/send failures (e.g.,
failureCount variable), increment it when frameResult.success is false, and when
it exceeds a small threshold (e.g., 1-3) abort the export: stop sending further
frames, set a failure flag, break/return from the sending function, and invoke
any existing cleanup/abort routine or emit an error to the caller (ensure you
propagate the failure by rejecting the promise or throwing) so callers can
perform cleanup instead of silently continuing.
---
Nitpick comments:
In `@electron/ipc/handlers.ts`:
- Around line 972-982: The ffmpegSessions Map can retain entries if a
ChildProcess exits unexpectedly; modify the session creation logic where
exitPromise is set (for entries stored in ffmpegSessions) to attach a
.then/.finally handler that removes the corresponding Map entry when exitPromise
resolves, and also consider setting a per-session watchdog timeout (e.g., using
setTimeout at creation) that will remove the session after a configurable grace
period; update the cleanup to run whether the process finished normally or via
unexpected exit so that finish/cancel are not the only paths that delete the
ffmpegSessions entry.
- Around line 1219-1224: In the FFmpeg cancellation path update the
fire-and-forget temp-file delete to await the unlink so deletion completes (or
errors are handled) after you mark session.finished, kill the process and remove
it from ffmpegSessions; specifically locate the block that sets session.finished
= true, calls session.process.kill("SIGKILL"), and
ffmpegSessions.delete(sessionId) and change the
fs.unlink(session.outputPath).catch(() => {}) call to an awaited operation with
proper error handling (e.g., await fs.unlink(session.outputPath) inside a
try/catch) so any race or unlink failure is observed and handled.
In `@src/lib/exporter/ffmpegExporter.ts`:
- Around line 375-380: The cancel() method fires cancelFFmpeg() without awaiting
it then immediately calls cleanup(), which risks race conditions with async IPC;
modify cancel() to await this.cancelFFmpeg() before calling this.cleanup()
(i.e., make cancel() async and await this.cancelFFmpeg()), while still calling
this.streamingDecoder?.cancel() immediately; update any callers/tests to handle
the async signature change (or alternatively explicitly chain the promise:
this.cancelFFmpeg().finally(() => this.cleanup())) so cleanup runs only after
cancelFFmpeg completes.
- Around line 1-10: Update the file header comment in
src/lib/exporter/ffmpegExporter.ts to accurately describe behavior: replace the
claim that the module "pipes raw RGBA frames to an FFmpeg child process" with a
description that it sends hardware-encoded H.264 (via the WebCodecs
VideoEncoder) to an FFmpeg child process in the Electron main process, and note
that it avoids raw RGBA piping and falls back to libx264 when no GPU encoder is
available; reference the module name (ffmpegExporter / FFmpeg-based video
exporter) in the revised comment so readers can quickly find the implementation.
- Around line 280-286: The key-frame interval is hardcoded to 150 in the
vidEncoder.encode call which makes keyframes vary by export framerate
(frameIndex % 150 === 0); change this to a configurable or framerate-derived
value (e.g., compute keyframeInterval = Math.round(framerate *
desiredKeyframeSeconds) or read from an export config) and use
vidEncoder.encode(exportFrame, { keyFrame: frameIndex % keyframeInterval === 0
}); update any export API/option parsing to accept the new setting and ensure
frameIndex, exportFrame and vidEncoder.encode usage remain consistent.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 79c9e371-760d-46b0-8be6-18a81e24bf23
📒 Files selected for processing (7)
electron/ipc/handlers.tsscripts/download-ffmpeg.mjssrc/components/video-editor/VideoEditor.tsxsrc/lib/exporter/ffmpegExporter.tssrc/lib/exporter/gifExporter.tssrc/lib/exporter/types.tssrc/lib/exporter/videoExporter.ts
✅ Files skipped from review due to trivial changes (1)
- scripts/download-ffmpeg.mjs
🚧 Files skipped from review as they are similar to previous changes (3)
- src/lib/exporter/types.ts
- src/lib/exporter/videoExporter.ts
- src/components/video-editor/VideoEditor.tsx
… to avoid output corruption
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces a radically optimized, drop-in replacement export pipeline (
FFmpegExporter) that marries Chrome’s nativeWebCodecsGPU compression with a nativeFFmpegbackend. By completely eliminating the massive CPU/IPC stalls caused by transferring raw RGBA pixels frame-by-frame, export times for high-resolution 60fps clips drop from ~10+ minutes to ~3 minutes while reducing overall memory consumption.A seamless automatic fallback to the existing
VideoExporteris fully preserved, ensuring zero disruption for environments where FFmpeg is unavailable.Motivation
Previously, extracting frames from the Canvas via
getImageData()forced a synchronous GPU-to-CPU stall, generating massive ~8.3MB raw uncompressed frames. Attempting to serialize ~500MB/s of data across the Electron IPC bridge to the main process saturated V8 garbage collection, causing severe frame rate throttling (~1 FPS exports) and potential out-of-memory crashes on lengthy timelines.The Hybrid Fix:
VideoEncoder(avc1) withprefer-hardwareto instantly encode theHTMLCanvasElementdirectly on the GPU (bypassing CPU readbacks).~20KBH.264 annex-b chunks across IPC.ffmpegchild process natively multiplexes the H.264 pipe (-c:v copy) alongside the pristine audio stream directly to the disk, bypassing in-memory media muxers entirely.Type of Change
Related Issue(s)
fixes #439
Screenshots / Video
Since this is purely a backend engine transformation, there are no visual UI changes. However, you will notice a significant decrease in RAM usage and a massive acceleration in the export progress bar.
Testing
To verify the new native hardware pipeline:
node scripts/download-ffmpeg.mjslocally to fetch the platform-specific FFmpeg binary.npm run dev.[FFmpeg] Starting exportutilizing hardware encoders (e.g.,h264_nvenc,h264_qsvorh264_amf) and pulling data viapipe:0.vendor/ffmpegdirectory and export again. The app gracefully falls back to the softwareVideoExporter.Checklist
Thank you for contributing!
Summary by CodeRabbit