Windows: export always fails with "Failed to complete muxer: channel closed" — OS MPEG4 sink rejects PlaceMarker (MF_E_INVALIDTYPE)

[scream870102]

Summary

On recent Windows 11 builds, RealtimeInstantReplaySession.StopAndExportAsync always fails with:

UniEnc.UniEncException: UniEnc error (Error): Failed to complete muxer: channel closed
  at UniEnc.CallbackHelper+SimpleCallbackContext.GetResult ...
  at InstantReplay.RealtimeInstantReplaySession.MuxSegmentAsync ... (await muxer.CompleteAsync())

On package versions before the better_error_handling merge, the same condition was a hard Rust panic that killed the whole Unity process:

thread '<unnamed>' panicked at crates\unienc_windows_mf\src\mux\mod.rs:116:43:
called `Result::unwrap()` on an `Err` value: RecvError(())

Environment

• Windows 11 Home, OS build 10.0.26200 (mfmp4srcsnk.dll version 10.0.26100.8457)
• Unity 6 (Editor, win64), InstantReplay v1.6.6 (release branch head 6c5b36a)
• Video encoder MFT: NVIDIA H.264 Encoder MFT; audio: Microsoft AAC Audio Encoder MFT

Root cause

IMFStreamSink::PlaceMarker on the OS-provided MPEG4 media sink (MFCreateMPEG4MediaSink) returns MF_E_INVALIDTYPE (0xC00D36BD) for every marker type and every argument combination on this Windows build — including MFSTREAMSINK_MARKER_ENDOFSEGMENT, which unienc_windows_mf::mux::Stream's event loop calls when its sample channel closes:

MEStreamSinkRequestSample => {
    if let Some(sample) = sample_rx.recv().await {
        unsafe { stream_cap.ProcessSample(&*sample)? };
    } else {
        unsafe { stream_cap.PlaceMarker(MFSTREAMSINK_MARKER_ENDOFSEGMENT, ptr::null(), ptr::null())? }; // <- fails
        if let Some(finish_tx) = finish_tx.take() { finish_tx.send(())... };
    }
}

Consequences:

1. The ? kills the spawned event-loop task silently (its Result is discarded), so the real HRESULT never surfaces anywhere.
2. finish_tx is dropped without sending, so the muxer task's video_finish_rx.await? / audio_finish_rx.await? fails with RecvError → the user sees only "Failed to complete muxer: channel closed".
3. BeginFinalize never runs → the output file has ftyp + fully written mdat (all samples were accepted) but no moov.
4. sink.Shutdown() never runs → the output file handle leaks (the file stays locked by the process until exit).

Standalone repro (no Unity, no unienc)

A ~200-line C++ program that mirrors the muxer flow (MFCreateMPEG4MediaSink → set/start presentation clock → pump stream sink events → PlaceMarker on MEStreamSinkRequestSample) reproduces it directly:

[audio] event 305 (MEStreamSinkRequestSample)
[audio] ProcessSample #0 -> 0x00000000
[audio] ProcessSample #1 -> 0x00000000
[audio] PlaceMarker(ENDOFSEGMENT, NULL, NULL)   -> 0xC00D36BD  MF_E_INVALIDTYPE
[audio] PlaceMarker(ENDOFSEGMENT, &empty, &empty)-> 0xC00D36BD
[audio] PlaceMarker(DEFAULT, NULL, NULL)         -> 0xC00D36BD
[audio] PlaceMarker(TICK, NULL, NULL)            -> 0xC00D36BD
[audio] MEStreamSinkMarker never arrived

Markers fail both on empty streams and on streams that have successfully processed samples (ProcessSample itself returns S_OK). Full repro source is attached at the bottom.

This looks like a behavior change/regression in mfmp4srcsnk.dll (10.0.26100.8457); we found no public reports yet. On machines with this build every InstantReplay Windows export fails; before the error-propagation fix it crashed the entire player/editor process.

What we ruled out

• v1.6.6 (release head 6c5b36a) and branch fix/mux-export-error-handling (PR Fix error handling in realtime session export #149) both fail identically — the C#-side paths those change are never hit (all pushes and FinishVideo/FinishAudio succeed; only CompleteAsync fails).
• Not related to pause/resume, Video/AudioLagAdjustmentThreshold, or the audio sample provider (reproduced with a custom FMOD provider, a synthetic silence provider, and a never-firing provider).
• Encoders are fine: the leaked output files contain a fully written mdat with both H.264 and AAC data.

Suggested fixes in unienc

Even if this is ultimately a Windows regression, unienc can be made resilient:

1. Don't treat PlaceMarker failure as fatal — the marker is only used as a "samples consumed" signal. On failure, fall back (e.g. send the finish signal anyway, or poll until the request queue drains) so finalize still runs and the mp4 gets its moov.
2. Propagate stream-task errors — make the finish channel oneshot<Result<()>> (or log before ?) so the actual HRESULT reaches the caller instead of surfacing as an opaque channel closed.
3. Always Shutdown() the sink on the error path to avoid leaking the output file handle.

Repro source

mp4sink_repro.cpp (build: cl /EHsc mp4sink_repro.cpp)

// Minimal repro of unienc_windows_mf muxer flow:
// MFCreateMPEG4MediaSink -> clock Start(0) -> pump stream sink events ->
// PlaceMarker(ENDOFSEGMENT, NULL, NULL) on first RequestSample -> BeginFinalize.
// Prints the HRESULT of every step to find which one fails on this machine.
#include <windows.h>
#include <mfapi.h>
#include <mfidl.h>
#include <mferror.h>
#include <propvarutil.h>
#include <cstdio>

#pragma comment(lib, "mfplat.lib")
#pragma comment(lib, "mf.lib")
#pragma comment(lib, "mfuuid.lib")
#pragma comment(lib, "ole32.lib")
#pragma comment(lib, "propsys.lib")

#define CHECK(expr)                                                     \
    do {                                                                \
        HRESULT _hr = (expr);                                           \
        printf("%-60s -> 0x%08lX%s\n", #expr, _hr,                      \
               FAILED(_hr) ? "  <-- FAILED" : "");                      \
        if (FAILED(_hr)) { failed = true; }                             \
    } while (0)

static const char* EventName(MediaEventType t)
{
    switch (t) {
    case MEStreamSinkStarted: return "MEStreamSinkStarted";
    case MEStreamSinkStopped: return "MEStreamSinkStopped";
    case MEStreamSinkPaused: return "MEStreamSinkPaused";
    case MEStreamSinkRateChanged: return "MEStreamSinkRateChanged";
    case MEStreamSinkRequestSample: return "MEStreamSinkRequestSample";
    case MEStreamSinkMarker: return "MEStreamSinkMarker";
    case MEStreamSinkPrerolled: return "MEStreamSinkPrerolled";
    default: return "(other)";
    }
}

struct FinalizeCallback : IMFAsyncCallback
{
    IMFFinalizableMediaSink* sink;
    HANDLE done;
    HRESULT finalizeResult = E_PENDING;
    LONG refs = 1;

    FinalizeCallback(IMFFinalizableMediaSink* s) : sink(s)
    {
        done = CreateEventW(nullptr, TRUE, FALSE, nullptr);
    }

    STDMETHODIMP QueryInterface(REFIID riid, void** ppv) override
    {
        if (riid == IID_IUnknown || riid == IID_IMFAsyncCallback) {
            *ppv = static_cast<IMFAsyncCallback*>(this);
            AddRef();
            return S_OK;
        }
        *ppv = nullptr;
        return E_NOINTERFACE;
    }
    STDMETHODIMP_(ULONG) AddRef() override { return InterlockedIncrement(&refs); }
    STDMETHODIMP_(ULONG) Release() override
    {
        ULONG r = InterlockedDecrement(&refs);
        if (!r) delete this;
        return r;
    }
    STDMETHODIMP GetParameters(DWORD*, DWORD*) override { return E_NOTIMPL; }
    STDMETHODIMP Invoke(IMFAsyncResult* result) override
    {
        finalizeResult = sink->EndFinalize(result);
        SetEvent(done);
        return S_OK;
    }
};

// Pump events on a stream sink until RequestSample arrives, then place ENDOFSEGMENT.
// Mirrors unienc's Stream event loop with a closed sample channel.
static void DriveStream(IMFStreamSink* stream, const char* name, bool& failed)
{
    bool markerPlaced = false;
    for (int i = 0; i < 50 && !markerPlaced; i++) {
        IMFMediaEvent* ev = nullptr;
        HRESULT hr = stream->GetEvent(MF_EVENT_FLAG_NO_WAIT, &ev);
        if (hr == MF_E_NO_EVENTS_AVAILABLE) {
            Sleep(100);
            continue;
        }
        if (FAILED(hr)) {
            printf("[%s] GetEvent -> 0x%08lX  <-- FAILED\n", name, hr);
            failed = true;
            return;
        }
        MediaEventType type = MEUnknown;
        ev->GetType(&type);
        printf("[%s] event %ld (%s)\n", name, (long)type, EventName(type));
        if (type == MEStreamSinkRequestSample) {
            // Feed a few dummy samples first so the marker is placed on a stream
            // that has processed data (same as the real app when it fails).
            for (int n = 0; n < 3; n++) {
                IMFSample* sample = nullptr;
                IMFMediaBuffer* buf = nullptr;
                MFCreateSample(&sample);
                MFCreateMemoryBuffer(256, &buf);
                BYTE* p = nullptr;
                buf->Lock(&p, nullptr, nullptr);
                memset(p, 0x42, 256);
                buf->SetCurrentLength(256);
                buf->Unlock();
                sample->AddBuffer(buf);
                sample->SetSampleTime((LONGLONG)n * 333333);
                sample->SetSampleDuration(333333);
                sample->SetUINT32(MFSampleExtension_CleanPoint, 1);
                HRESULT phr = stream->ProcessSample(sample);
                printf("[%s] ProcessSample #%d -> 0x%08lX%s\n", name, n, phr,
                       FAILED(phr) ? "  <-- FAILED" : "");
                buf->Release();
                sample->Release();
                if (FAILED(phr)) break;
            }
            PROPVARIANT empty;
            PropVariantInit(&empty);
            struct Variant { const char* desc; MFSTREAMSINK_MARKER_TYPE t; const PROPVARIANT* v; const PROPVARIANT* c; };
            const Variant variants[] = {
                { "ENDOFSEGMENT, NULL, NULL", MFSTREAMSINK_MARKER_ENDOFSEGMENT, nullptr, nullptr },
                { "ENDOFSEGMENT, &empty, &empty", MFSTREAMSINK_MARKER_ENDOFSEGMENT, &empty, &empty },
                { "DEFAULT, NULL, NULL", MFSTREAMSINK_MARKER_DEFAULT, nullptr, nullptr },
                { "TICK, NULL, NULL", MFSTREAMSINK_MARKER_TICK, nullptr, nullptr },
            };
            bool anyOk = false;
            for (const auto& variant : variants) {
                HRESULT mhr = stream->PlaceMarker(variant.t, variant.v, variant.c);
                printf("[%s] PlaceMarker(%s) -> 0x%08lX%s\n",
                       name, variant.desc, mhr, FAILED(mhr) ? "  <-- FAILED" : "");
                if (SUCCEEDED(mhr)) { anyOk = true; break; }
            }
            if (!anyOk) failed = true;
            markerPlaced = true;
        }
        ev->Release();
    }
    if (!markerPlaced) {
        printf("[%s] never received MEStreamSinkRequestSample within timeout  <-- SUSPICIOUS\n", name);
        failed = true;
    }
}

// Wait for the MEStreamSinkMarker confirmation after ENDOFSEGMENT.
static void WaitMarkerEvent(IMFStreamSink* stream, const char* name)
{
    for (int i = 0; i < 30; i++) {
        IMFMediaEvent* ev = nullptr;
        HRESULT hr = stream->GetEvent(MF_EVENT_FLAG_NO_WAIT, &ev);
        if (hr == MF_E_NO_EVENTS_AVAILABLE) {
            Sleep(100);
            continue;
        }
        if (FAILED(hr)) {
            printf("[%s] GetEvent(wait marker) -> 0x%08lX\n", name, hr);
            return;
        }
        MediaEventType type = MEUnknown;
        ev->GetType(&type);
        HRESULT status = S_OK;
        ev->GetStatus(&status);
        printf("[%s] event %ld (%s), status 0x%08lX\n", name, (long)type, EventName(type), status);
        ev->Release();
        if (type == MEStreamSinkMarker) return;
    }
    printf("[%s] MEStreamSinkMarker never arrived  <-- SUSPICIOUS\n", name);
}

int main()
{
    bool failed = false;
    CHECK(CoInitializeEx(nullptr, COINIT_MULTITHREADED));
    CHECK(MFStartup(MF_VERSION, MFSTARTUP_NOSOCKET));

    IMFMediaType* videoType = nullptr;
    CHECK(MFCreateMediaType(&videoType));
    CHECK(videoType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video));
    CHECK(videoType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_H264));
    CHECK(videoType->SetUINT32(MF_MT_AVG_BITRATE, 1500000));
    CHECK(videoType->SetUINT64(MF_MT_FRAME_RATE, (30ULL << 32) | 1));
    CHECK(videoType->SetUINT64(MF_MT_FRAME_SIZE, (640ULL << 32) | 360));
    CHECK(videoType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive));
    CHECK(videoType->SetUINT32(MF_MT_MPEG2_PROFILE, 66 /* baseline */));

    // AAC-LC 48kHz stereo; user data = 12-byte HEAACWAVEINFO tail + AudioSpecificConfig 0x11 0x90
    IMFMediaType* audioType = nullptr;
    CHECK(MFCreateMediaType(&audioType));
    CHECK(audioType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Audio));
    CHECK(audioType->SetGUID(MF_MT_SUBTYPE, MFAudioFormat_AAC));
    CHECK(audioType->SetUINT32(MF_MT_AUDIO_SAMPLES_PER_SECOND, 48000));
    CHECK(audioType->SetUINT32(MF_MT_AUDIO_NUM_CHANNELS, 2));
    CHECK(audioType->SetUINT32(MF_MT_AUDIO_BITS_PER_SAMPLE, 16));
    CHECK(audioType->SetUINT32(MF_MT_AUDIO_AVG_BYTES_PER_SECOND, 16000));
    CHECK(audioType->SetUINT32(MF_MT_AUDIO_BLOCK_ALIGNMENT, 1));
    CHECK(audioType->SetUINT32(MF_MT_AAC_PAYLOAD_TYPE, 0));
    CHECK(audioType->SetUINT32(MF_MT_AAC_AUDIO_PROFILE_LEVEL_INDICATION, 0x29));
    const BYTE userData[] = { 0,0, 0,0, 0,0, 0,0, 0,0, 0,0, 0x11, 0x90 };
    CHECK(audioType->SetBlob(MF_MT_USER_DATA, userData, sizeof(userData)));

    IMFByteStream* file = nullptr;
    CHECK(MFCreateFile(MF_ACCESSMODE_READWRITE, MF_OPENMODE_DELETE_IF_EXIST,
                       MF_FILEFLAGS_NONE, L"mp4sink_repro_out.mp4", &file));

    IMFMediaSink* sink = nullptr;
    CHECK(MFCreateMPEG4MediaSink(file, videoType, audioType, &sink));
    if (!sink) { printf("sink creation failed, aborting\n"); return 1; }

    DWORD characteristics = 0;
    CHECK(sink->GetCharacteristics(&characteristics));
    printf("sink characteristics: 0x%08lX (RATELESS=%d)\n", characteristics,
           !!(characteristics & MEDIASINK_RATELESS));

    IMFStreamSink* videoStream = nullptr;
    IMFStreamSink* audioStream = nullptr;
    CHECK(sink->GetStreamSinkByIndex(0, &videoStream));
    CHECK(sink->GetStreamSinkByIndex(1, &audioStream));

    IMFPresentationClock* clock = nullptr;
    IMFPresentationTimeSource* timeSource = nullptr;
    CHECK(MFCreatePresentationClock(&clock));
    CHECK(MFCreateSystemTimeSource(&timeSource));
    CHECK(clock->SetTimeSource(timeSource));
    CHECK(sink->SetPresentationClock(clock));
    CHECK(clock->Start(0));

    DriveStream(videoStream, "video", failed);
    DriveStream(audioStream, "audio", failed);

    WaitMarkerEvent(videoStream, "video");
    WaitMarkerEvent(audioStream, "audio");

    IMFFinalizableMediaSink* finalizable = nullptr;
    CHECK(sink->QueryInterface(IID_PPV_ARGS(&finalizable)));
    if (finalizable) {
        auto* cb = new FinalizeCallback(finalizable);
        CHECK(finalizable->BeginFinalize(cb, nullptr));
        if (WaitForSingleObject(cb->done, 10000) == WAIT_OBJECT_0)
            printf("EndFinalize -> 0x%08lX%s\n", cb->finalizeResult,
                   FAILED(cb->finalizeResult) ? "  <-- FAILED" : "");
        else
            printf("BeginFinalize callback never invoked (10s timeout)  <-- SUSPICIOUS\n");
        CHECK(sink->Shutdown());
    }

    printf("\n%s\n", failed ? "=== REPRO: at least one step FAILED ===" : "=== all steps OK ===");
    return failed ? 1 : 0;
}

[scream870102]

バージョン 1.5.4 からアップデートした際に発生した不具合について、参考までに共有いたします。開発の参考になれば幸いです。いつも使いやすくて素晴らしいパッケージを提供していただき、本当にありがとうございます！

[scream870102]

その後の進展を2点共有いたします。

1. 同一ビルドの別マシンでも再現を確認しました（10.0.26200、mfmp4srcsnk.dll 10.0.26100.8457）。C++ 再現コードの出力も完全に一致します（すべての PlaceMarker バリアントが MF_E_INVALIDTYPE を返す）。マシン固有ではなく、ビルド全体の問題のようです。
2. 回避策の動作を確認しました: PlaceMarker の失敗を非致命的として扱うだけで十分でした。マーカーなしでも BeginFinalize/EndFinalize は成功し、生成された mp4 は正常に再生できます。パッチを適用したビルドを問題なく運用できています。修正を fix: treat PlaceMarker failure as non-fatal and shut down sink on error #157 として提出しましたので、ご確認いただければ幸いです。