SDL3 WASAPI Audio Format Oddities

[CasualPokePlayer]

I'm noticing some oddities with the audio format handling, at least with SDL3's WASAPI handling (haven't checked other backends).

My code currently attempts to set the current audio device if it sees that it's been removed (SDL_EVENT_AUDIO_DEVICE_REMOVED), or attempts to adjust to a format change if that occurs (SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED).

However, for one of my devices, I'm seeing rather strange behavior. My headphones are a soundcore Space Q45. This pair of headphones has an odd case of having options for either "Headphones" or "Headset" with the latter having much worse audio quality. Nothing problematic for SDL (that's just a quirk of these headphones), but SDL oddly is supposedly "changing" the format of this "Headset" option underneath me (telling me the frequency has changed from 16000Hz to 44100Hz), without firing a SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED event telling me so (although there is a SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED event being fired, just with a different device ID?), the change only being visible if I just call SDL_GetAudioDeviceFormat periodically and check if the format has changed.

However, doing an adjustment with this change seems to result in "wrong" results? The audio appears to be rather distorted treating the device as 44100Hz as opposed to 16000Hz (16000Hz instead having obvious aliasing issues, but otherwise sounds correct). This could just be some weird hardware bug perhaps? (44100Hz makes sense for the "Headphones" option rather than the "Headset" option).

Another oddity comes with the default device. I set my headphones as the default device, although if I power off the headphones, the default device of course changes over to my PC speakers. Nothing unexpected. However, if I proceed to have the default device currently as PC Speakers, then proceed to power on my headphones, there appears to be another format change occurring underneath me, without the SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED event telling me so. These appears to be multiple SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED events anyways signalling something is changing, but none of the IDs end up matching the device ID returned by SDL_GetAudioStreamDevice. Again, the change becomes only visible if I just call SDL_GetAudioDeviceFormat periodically and check if the format has changed (and in this case I must adjust things, otherwise the resutling audio is distorted).

[icculus]

The device format change notifications are probably handing you physical devices IDs and not the logical device generated when opening a physical one. It should also generate an event for each affected logical device, though. It might be helpful to call SDL_GetAudioDeviceName during the event handler and see what hardware it's actually talking about.

[icculus]

Should I take it from CasualPokePlayer/GSE@358c0bd that this problem is resolved? Or is this still causing you problems?

[CasualPokePlayer]

It more or less works around these issues. It is a bit of a wonky workaround, as I'm only doing the name matching when the current device is the default audio device. If I proceed to do name matching always, I end up triggering the first issue I described where I end up getting alerted that this "Headset" audio device format changed from 16000Hz -> 44100Hz (as it seems the SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED that uses a different device ID ends up referring to the same device name), but in reality trying to pump audio through with that change results in very distorted audio (not just standard aliasing like I'd expect from 16000Hz)

[icculus]

Oh, I was just saying to use the device name to see what the event is telling you about, as something human-readable.

To be clear: all device opens generate a logical device ID, whether you request the default device or not, and you should get events for those logical device ID if the format changes. The suggestion to check the device name was because you don't seem to be getting these events, so it would be useful to see what device it's actually sending events about, if not yours.

Also, you shouldn't have to do anything on a format change: your SDL_AudioStream will automatically adjust to the new format, so you just keep giving it data in whatever format you were giving it before the device change.

The only way I could see this going bad is if for some reason it didn't think your device changed formats, so the audio stream didn't adjust, which would definitely be an SDL bug if so. Hence the suggestion to find out exactly what devices are mentioned in the events.

[CasualPokePlayer]

Also, you shouldn't have to do anything on a format change: your SDL_AudioStream will automatically adjust to the new format, so you just keep giving it data in whatever format you were giving it before the device change.

So in any case, even if the frequency did change with the format (for whatever reason) SDL_AudioStream will be resampling underneath me anyways? That makes some sense, although ideally, I'd prefer if I can make it so the stream does not resample underneath me (since I'm already handling resampling, so I don't want a secondary resampling step if I can help it). I suppose that then makes sense why adjusting 16000Hz -> 44100Hz ended up causing distortion, because the stream, originally opened with 16000Hz, is expecting 16000Hz, not the "new" 44100Hz device format.

It do need to at least be watching sample_frames's return (which does appear to indicate a difference in how much the callback tries to collect each iteration), I do some handling with that value to handle underruns occurring (the handling sucks but it's needed for now to avoid drastic audio desyncs due to system hiccups sticking around, also the handling is more or less copy pasted from how I handled it in SDL2).

That said, the device name that ends up appearing does indeed appear to by my device (along with other devices sometimes), but the IDs themselves never match (outside of SDL_EVENT_AUDIO_DEVICE_REMOVED when there's an explicit device / not default selected)

; Selecting Headset option
Got SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED with 19 / Headset (soundcore Space Q45 Hands-Free AG Audio) (current device 37 / Headset (soundcore Space Q45 Hands-Free AG Audio))
Freq 16000 -> 44100 / Sample Frames 441 -> 441
; Turning off headphones while Headset is selected
Got SDL_EVENT_AUDIO_DEVICE_REMOVED with 37 / Headset (soundcore Space Q45 Hands-Free AG Audio) (current device 37 / Headset (soundcore Space Q45 Hands-Free AG Audio))
; default device would be connected at this point
Got SDL_EVENT_AUDIO_DEVICE_REMOVED with 19 / <null> (current device 41 / Speakers (Realtek(R) Audio))
Got SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED with 15 / Speakers (Realtek(R) Audio) (current device 41 / Speakers (Realtek(R) Audio))
Freq 48000 -> 48000 / Sample Frames 128 -> 128
; Turning headphones back on (this is my preferred default device)
Got SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED with 23 / Headphones (soundcore Space Q45 Stereo) (current device 41 / Headphones (soundcore Space Q45 Stereo))
Freq 48000 -> 48000 / Sample Frames 128 -> 480
Got SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED with 15 / Speakers (Realtek(R) Audio) (current device 41 / Speakers (Realtek(R) Audio))
Freq 48000 -> 48000 / Sample Frames 480 -> 128
Got SDL_EVENT_AUDIO_DEVICE_REMOVED with 23 / <null> (current device 41 / Speakers (Realtek(R) Audio))
Got SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED with 51 / Headphones (soundcore Space Q45 Stereo) (current device 41 / Speakers (Realtek(R) Audio))

[icculus]

I suppose that then makes sense why adjusting 16000Hz -> 44100Hz ended up causing distortion, because the stream, originally opened with 16000Hz, is expecting 16000Hz, not the "new" 44100Hz device format.

Exactly. It'll automatically adjust to the new output format the hardware wants, but it assumes the app wants to keep feeding it in the format it started with, unless you tell it otherwise.

If you are planning to handle this yourself, you can change the audio stream's input format to match its output (call SDL_SetAudioStreamFormat) when you get that event, and then it won't do any resampling...just make sure you always give it new data in the format you've last set it to. It's smart enough to deal with situations where it's still processing data in a previous format as new-format data arrives, too, so it should work without even a blip of distortion when the device changes.

[CasualPokePlayer]

If you are planning to handle this yourself, you can change the audio stream's input format to match its output (call SDL_SetAudioStreamFormat) when you get that event

Looking at that, I notice this in the documentation:

If a stream is bound to a device, then the format of the side of the stream bound to a device cannot be changed (src_spec for recording devices, dst_spec for playback devices). Attempts to make a change to this side will be ignored, but this will not report an error. The other side's format can be changed.

So this wouldn't exactly work (and indeed trying it out it doesn't appear to change things). However, if I proceed to unbind the stream before changing the format then rebinding it... it doesn't appear to work either, and even worse doing that appears to just cause a crash on SDL_QuitSubSystem:

[CasualPokePlayer]

If a stream is bound to a device, then the format of the side of the stream bound to a device cannot be changed (src_spec for recording devices, dst_spec for playback devices). Attempts to make a change to this side will be ignored, but this will not report an error. The other side's format can be changed.

Oh, looks like I confused myself with this documentation, dst_spec is the spec going to the speaker (since this is a playback device), while src_spec is the spec coming from my application (the input), so I shouldn't need to unbind and rebind streams (although just doing the act of unbinding and rebinding streams causes a crash in SDL_QuitSubSystem, which is ???, although maybe that's just some UB as far as SDL's API is concerned?). Properly changing src_spec works as expected.

[icculus]

It definitely shouldn't crash, so I'll look into that.

Are you otherwise good to go, now?

[CasualPokePlayer]

Apologies for having the issue go off-topic, but I guess this is the state of it:

1. The SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED event for the Headset immediately upon connecting it is being fired, it's just using a different ID (it appears to be the physical device ID used to open it up i.e. queried from SDL_GetAudioPlaybackDevices), so if I only compare with the logical ID obtained from SDL_GetAudioStreamDevice I miss the event, if I compare with the physical device ID too I don't miss the event:

Opened Headset (soundcore Space Q45 Hands-Free AG Audio) using device ID 19 (stream ID 33), got freq 16000 / batch size 441
Got SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED with 19 / Headset (soundcore Space Q45 Hands-Free AG Audio) (current device 33 / Headset (soundcore Space Q45 Hands-Free AG Audio))

2. The "adjustment" I was doing with 16000Hz -> 44100Hz resulting in audio distortion was just user error, as the playback audio stream would ensure that the input format never changes (which makes sense anyways). If I wanted to change the frequency to match this change I would need to call SDL_SetAudioStreamFormat with src_spec (and doing this makes this adjustment work, granted I'm not sure why SDL is even deciding to change the device output over to 44100Hz, as 16000Hz is the actual sample rate of the device anyways, but w/e).
3. With the default device, I don't have any physical device ID to actually look at, only a logical one (either SDL_AUDIO_DEVICE_DEFAULT_PLAYBACK or what I get from SDL_GetAudioStreamDevice). As such I miss any format change events with the default device (which can be problematic for me since I need to watch sample_frames from SDL_GetAudioDeviceFormat for my audio handling). As of now I can workaround it by simply matching by name, but of course this is undesirable.
4. I don't have any issues it seems anyways with SDL_EVENT_AUDIO_DEVICE_REMOVED events, as those end up just passing the logical ID (i.e. what SDL_GetAudioStreamDevice returned) anyways, so I can appear to always gracefully fallback on a default device if an explicit device choice ends up becoming unavailable. Most of this issue just appears to come from SDL not passing the logical device IDs over with SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED events (which can be worked around by just holding onto the physical device ID, or doing a name comparision in the case of the default device as there is no physical device ID to check there).

[CasualPokePlayer]

Debugging this a little:

1. I see that SDL does attempt to set all logical devices associated with a physical device. However, the point which the first SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED event is created is in the middle of the WASAPI_OpenDevice call. This is within OpenPhysicalDevice, which is called before the logical device ID is even created, so the event cannot be fired for the logical device, only the physical device (other devices appear to end up having the same issue anyways besides the Headset, the Headset one was just noticed since it changed frequency).
2. The 16000Hz -> 44100Hz change I saw appears to just be imposed by SDL, with no way to disable such an adjustment. Kind of off topic, but I would think there probably should be a hint for this behavior (for users that are really sure about the format they're opening in because that's the actual output device sample rate).
3. I think this is just issue 1 again actually? It seems like there are SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED events firing for the logical ID returned when opening the default device, but only for when the default device is changing, not when first opening up the device (which then includes my app's logic for opening up a default device after an explicit device choice is lost), so it's just another case of the logical device ID not yet existing when the event is actually created.

EDIT: I'm now not getting SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED events when the default device changes, so I'm just confused on what could even be going wrong with the default device, sometimes it wants to work correctly and sometimes it doesn't?

[icculus]

I see that SDL does attempt to set all logical devices associated with a physical device. However, the point which the first SDL_EVENT_AUDIO_DEVICE_FORMAT_CHANGED event is created is in the middle of the WASAPI_OpenDevice call. This is within OpenPhysicalDevice, which is called before the logical device ID is even created, so the event cannot be fired for the logical device, only the physical device (other devices appear to end up having the same issue anyways besides the Headset, the Headset one was just noticed since it changed frequency).

Okay, yes, this happens because there are several things we don't know about the WASAPI device before we formally open it, so we correct things as new information arrives.

Best practice here would be to open the device, and then make adjustments from there:

// (C# is not my area of expertise, adjust accordingly if I screwed this up.)

var audioDeviceStream = SDL_OpenAudioDeviceStream(
	devid: deviceId,
	spec: ref wantedAudioSpec,
	callback: &SDLAudioCallback,
	userdata: GCHandle.ToIntPtr(_sdlUserData)
);

// (assuming audioDeviceStream is valid here.)

var deviceAudioSpec = default(SDL_AudioSpec);
SDL_GetAudioDeviceFormat(SDL_GetAudioStreamDevice(audioDeviceStream), out deviceAudioSpec, out _);
SDL_SetAudioStreamFormat(audioDeviceStream, deviceAudioSpec, null);

// now always feed audioDeviceStream data in deviceAudioSpec format, no matter what it is.

The 16000Hz -> 44100Hz change I saw appears to just be imposed by SDL, with no way to disable such an adjustment. Kind of off topic, but I would think there probably should be a hint for this behavior (for users that are really sure about the format they're opening in because that's the actual output device sample rate).

This is a good idea, I'll add this. The 44.1Khz minimum is to be friendly to other things that might want to use audio, but if you control the whole application, there's no reason not to allow you to bypass this.

But even with this hint in place: WASAPI sometimes does whatever it wants, so you'll definitely want to do something like the code I posted above, in case it goes rogue on you.