Re-working voices for Pocket TTS

[HadrienGardeur]

Caution

This is a WIP and will require to update the API, docs and integration with Pocket TTS.

With this draft PR, I've vastly simplified the list of voices for Pocket TTS, hoping to set a template for future integrations as well.

The following keys are currently used:

• name for the name of the voice meant to be displayed to the user
• originalName for the name of the voice used by the hosted model behind the scenes
• identifier is a unique identifier that is used to reference the voice in all requests to synthesize audio
• language for the primary language support for a voice
• otherLanguages for a list of other languages that the voice can support
• gender
• and quality

Overall, the idea is to avoid repeating information that can be documented at a model-level and doesn't require voice-level information.

Compared to what we had before:

• this is much more compact
• we're no longer repeating voices per language (integrators should combine identifierwith language in their requests instead)
• some voices have been further documented with regional information using BCP 47 language tags

Note

In the case of Pocket TTS, quality is a good candidate of a property that could be moved to the model for now since all voices are currently labeled as veryHigh.

Overall, the idea is to:

• document as many info as we can at a model level
• adopt a leaner approach for the list of voices and only document a property if:
  • it is voice-specific
  • and/or needs to override the model defaults
• but the API (/voices) should merge model and voice-level information, to provide a complete view to the client

In the case of Pocket TTS:

• there's no support for pitch and speed
• it cannot handle SSML
• there's no support for word boundaries
• output is always:
  • Sample Rate: 24kHz
  • Channels: Mono
  • Bit Depth: 16-bit PCM
  • Format: Standard WAV file

All of these info could be part of the service that we create for each integration/model.

At this point, the output format remains my biggest question:

• Is this information useful at all in the list of voices?
• If so, should we include strictly what the model is capable of?
• Or also include what the server is capable of transcoding to?

For now, I would prefer avoiding repeating this info at a voice-level, to keep the payload reasonably small.

[HadrienGardeur]

Let's illustrate things with an updated voice from voices.json where we go a step beyond the current PR and remove quality as well:

{
  "name": "Alba",
  "originalName": "alba",
  "identifier": "urn:readium:tts:pocket:alba",
  "language": "en-US",
  "otherLanguages": ["fr", "es", "de", "it", "pt-PT"],
  "gender": "male"
}

For Pocket TTS itself, we would have the following info:

{
  "quality": "veryHigh",
  "controls": {
    "pitch": false,
    "speed": false,
    "ssml": false,
    "boundary": false
  }
}

Given what we know about the model, we could return the following representation when listing voices:

{
  "name": "Alba",
  "originalName": "alba",
  "provider": "pocket",
  "identifier": "urn:readium:tts:pocket:alba",
  "language": "en-US",
  "otherLanguages": ["fr", "es", "de", "it", "pt-PT"],
  "gender": "male",
  "quality": "veryHigh",
  "controls": {
    "pitch": false,
    "speed": false,
    "ssml": false,
    "boundary": false
  }
}

With this approach:

• we inject model-level info per voice
• but voice-level info can override any property
• we avoid info about the output since it's probably irrelevant in the list of voices and can be worked around by the Speech Server transcoding capabilities

With good defaults on controls, this could be further simplified like this:

{
  "name": "Alba",
  "originalName": "alba",
  "provider": "pocket",
  "identifier": "urn:readium:tts:pocket:alba",
  "language": "en-US",
  "otherLanguages": ["fr", "es", "de", "it", "pt-PT"],
  "gender": "male",
  "quality": "veryHigh"
}

In the future, we might encounter models with support for SSML or boundaries only on certain voices (that's the case with Microsoft Azure's API).

In such cases, their voices.json would include this info and override model-level info like this:

{
  "name": "Joe",
  "originalName": "joe",
  "identifier": "urn:readium:tts:microsoft:joe",
  "language": "en-US",
  "gender": "male",
  "controls": {
    "ssml": true
  }
}

[ronibhakta1]

@HadrienGardeur There are few other things I would ask is, Currently the number of MAX_TEXT_LENGTH are set to 2000 by default. Which can be reset by changing the values in .env file.

Is there any specific default limit you would like to set here or 2000 seems ok ?

1. Is this information useful at all in the list of voices?

As of now, Yes. we are keeping thing as much as we can per Model like Pocket-tts

2. If so, should we include strictly what the model is capable of?

It should be a intersection list of features as per me.

3. Or also include what the server is capable of transcoding to?

We should show what the server is capable of on top of it as the user just want to use the highest quality audio.


If we set "quality": "veryHigh" and the default format is set to mp3 or wav rather than opus user might be confused.

My question is: How do we handle the "quality": "veryHigh" senario with different formats.

[HadrienGardeur]

Is there any specific default limit you would like to set here or 2000 seems ok ?

It's fine as-is. We'll need to default per model defaults and see if we can provide a configuration for all models.

If we set "quality": "veryHigh" and the default format is set to mp3 or wav rather than opus user might be confused.

My question is: How do we handle the "quality": "veryHigh" senario with different formats.

In this context, we're talking about the quality of the voice, not the quality of the audio output. This is consistent with what we've done in Readium Speech.

For the output, we should default to WAV and only transcode to OPUS or MP3 if asked for.

[ronibhakta1]

Is there any specific default limit you would like to set here or 2000 seems ok ?

It's fine as-is. We'll need to default per model defaults and see if we can provide a configuration for all models.

If we set "quality": "veryHigh" and the default format is set to mp3 or wav rather than opus user might be confused.
My question is: How do we handle the "quality": "veryHigh" senario with different formats.

In this context, we're talking about the quality of the voice, not the quality of the audio output. This is consistent with what we've done in Readium Speech.

For the output, we should default to WAV and only transcode to OPUS or MP3 if asked for.

That makes sence to me now. Thank you.