AllTalk TTS

🛠️ About this project

AllTalk is a labour of love, developed and supported in my personal free time. As such, my ability to respond to support requests is limited. I prioritize issues based on their impact and the number of users affected. I appreciate your understanding and patience. If your inquiry isn't covered by the documentation or existing discussions, and it's not related to a bug or feature request, I'll do my best to assist as time allows.

💖 Showing Your Support

If AllTalk has been helpful to you, consider showing your support through a donation on my Ko-fi page. Your support is greatly appreciated and helps ensure the continued development and improvement of AllTalk.

🗣️ Demo/Test TTS

🔊 Using Voice Samples

Where are the sample voices stored?

Voice samples are stored in /alltalk_tts/voices/ and should be named using the following format name.wav

🔊 Where are the outputs stored & Automatic output wav file deletion.

Voice outputs are stored in /alltalk_tts/outputs/

You can configure automatic maintenance deletion of old wav files by setting Del WAV's older than in the settings above.

When Disabled your output wav files will be left untouched. When set to a setting 1 Day or greater, your output wav files older than that time period will be automatically deleted on start-up of AllTalk.

🔊 Where are the models stored?

This extension will download the 2.0.2 model to /alltalk_tts/models/

This TTS engine will also download the latest available model and store it wherever your OS normally stores it (Windows/Linux/Mac).

🔊 How do I create a new voice sample?

To create a new voice sample, you need to make a wav file that is 22050Hz, Mono, 16 bit and between 6 to 30 seconds long, though 8 to 10 seconds is usually good enough. The model can handle up to 30 second samples, however I've not noticed any improvement in voice output from much longer clips.

You want to find a nice clear selection of audio, so lets say you wanted to clone your favourite celebrity. You may go looking for an interview where they are talking. Pay close attention to the audio you are listening to and trying to sample. Are there noises in the background, hiss on the soundtrack, a low humm, some quiet music playing or something? The better quality the audio the better the final TTS result. Don't forget, the AI that processes the sounds can hear everything in your sample and it will use them in the voice its trying to recreate.

Try make your clip one of nice flowing speech, like the included example files. No big pauses, gaps or other sounds. Preferably a sample that the person you are trying to copy will show a little vocal range and emotion in their voice. Also, try to avoid a clip starting or ending with breathy sounds (breathing in/out etc).

🔊 Editing your sample!

So, you’ve downloaded your favourite celebrity interview off YouTube, from here you need to chop it down to 6 to 30 seconds in length and resample it.

If you need to clean it up, do audio processing, volume level changes etc, do this before down-sampling.

Using the latest version of Audacity select/highlight your 6 to 30 second clip and:

                  Tracks > Resample to 22050Hz then
                  Tracks > Mix > Stereo to Mono then
                  File > Export Audio saving it as a WAV of 22050Hz.

Save your generated wav file in the /alltalk_tts/voices/ folder.

Its worth mentioning that using AI generated audio clips and bad quality samples may introduce unwanted sounds into TTS generation.

🔊 Why doesnt it sound like XXX Person?

Maybe you might be interested in trying Finetuning of the model. Otherwise, the reasons can be that you:

• Didn't down-sample it as above.
• Have a bad quality voice sample.
• Try using the 3x different generation methods: API TTS, API Local, and XTTSv2 Local within the web interface, as they generate output in different ways and sound different.

Some samples just never seem to work correctly, so maybe try a different sample. Always remember though, this is an AI model attempting to re-create a voice, so you will never get a 100% match.

Back to top of page

🧮 Low VRAM

The Low VRAM option is a crucial feature designed to enhance performance under constrained (VRAM) conditions, as the TTS models require 2GB-3GB of VRAM to run effectively. This feature strategically manages the relocation of the Text-to-Speech (TTS) model between your system's Random Access Memory (RAM) and VRAM, moving it between the two on the fly. Obviously, this is very useful for people who have smaller graphics cards or people whos LLM has filled their VRAM.

When you don't have enough VRAM free after loading your LLM model into your VRAM (Normal Mode example below), you can see that with so little working space, your GPU will have to swap in and out bits of the TTS model, which causes horrible slowdown.

Note: An Nvidia Graphics card is required for the LowVRAM option to work, as you will just be using system RAM otherwise. 

🧮 How It Works

The Low VRAM mode intelligently orchestrates the relocation of the entire TTS model and stores the TTS model in your system RAM. When the TTS engine requires VRAM for processing, the entire model seamlessly moves into VRAM, causing your LLM to unload/displace some layers, ensuring optimal performance of the TTS engine.

Post-TTS processing, the model moves back to system RAM, freeing up VRAM space for your Language Model (LLM) to load back in the missing layers. This adds about 1-2 seconds to both text generation by the LLM and the TTS engine.

By transferring the entire model between RAM and VRAM, the Low VRAM option avoids fragmentation, ensuring the TTS model remains cohesive and has all the working space it needs in your GPU, without having to just work on small bits of the TTS model at a time (which causes terrible slow down).

This creates a TTS generation performance Boost for Low VRAM Users and is particularly beneficial for users with less than 2GB of free VRAM after loading their LLM, delivering a substantial 5-10x improvement in TTS generation speed.

🧮 Visual explanation

Back to top of page

🏰 SillyTavern Support

Important note for Text-generation-webui users

You HAVE to disable Enable TTS within the Text-generation-webui AllTalk interface, otherwise Text-generation-webui will also generate TTS due to the way it sends out text. You can do this each time you start up Text-generation-webui or set it in the start-up settings at the top of this page.

🏰 Quick Tips

• Only change DeepSpeed, Low VRAM or Model one at a time. Wait for it to say Ready before changing something else.
• You can permanently change the AllTalk startup settings or DeepSpeed, Low VRAM and Model at the top of this page.
• Different AI models use quotes and asterisks differently, so you may need to change "Text not inside" depending on model.
• Add new voice samples to the voices folder. You can Finetune a model to make it sound even closer to the original sample.
• DeepSpeed will improve processing time to TTS to be 2-3x faster.
• Low VRAM can be very beneficial if you don't have much memory left after loading your LLM.

🏰 TTS Generation Methods in SilllyTavern

You have 2 types of audio generation options, Streaming and Standard.

The Streaming Audio Generation method is designed for speed and is best suited for situations where you just want quick audio playback. This method, however, is limited to using just one voice per TTS generation request. This means a limitation of the Streaming method is the inability to utilize the AllTalk narrator function, making it a straightforward but less nuanced option.

On the other hand, the Standard Audio Generation method provides a richer auditory experience. It's slightly slower than the Streaming method but compensates for this with its ability to split text into multiple voices. This functionality is particularly useful in scenarios where differentiating between character dialogues and narration can enhance the storytelling and delivery. The inclusion of the AllTalk narrator functionality in the Standard method allows for a more layered and immersive experience, making it ideal for content where depth and variety in voice narration add significant value.

In summary, the choice between Streaming and Standard methods in AllTalk TTS depends on what you want. Streaming is great for quick and simple audio generation, while Standard is preferable for a more dynamic and engaging audio experience.

• AllTalk TTS Generation Method:
  • Select between Standard and Streaming Audio Generation methods.
  • This setting impacts the AllTalk narrator functionality.
• Language Selection:
  
  • Select your preferred TTS generation language from the "Language" dropdown.
• Model Switching:
  • Switch between different TTS models like API TTS, API Local, XTTSv2 Local, and optionally XTTSv2 FT if you have a finetuned model available.
  • Fine-tuned model availability (XTTSv2 FT) will only show when a finetuned model is detected by AllTalk.
  • See TTS Models/Methods for more information (though most people will want to stick with XTTSv2 Local).
• DeepSpeed and Low VRAM Options:
  • Optimize performance with DeepSpeed and Low VRAM settings.
  • DeepSpeed can offer a 2-3x performance boost on TTS generation. (Requires installation)
  • See the relevant sections in this documentation for details.

Changing model or DeepSpeed or Low VRAM each take about 15 seconds so you should only change one at a time and wait for Ready before changing the next setting. To set these options long term you can apply the settings at the top of this page.

🏰 AllTalk Narrator

Only available on the Standard Audo Generation method.

• Narrator Voice Selection:
  • Allows users to choose different narrator voices.
  • Access via the "Narrator Voice" dropdown.
• AllTalk Narrator:
  • Toggle the AllTalk narrator feature.
  • Access via "AT Narrator" dropdown with Enabled/Disabled options.
• Text Outside Asterisks Handling:
  • Choose how text outside asterisks is interpreted (as Narrator or Character voice).
  • Managed via the "Text Not Inside * or "" dropdown.
  • Note, only available when the AllTalk Narrator is enabled.

🏰 Usage Notes:

• On startup of SillyTavern, it will pull your current settings from AllTalk (Current model, DeepSpeed status, Low VRAM status and Finetuned model availability).
• Enabling the narrator automatically unchecks certain checkboxes related to text handling.
• Changes in model or settings might trigger multiple requests to the server; patience is advised.

🏰 Troubleshooting:

• If experiencing issues, use the Reload button in SillyTavern's TTS extension to reinitialize the connection to AllTalk and check if AllTalk is started correctly.

Back to top of page

🟩 Good to know

🟩 A note on Character Cards, Greeting Messages & Text-not-inside

Click to expand

Messages intended for the Narrator should be enclosed in asterisks * and those for the character inside quotation marks ". However, AI systems often deviate from these rules, resulting in text that is neither in quotes nor asterisks. Sometimes, text may appear with only a single asterisk, and AI models may vary their formatting mid-conversation. For example, they might use asterisks initially and then switch to unmarked text. A properly formatted line should look like this:

"Hey! I'm so excited to finally meet you. I've heard so many great things about you and I'm eager to pick your brain about computers." *She walked across the room and picked up her cup of coffee*

Most narrator/character systems switch voices upon encountering an asterisk or quotation marks, which is somewhat effective. AllTalk has undergone several revisions in its sentence splitting and identification methods. While some irregularities and AI deviations in message formatting are inevitable, any line beginning or ending with an asterisk should now be recognized as Narrator dialogue. Lines enclosed in double quotes are identified as Character dialogue. For any other text, you can choose how AllTalk handles it: whether it should be interpreted as Character or Narrator dialogue (most AI systems tend to lean more towards one format when generating text not enclosed in quotes or asterisks).

With improvements to the splitter/processor, I'm confident it's functioning well. You can monitor what AllTalk identifies as Narrator lines on the command line and adjust its behavior if needed (Text Not Inside - Function).

🟩 Changing AllTalks IP address & Accessing AllTalk over your Network

🟩 Text-geneneration-webui & Stable-Diffusion Plugin - Load Order & stripped text

🟩 I want to know more about the XTTS AI model used

Back to top of page

🟪 Updating

Maintaining the latest version of your setup ensures access to new features and improvements. Below are the steps to update your installation, whether you're using Text-Generation-webui or running as a Standalone Application.

🟪 Resolving Update Issues

If you encounter problems during or after an update, following these steps can help resolve the issue by refreshing your installation while preserving your data:

Back to top of page

🔵🟢 DeepSpeed Installation Options

DeepSpeed requires an Nvidia Graphics card

DeepSpeed provides a 2x-3x speed boost for Text-to-Speech and AI tasks. It's all about making AI and TTS happen faster and more efficiently.

🔵 Linux Installation

DeepSpeed requires access to the Nvidia CUDA Development Toolkit to compile on a Linux system. It's important to note that this toolkit is distinct and unrealted to your graphics card driver or the CUDA version the Python environment uses.

🟢 Windows Installation

You have 2x options for how to setup DeepSpeed on Windows. Pre-compiled wheel files for specific Python, CUDA and Pytorch builds, or manually compiling DeepSpeed.

🔵🟢 DeepSpeed Performance Example

Back to top of page

🆘 Support Requests, Troubleshooting & Feature requests

I'm thrilled to see the enthusiasm and engagement with AllTalk! Your feedback and questions are invaluable, helping to make this project even better. To ensure everyone gets the help they need efficiently, please consider the following before submitting a support request:

Consult the Documentation: A comprehensive guide and FAQ sections (below) are available to help you navigate AllTalk. Many common questions and troubleshooting steps are covered here.

Search Past Discussions: Your issue or question might already have been addressed in the discussions area or closed issues. Please use the search function to see if there's an existing solution or advice that applies to your situation.

Bug Reports: If you've encountered what you believe is a bug, please first check the Updates & Bug Fixes List to see if it's a known issue or one that's already been resolved. If not, I encourage you to report it by raising a bug report in the Issues section, providing as much detail as possible to help identify and fix the issue.

Feature Requests: The current Feature request list can be found here. I love hearing your ideas for new features! While I can't promise to implement every suggestion, I do consider all feedback carefully. Please share your thoughts in the Discussions area or via a Feature Request in the Issues section.

Back to top of page

🟨 Help with problems

     🔄 Minor updates/bug fixes list can be found here

      Please check Github for the most up to date help list here

🟨 How to make a diagnostics report file

If you are on a Windows machine or a Linux machine, you should be able to use the atsetup.bat or ./atsetup.sh utility to create a diagnositcs file. If you are unable to use the atsetup utility, please follow the instructions below.

Manually making a diagnostics report file

1. Open a command prompt window and start the Python environment. Depending on your setup (Text-generation-webui or Standalone AllTalk), the steps to start the Python environment vary:
   

• For Text-generation-webui Users:

  • Navigate to the Text-generation-webui directory:
    • cd text-generation-webui
  • Start the Python environment suitable for your OS:
    • Windows: cmd_windows.bat
    • Linux: ./cmd_linux.sh
    • macOS: cmd_macos.sh
    • WSL (Windows Subsystem for Linux): cmd_wsl.bat
  • Move into the AllTalk directory:
    • cd extensions/alltalk_tts

• For Standalone AllTalk Users:

  • Navigate to the alltalk_tts folder:
    • cd alltalk_tts
  • Start the Python environment:
    • Windows: start_environment.bat
    • Linux: ./start_environment.sh
      
      

  If you're unfamiliar with Python environments and wish to learn more, consider reviewing Understanding Python Environments Simplified in the Help section.

2. Run the diagnostics and select the requirements file name you installed AllTalk with:
   

   • python diagnostics.py

3. You will have an on screen output showing your environment setttings, file versions request vs whats installed and details of your graphics card (if Nvidia). This will also create a file called diagnostics.log in the alltalk_tts folder, that you can upload if you need to create a support ticket on here.
   
   

Installation and Setup Issues

🟨 Understanding Python Environments Simplified

Think of Python environments like different rooms in your house, each designed for a specific purpose. Just as you wouldn't cook in the bathroom or sleep in the kitchen, different Python applications need their own "spaces" or environments because they have unique requirements. Sometimes, these requirements can clash with those of other applications (imagine trying to cook a meal in a bathroom!). To avoid this, you can create separate Python environments.

Why Separate Environments?

Separate environments, like separate rooms, keep everything organized and prevent conflicts. For instance, one Python application might need a specific version of a library or dependency, while another requires a different version. Just as you wouldn't store kitchen utensils in the bathroom, you wouldn't want these conflicting requirements to interfere with each other. Each environment is tailored and customized for its application, ensuring it has everything it needs without disrupting others.

How It Works in Practice:

Standalone AllTalk Installation: When you install AllTalk standalone, it's akin to adding a new room to your house specifically designed for your AllTalk activities. The setup process, using the atsetup utility, constructs this custom "room" (Python environment alltalk_environment) with all the necessary tools and furnishings (libraries and dependencies) that AllTalk needs to function smoothly, without meddling with the rest of your "house" (computer system). The AllTalk environment is started each time you run start_alltalk or start_environment within the AllTalk folder.

Text-generation-webui Installation: Similarly, installing Text-generation-webui is like setting up another specialized room. Upon installation, it automatically creates its own tailored environment, equipped with everything required for text generation, ensuring a seamless and conflict-free operation. The Text-generation-webui environment is started each time you run start_*your-os-version* or cmd_*your-os-version* within the Text-generation-webui folder.

Managing Environments:

Just as you might renovate a room or bring in new furniture, you can also update or modify Python environments as needed. Tools like Conda or venv make it easy to manage these environments, allowing you to create, duplicate, activate, or delete them much like how you might manage different rooms in your house for comfort and functionality.

Once you're in the right environment, by activating it, installing or updating dependencies (the tools and furniture of your Python application) is straightforward. Using pip, a package installer for Python, you can easily add what you need. For example, to install all required dependencies listed in a requirements.txt file, you'd use:

pip install -r requirements.txt

This command tells pip to read the list of required packages and versions from the requirements.txt file and install them in the current environment, ensuring your application has everything it needs to operate. It's like having a shopping list for outfitting a room and ensuring you have all the right items delivered and set up.

Remember, just as it's important to use the right tools for tasks in different rooms of your house, it's crucial to manage your Python environments and dependencies properly to ensure your applications run as intended.

How do I know if I am in a Python environment?:

When a Python environment starts up, it changes the command prompt to show the Python environment that it currently running within that terminal/console.

Networking and Access Issues

Configuration and Usage Issues

Performance and Compatibility Issues

Application Specific Issues

TTS Generation Issues & Questions

Back to top of page

⚫ Finetuning a model

If you have a voice that the model doesnt quite reproduce correctly, or indeed you just want to improve the reproduced voice, then finetuning is a way to train your "XTTSv2 local" model (stored in /alltalk_tts/models/xxxxx/) on a specific voice. For this you will need:

• An Nvidia graphics card. (Please see the help section note if you have multiple Nvidia GPU's).
• 18GB of disk space free (most of this is used temporarily)
• At least 2 minutes of good quality speech from your chosen speaker in mp3, wav or flacc format, in one or more files (have tested as far as 20 minutes worth of audio).
• As a side note, many people seem to think that the Whisper v2 model (used on Step 1) is giving better results at generating training datasets, so you may prefer to try that, as opposed to the Whisper 3 model.

⚫ How will this work/How complicated is it?

Everything has been done to make this as simple as possible. At its simplest, you can literally just download a large chunk of audio from an interview, and tell the finetuning to strip through it, find spoken parts and build your dataset. You can literally click 4 buttons, then copy a few files and you are done. At it's more complicated end you will clean up the audio a little beforehand, but its still only 4x buttons and copying a few files.

⚫ The audio you will use

I would suggest that if its in an interview format, you cut out the interviewer speaking in audacity or your chosen audio editing package. You dont have to worry about being perfect with your cuts, the finetuning Step 1 will go and find spoken audio and cut it out for you. Is there is music over the spoken parts, for best quality you would cut out those parts, though its not 100% necessary. As always, try to avoid bad quality audio with noises in it (humming sounds, hiss etc). You can try something like Audioenhancer to try clean up noisier audio. There is no need to down-sample any of the audio, all of that is handled for you. Just give the finetuning some good quality audio to work with.

⚫ Can I Finetune a model more than once on more than one voice

Yes you can. You would do these as multiple finetuning's, but its absolutely possible and fine to do. Finetuning the XTTS model does not restrict it to only being able to reproduce that 1x voice you trained it on. Finetuning is generally nuding the model in a direction to learn the ability to sound a bit more like a voice its not heard before.

⚫ A note about anonymous training Telemetry information & disabling it

Portions of Coqui's TTS trainer scripts gather anonymous training information which you can disable. Their statement on this is listed here. If you start AllTalk Finetuning with start_finetuning.bat or ./start_finetuning.sh telemetry will be disabled. If you manually want to disable it, please expand the below:

⚫ Prerequisites for Fine-tuning with Nvidia CUDA Development Toolkit 11.8

All the requirements for Finetuning will be installed by using the atsetup utility and installing your correct requirements (Standalone or for Text-generation-webui). The legacy manual instructions are stored below, however these shouldnt be required.

Additional Note on Torch and Torchaudio:

• Ensure Torch and Torchaudio are CUDA-enabled (any version), which is separate from the CUDA Toolkit installation. CUDA 11.8 corresponds to cu118 and CUDA 12.1 to cu121 in AllTalk diagnostics.
• Failure to install CUDA for Torch and Torchaudio will result in Step 2 of fine-tuning failing. These requirements are distinct from the CUDA Toolkit installation, so avoid conflating the two.
  

⚫ Starting Fine-tuning

NOTE: Ensure AllTalk has been launched at least once after any updates to download necessary files for fine-tuning.

1. Close Resource-Intensive Applications:

   • Terminate any applications that are using your GPU/VRAM to ensure enough resources for fine-tuning.

2. Organize Voice Samples:

   • Place your audio samples into the following directory: /alltalk_tts/finetune/put-voice-samples-in-here/

Depending on your setup (Text-generation-webui or Standalone AllTalk), the steps to start the Python environment vary:

• For Standalone AllTalk Users:

  • Navigate to the alltalk_tts folder:
    • cd alltalk_tts
  • Start the Python environment:
    • Windows: start_finetune.bat
    • Linux: ./start_finetune.sh

• For Text-generation-webui Users:

  • Navigate to the Text-generation-webui directory:
    • cd text-generation-webui
  • Start the Python environment suitable for your OS:
    • Windows: cmd_windows.bat
    • Linux: ./cmd_linux.sh
    • macOS: cmd_macos.sh
    • WSL (Windows Subsystem for Linux): cmd_wsl.bat
  • Move into the AllTalk directory:
    • cd extensions/alltalk_tts
  • Linux users only need to run this command:
    • export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`
  • Start the fine-tuning process with the command:
    • python finetune.py
      
      

  If you're unfamiliar with Python environments and wish to learn more, consider reviewing Understanding Python Environments Simplified in the Help section.

3. Pre-Flight Checklist:

   • Go through the pre-flight checklist to ensure readiness. Address any issues flagged as "Fail".

4. Post Fine-tuning Actions:

   • Upon completing fine-tuning, the final tab will guide you on managing your files and relocating your newly trained model to the appropriate directory.

These steps guide you through the initial preparations, starting the Python environment based on your setup, and the fine-tuning process itself. Ensure all prerequisites are met to facilitate a smooth fine-tuning experience.

⚫ How many Epochs etc is the right amount?

In finetuning the suggested/recommended amount of epochs, batch size, evaluation percent etc is already set. However, there is no absolutely correct answer to what the settings should be, it all depends on what you are doing.

• If you just want to train a normal human voice that is in an existing language, for most people’s needs, the base settings would work fine. You may choose to increase the epochs up to maybe 20, or run a second round of training if needed.
• If you were training an entirely new language, you would need a huge amount of training data and it requires around 1000 epochs (based on things I can find around the internet of people who tried this).
• If you are training a cartoon style voice in an existing language, it may need well upwards of 40 epochs until it can reproduce that voice with some success.

There are no absolute correct settings, as there are too many variables, ranging from the amount of samples you are using (5 minutes worth? 4 hours worth? etc), if they are similar samples to what the AI model already understands, so on and so forth. Coqui whom originally trained the model usually say something along the lines of, once you’ve trained it X amount, if it sounds good then you are done and if it doesn’t, train it more.

⚫ Evaluation Data Percentage

In the process of finetuning, it's crucial to balance the data used for training the model against the data reserved for evaluating its performance. Typically, a portion of the dataset is set aside as an 'evaluation set' to assess the model's capabilities in dealing with unseen data. On Step 1 of finetuning you have the option to adjust this evaluation data percentage, offering more control over your model training process.

Why Adjust the Evaluation Percentage?

Adjusting the evaluation percentage can be beneficial in scenarios with limited voice samples. When dealing with a smaller dataset, allocating a slightly larger portion to training could enhance the model's ability to learn from these scarce samples. Conversely, with abundant data, a higher evaluation percentage might be more appropriate to rigorously test the model's performance. There are currently no absolutely optimal split percentages as it varies by dataset.

• Default Setting: The default evaluation percentage is set at 15%, which is a balanced choice for most datasets.

• Adjustable Range: Users can now adjust this percentage, but it’s generally recommend keeping it between 5% and 30%.

  • Lower Bound: A minimum of 5% ensures that there's enough data to evaluate model performance.
  • Upper Bound: Its suggested not exceeding 30% for evaluation to avoid limiting the amount of data available for training.

• Understanding the Impact: Before adjusting this setting, it's important to understand its impact on model training and evaluation. Incorrect adjustments can lead to suboptimal model performance.

• Gradual Adjustments: For those unfamiliar with the process, we recommend reading up on training data and training sets, then making small, incremental changes and observing their effects.

• Data Quality: Regardless of the split, the quality of the audio data is paramount. Ensure that your datasets are built from good quality audio with enough data within them.

⚫ Using a Finetuned model in Text-generation-webui

At the end of the finetune process, you will have an option to Compact and move model to /trainedmodel/ this will compact the raw training file and move it to /model/trainedmodel/. When AllTalk starts up within Text-generation-webui, if it finds a model in this location a new loader will appear in the interface for XTTSv2 FT and you can use this to load your finetuned model.

Be careful not to train a new model from the base model, then overwrite your current /model/trainedmodel/ if you want a seperately trained model. This is why there is an OPTION B to move your just trained model to /models/lastfinetuned/.

⚫ Training one model with multiple voices

At the end of the finetune process, you will have an option to Compact and move model to /trainedmodel/ this will compact the raw training file and move it to /model/trainedmodel/. This model will become available when you start up finetuning. You will have a choice to train the Base Model or the Existing finetuned model (which is the one in /model/trainedmodel/). So you can use this to keep further training this model with additional voices, then copying it back to /model/trainedmodel/ at the end of training.

⚫ Do I need to keep the raw training data/model?

If you've compacted and moved your model, its highly unlikely you would want to keep that data, however the choice is there to keep it if you wish. It will be between 5-10GB in size, so most people will want to delete it.

Back to top of page

⬜ AllTalk TTS Generator

• AllTalk TTS Generator Link

AllTalk TTS Generator is the solution for converting large volumes of text into speech using the voice of your choice. Whether you're creating audio content or just want to hear text read aloud, the TTS Generator is equipped to handle it all efficiently. Please see here for a quick demo

The link to open the TTS generator can be found on the built-in Settings and Documentation page.

DeepSpeed is highly recommended to speed up generation. Low VRAM would be best turned off and your LLM model unloaded from your GPU VRAM (unload your model). No Playback will reduce memory overhead on very large generations (15,000 words or more). Splitting Export to Wav into smaller groups will also reduce memory overhead at the point of exporting your wav files (so good for low memory systems).

⬜ Estimated Throughput

This will vary by system for a multitude of reasons, however, while generating a 58,000 word document to TTS, with DeepSpeed enabled, LowVram disabled, splitting size 2 and on an Nvidia RTX 4070, throughput was around 1,000 words per minute. Meaning, this took 1 hour to generate the TTS. Exporting to combined wavs took about 2-3 minutes total.

⬜ Quick Start

• Text Input: Enter the text you wish to convert into speech in the 'Text Input' box.
• Generate TTS: Hit this to start the text-to-speech conversion.
• Pause/Resume: Used to pause and resume the playback of the initial generation of wavs or the stream.
• Stop Playback: This will stop the current audio playing back. It does not stop the text from being generated however. Once you have sent text off to be generated, either as a stream or wav file generation, the TTS server will remain busy until this process has competed. As such, think carefully as to how much you want to send to the server. If you are generating wav files and populating the queue, you can generate one lot of text to speech, then input your next lot of text and it will continue adding to the list.

⬜ Customization and Preferences

• Character Voice: Choose the voice that will read your text.
• Language: Select the language of your text.
• Chunk Sizes: Decide the size of text chunks for generation. Smaller sizes are recommended for better TTS quality.

⬜ Interface and Accessibility

• Dark/Light Mode: Switch between themes for your visual comfort.
• Word Count and Generation Queue: Keep track of the word count and the generation progress.

⬜ TTS Generation Modes

• Wav Chunks: Perfect for creating audio books, or anything you want to keep long term. Breaks down your text into manageable wav files and queues them up. Generation begins automatically, and playback will start after a few chunks have been prepared ahead. You can set the volume to 0 if you don’t want to hear playback. With Wav chunks, you can edit and/or regenerate portions of the TTS as needed.
• Streaming: For immediate playback without the ability to save. Ideal for on-the-fly speech generation and listening. This will not generate wav files and it will play back through your browser. You cannot stop the server generating the TTS once it has been sent.
  
  With wav chunks you can either playback “In Browser” which is the web page you are on, or “On Server” which is through the console/terminal where AllTalk is running from, or "No Playback". Only generation “In Browser” can play back smoothly and populate the Generated TTS List. Setting the Volume will affect the volume level played back both “In Browser” and “On Server”.
  
  For generating large amounts of TTS, it's recommended to select the No Playback option. This setting minimizes the memory usage in your web browser by avoiding the loading and playing of audio files directly within the browser, which is particularly beneficial for handling extensive audio generations. The definition of large will vary depending on your system RAM availability (will update when I have more information as to guidelines). Once the audio is generated, you can export your list to JSON (for safety) and use the Play List option to play back your audio.

⬜ Playback and List Management

• Playback Controls: Utilize 'Play List' to start from the beginning or 'Stop Playback' to halt at any time.
• Custom Start: Jump into your list at a specific ID to hear a particular section.
• Regeneration and Editing: If a chunk isn't quite right, you can opt to regenerate it or edit the text directly. Click off the text to save changes and hit regenerate for the specific line.
• Export/Import List: Save your TTS list as a JSON file or import one. Note: Existing wav files are needed for playback. Exporting is handy if you want to take your files away into another program and have a list of which wav is which, or if you keep your audio files, but want to come back at a later date, edit one or two lines, regenerate the speech and re-combine the wav’s into one new long wav.

⬜ Exporting Your Audio

• Export to WAV: Combine all generated TTS from the list, into one single WAV file for easy download and distribution. Its always recommended to export your list to a JSON before exporting, so that you have a backup, should something go wrong. You can simply re-import the list and try exporting again.
  
  When exporting, there is a file size limit of 1GB and as such you have the option to choose how many files to include in each block of audio exported. 600 is just on the limit of 1GB, depending on the average file size, so 500 or less is a good amount to work with. You can combine the generated files after if you wish, in Audacity or similar.
  
  Additionally, lower export batches will lower the memory requirements, so if your system is low on memory (maybe 8 or 16GB system), you can use smaller export batches to keep the memory requirement down.

⬜ Exporting Subtitles (SRT file)

• Export SRT: This will scan through all wav files in your list and generate a subtitles file that will match your exported wav file.

⬜ Analyzing generated TTS for errors

• Analyze TTS: This will scan through all wav files comparing each ID's orignal text with the TTS generated for that ID and then flag up inconsistences. Its important to understand this is a best effort process and not 100% perfect, for example:
  
  
  • Your text may have the word their and the automated routine that listens to your generated TTS interprets the word as there, aka a spelling difference.
  • Your text may have Examples are: (note the colon) and the automated routine that listens to your generated TTS interprets the word as "Examples are` (note no colon as you cannot sound out a colon in TTS), aka a punctuation difference.
  • Your text may have There are 100 items and the automated routine that listens to your generated TTS interprets the word as There are one hundred items, aka numbers vs the number written out in words.
  • There will be other examples such as double quotes. As I say, please remember this is a best effort to help you identify issues.
    

As such, there is a % Accuracy setting. This uses a couple of methods to try find things that are similar e.g. taking the their and there example from above, it would identify that they both sound the same, so even if the text says their and the AI listening to the generated TTS interprets the word as there, it will realise that both sound the same/are similar so there is no need to flag that as an error. However, there are limits to this and some things may slip through or get picked up when you would prefer them not to be flagged.

The higher the accuracy you choose, the more things it will flag up, however you may get more unwanted detections. The lower the less detections. Based on my few tests, accuracy settings between 96 to 98 seem to generally give the best results. However, I would highly recommend you test out a small 10-20 line text and test out the Analyze TTS button to get a feel for how it responds to different settings, as well as things it flags up.

You will be able to see the ID's and Text (orignal and as interpreted) by looking at the terminal/command prompt window.

The Analyze TTS feature uses the Whisper Larger-v2 AI engine, which will download on first use if necessary. This will require about 2.5GB's of disk space and could take a few minutes to download, depending on your internet connection.

⬜ Tricks to get the model to say things correctly

Sometimes the AI model won’t say something the way that you want it to. It could be because it’s a new word, an acronym or just something it’s not good at for whatever reason. There are some tricks you can use to improve the chances of it saying something correctly.

Adding pauses
You can use semi-colons ";" and colons ":" to create a pause, similar to a period "." which can be helpful with some splitting issues.

Acronyms
Not all acronyms are going to be pronounced correctly. Let’s work with the word ChatGPT. We know it is pronounced "Chat G P T" but when presented to the model, it doesn’t know how to break it down correctly. So, there are a few ways we could get it to break out "Chat" and the G P and T. e.g.

Chat G P T. Chat G,P,T. Chat G.P.T. Chat G-P-T. Chat gee pee tea

All bar the last one are using ways within the English language to split out "Chat" into one word being pronounced and then split the G, P and T into individual letters. The final example, which is to use Phonetics will sound perfectly fine, but clearly would look wrong as far as human readable text goes. The phonetics method is very useful in edge cases where pronunciation difficult.

⬜ Notes on Usage

• For seamless TTS generation, it's advised to keep text chunks under 250 characters, which you can control with the Chunk sizes.
• Generated audio can be played back from the list, which also highlights the currently playing chunk.
• The TTS Generator remembers your settings, so you can pick up where you left off even after refreshing the page.

Back to top of page

🟠 API Suite and JSON-CURL

🟠Overview

The Text-to-Speech (TTS) Generation API allows you to generate speech from text input using various configuration options. This API supports both character and narrator voices, providing flexibility for creating dynamic and engaging audio content.

🟠 Ready Endpoint

Check if the Text-to-Speech (TTS) service is ready to accept requests.

• URL: http://127.0.0.1:7851/api/ready
  - Method: GET
  

  curl -X GET "http://127.0.0.1:7851/api/ready"

  Response: Ready

🟠 Voices List Endpoint

Retrieve a list of available voices for generating speech.

• URL: http://127.0.0.1:7851/api/voices
  - Method: GET
  

  curl -X GET "http://127.0.0.1:7851/api/voices"

  JSON return: {"voices": ["voice1.wav", "voice2.wav", "voice3.wav"]}

🟠 Current Settings Endpoint

Retrieve a list of available voices for generating speech.

• URL: http://127.0.0.1:7851/api/currentsettings
  - Method: GET
  

  curl -X GET "http://127.0.0.1:7851/api/currentsettings"

  JSON return: {"models_available":[{"name":"Coqui","model_name":"API TTS"},{"name":"Coqui","model_name":"API Local"},{"name":"Coqui","model_name":"XTTSv2 Local"}],"current_model_loaded":"XTTSv2 Local","deepspeed_available":true,"deepspeed_status":true,"low_vram_status":true,"finetuned_model":false}

  name & model_name = listing the currently available models.
  current_model_loaded = what model is currently loaded into VRAM.
  deepspeed_available = was DeepSpeed detected on startup and available to be activated.
  deepspeed_status = If DeepSpeed was detected, is it currently activated.
  low_vram_status = Is Low VRAM currently enabled.
  finetuned_model = Was a finetuned model detected. (XTTSv2 FT).
  

🟠 Preview Voice Endpoint

Generate a preview of a specified voice with hardcoded settings.

• URL: http://127.0.0.1:7851/api/previewvoice/
  - Method: POST
  - Content-Type: application/x-www-form-urlencoded
  

  curl -X POST "http://127.0.0.1:7851/api/previewvoice/" -F "voice=female_01.wav"

  Replace female_01.wav with the name of the voice sample you want to hear.

  JSON return: {"status": "generate-success", "output_file_path": "/path/to/outputs/api_preview_voice.wav", "output_file_url": "http://127.0.0.1:7851/audio/api_preview_voice.wav"}

🟠 Switching Model Endpoint

• URL: http://127.0.0.1:7851/api/reload
  - Method: POST

curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20Local"
curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20TTS"
curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=XTTSv2%20Local"
  

  Switch between the 3 models respectively.

  curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=XTTSv2%20FT"
  

  If you have a finetuned model in /models/trainedmodel/ (will error otherwise)

  JSON return {"status": "model-success"}

🟠 Switch DeepSpeed Endpoint

• URL: http://127.0.0.1:7851/api/deepspeed
  - Method: POST

curl -X POST "http://127.0.0.1:7851/api/deepspeed?new_deepspeed_value=True"

  Replace True with False to disable DeepSpeed mode.

  JSON return {"status": "deepspeed-success"}

🟠 Switching Low VRAM Endpoint

• URL: http://127.0.0.1:7851/api/lowvramsetting
  - Method: POST

curl -X POST "http://127.0.0.1:7851/api/lowvramsetting?new_low_vram_value=True"

  Replace True with False to disable Low VRAM mode.

  JSON return {"status": "lowvram-success"}

🟠 TTS Generation Endpoint (Standard Generation)

Streaming endpoint details are further down the page.

• URL: http://127.0.0.1:7851/api/tts-generate
  - Method: POST
  - Content-Type: application/x-www-form-urlencoded
  

🟠 Example command lines (Standard Generation)

Standard TTS generation supports Narration and will generate a wav file/blob. Standard TTS speech Example (standard text) generating a time-stamped file

curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=All of this is text spoken by the character. This is text not inside quotes, though that doesnt matter in the slightest" -d "text_filtering=standard" -d "character_voice_gen=female_01.wav" -d "narrator_enabled=false" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d "language=en" -d "output_file_name=myoutputfile" -d "output_file_timestamp=true" -d "autoplay=true" -d "autoplay_volume=0.8"

Narrator Example (standard text) generating a time-stamped file

curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=*This is text spoken by the narrator* \"This is text spoken by the character\". This is text not inside quotes." -d "text_filtering=standard" -d "character_voice_gen=female_01.wav" -d "narrator_enabled=true" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d "language=en" -d "output_file_name=myoutputfile" -d "output_file_timestamp=true" -d "autoplay=true" -d "autoplay_volume=0.8"

Note that if your text that needs to be generated contains double quotes you will need to escape them with \" (Please see the narrator example).

🟠 Request Parameters

🟠 text_input: The text you want the TTS engine to produce. Use escaped double quotes for character speech and asterisks for narrator speech if using the narrator function. Example:

-d "text_input=*This is text spoken by the narrator* \"This is text spoken by the character\". This is text not inside quotes."

🟠 text_filtering: Filter for text. Options:

• none No filtering. Whatever is sent will go over to the TTS engine as raw text, which may result in some odd sounds with some special characters.
  
• standard Human-readable text and a basic level of filtering, just to clean up some special characters.
  
• html HTML content. Where you are using HTML entity's like "
  

-d "text_filtering=none"
-d "text_filtering=standard"
-d "text_filtering=html"

Example:

• Standard Example: *This is text spoken by the narrator* "This is text spoken by the character" This is text not inside quotes.
  
• HTML Example: *This is text spoken by the narrator* "This is text spoken by the character" This is text not inside quotes.
  
• None: Will just pass whatever characters/text you send at it.
  

🟠 character_voice_gen: The WAV file name for the character's voice.

-d "character_voice_gen=female_01.wav"

🟠 narrator_enabled: Enable or disable the narrator function. If true, minimum text filtering is set to standard. Anything between double quotes is considered the character's speech, and anything between asterisks is considered the narrator's speech.

-d "narrator_enabled=true"
-d "narrator_enabled=false"

🟠 narrator_voice_gen: The WAV file name for the narrator's voice.

-d "narrator_voice_gen=male_01.wav"

🟠 text_not_inside: Specify the handling of lines not inside double quotes or asterisks, for the narrator feature. Options:

• character: Treat as character speech.
  
• narrator: Treat as narrator speech.
  

-d "text_not_inside=character"
-d "text_not_inside=narrator"

🟠 language: Choose the language for TTS. Options:

ar Arabic
zh-cn Chinese (Simplified)
cs Czech
nl Dutch
en English
fr French
de German
hi Hindi
hu Hungarian
it Italian
ja Japanese
ko Korean
pl Polish
pt Portuguese
ru Russian
es Spanish
tr Turkish

-d "language=en"

🟠 output_file_name: The name of the output file (excluding the .wav extension).

-d "output_file_name=myoutputfile"

🟠 output_file_timestamp: Add a timestamp to the output file name. If true, each file will have a unique timestamp; otherwise, the same file name will be overwritten each time you generate TTS.

-d "output_file_timestamp=true"
-d "output_file_timestamp=false"

🟠 autoplay: Enable or disable playing the generated TTS to your standard sound output device at time of TTS generation.

-d "autoplay=true"
-d "autoplay=false"

🟠 autoplay_volume: Set the autoplay volume. Should be between 0.1 and 1.0. Needs to be specified in the JSON request even if autoplay is false.

-d "autoplay_volume=0.8"

🟠 TTS Generation Response

The API returns a JSON object with the following properties:

• status Indicates whether the generation was successful (generate-success) or failed (generate-failure).
  
• output_file_path The on-disk location of the generated WAV file.
  
• output_file_url The HTTP location for accessing the generated WAV file for browser playback.
  
• output_cache_url The HTTP location for accessing the generated WAV file as a pushed download.
  

Example JSON TTS Generation Response:

{"status":"generate-success","output_file_path":"C:\\text-generation-webui\\extensions\\alltalk_tts\\outputs\\myoutputfile_1704141936.wav", "output_file_url":"http://127.0.0.1:7851/audio/myoutputfile_1704141936.wav", "output_cache_url":"http://127.0.0.1:7851/audiocache/myoutputfile_1704141936.wav"}

🟠 TTS Generation Endpoint (Streaming Generation)

Streaming TTS generation does NOT support Narration and will generate an audio stream. Streaming TTS speech JavaScript Example:

• URL: http://localhost:7851/api/tts-generate-streaming
  - Method: POST
  - Content-Type: application/x-www-form-urlencoded
  
  

// Example parameters
        const text = "Here is some text";
        const voice = "female_01.wav";
        const language = "en";
        const outputFile = "stream_output.wav";
        // Encode the text for URL
        const encodedText = encodeURIComponent(text);
        // Create the streaming URL
        const streamingUrl = `http://localhost:7851/api/tts-generate-streaming?text=${encodedText}&voice=${voice}&language=${language}&output_file=${outputFile}`;
        // Create and play the audio element
        const audioElement = new Audio(streamingUrl);
        audioElement.play(); // Play the audio stream directly
        

• Text (text): This is the actual text you want to convert to speech. It should be a string and must be URL-encoded to ensure that special characters (like spaces and punctuation) are correctly transmitted in the URL. Example: Hello World becomes Hello%20World when URL-encoded.
  
• Voice (voice): This parameter specifies the voice type to be used for the TTS. The value should match one of the available voice options in AllTalks voices folder. This is a string representing the file, like female_01.wav.
  
• Language (language): This setting determines the language in which the text should be spoken. A two-letter language code (like en for English, fr for French, etc.).
  
• Output File (output_file): This parameter names the output file where the audio will be streamed. It should be a string representing the file name, such as stream_output.wav. AllTalk will not save this as a file in its outputs folder.
  

Back to top of page

Thanks & References

Coqui TTS Engine

• Coqui License
• Coqui TTS GitHub Repository

Extension coded by

• Erew123 GitHub Profile

Thanks to & Text generation webUI

• Ooobabooga GitHub Repository (Portions of orginal Coquii_TTS extension)

Thanks to

• daswer123 GitHub Profile (Assistance with cuda to cpu moving)
• S95Sedan GitHub Profile (Editing the Microsoft DeepSpeed v11.x installation files so they work)
• kanttouchthis GitHub Profile (Portions of orginal Coquii_TTS extension)
• Wuzzooy GitHub Profile (Trying out the code while in development)

Back to top of page