Stable Diffusion & OpenAI Whisper: A Guide to Creating Images from Speech

Unlocking Creativity: A Guide to Voice-Activated Image Generation

The world of artificial intelligence is developing incredibly fast! With the latest models, we can now create stunning images from spoken words, opening up endless possibilities for creativity and innovation. In this tutorial, we will walk you through the basics of building your own application that harnesses this groundbreaking technology.

Getting Started

Before diving in, note that this tutorial uses Google Colab for convenience, especially for those without a dedicated GPU. However, feel free to run it on your local machine, provided you have a GPU available!

Step 1: Install Necessary Dependencies

We need to install FFmpeg, a powerful tool to record, convert, and stream audio and video. After that, we will install other required packages. If you encounter any issues installing Whisper, you can refer here for guidance.

Step 2: Authenticate with Hugging Face

Next, we will authenticate our Stable Diffusion access with Hugging Face. This step is critical for enabling image generation from text.

Step 3: Check GPU Availability

Before proceeding, it's important to check if we are using a GPU. If everything is set, we are ready to start coding!

Coding Your Application

Speech to Text Conversion

We will begin by converting speech to text. To save time, I recorded my prompt and stored it in the main directory. Using OpenAI's Whisper small model, we will extract the spoken prompt. There are various sizes of models available, so feel free to choose based on your requirements.

Extracting the Text

For the extraction process, I utilized code from the official repository and added some "tips" to enhance the prompt further.

Text to Image Generation

Now, we will transition from text to images using Stable Diffusion. First, we'll load the model.

Using the processing pipeline, we will generate an image from the text extracted from our voice.

View the Results!

Let’s check the generated results. While we may not have fine-tuned every parameter, the main achievement here is the ability to create images directly from voice prompts. Isn’t that amazing? When reflecting on where we were a decade ago and considering the advancements of today, it’s truly inspiring!

Conclusion

Thank you for joining me in this venture to create a voice-activated image generator! I hope you had as much fun as I did while coding this application. Be sure to check back for more exciting tutorials and updates in the field of artificial intelligence!

— Jakub Misio, Junior Data Scientist at New Native