OpenAI Whisper Tutorial: Building a Speaker Identification App using Pyannote

Discovering OpenAI Whisper: A Breakthrough in Speech Recognition

OpenAI has introduced Whisper, a cutting-edge speech recognition system that has been meticulously fine-tuned with over 680,000 hours of multilingual and multitask supervised data from the web. This extensive dataset significantly improves Whisper's ability to handle accents, background noise, and specialized language, making it a robust solution for transcription in multiple languages. Additionally, Whisper can even translate content into English, adding another layer of functionality.

Despite its impressive capabilities, Whisper does face challenges in identifying individual speakers during conversation. Diarization—the process of determining who is speaking when—is essential for analyzing conversations effectively. In this OpenAI Whisper tutorial, we will explore how to recognize speakers and align them with Whisper transcriptions using pyannote-audio. Let’s dive in!

Preparing the Audio

The initial step involves preparing the audio file for processing. For this, we will utilize the first 20 minutes of the Lex Fridman podcast featuring Yann LeCun. To download the video and subsequently extract the audio, we will employ the yt-dlp package, along with ensuring that ffmpeg is installed on our system.

The process is straightforward, requiring just a few commands in the terminal. Once we've executed these commands, we will have a file named download.wav in our working directory. Next, we will use the pydub package to trim this audio file to the first 20 minutes, resulting in a file named audio.wav.

Diarization with Pyannote

pyannote.audio is an open-source toolkit developed in Python specifically for speaker diarization. Built on the PyTorch machine learning framework, this toolkit provides a series of trainable building blocks that can create and optimize speaker diarization processes.

With the pretrained models and pipelines available in pyannote.audio, users can effectively perform tasks such as voice activity detection, speaker segmentation, and overlapped speech detection.

To begin, we will install Pyannote and run it using the audio extracted from the podcast. After running the diarization process, we'll review the output, which typically displays segments detailing the start and end times for each speaker's contributions, alongside an indicator of whether the speaker is Lex or not.

Preparing the Audio File from Diarization

Following the diarization process, we will organize the audio segments based on the diarization data, using a spacer as the delimiter to ensure clarity in speaker transitions.

Transcription with Whisper

With the audio segments prepared, we will now proceed to transcribe these segments using Whisper. However, it’s important to note that there might be a version conflict with pyannote.audio that could present an error. Our solution is to run Pyannote first and then execute Whisper, allowing for seamless transcription without concern for the error.

The installation of OpenAI Whisper can be done easily. Once installed, we run Whisper on the prepared audio file, which writes the transcription into a specified file. Users can customize the model size according to their needs by referring to the model card on GitHub.

For processing .vtt files effectively, we will also need to install the webvtt-py library.

Matching Transcriptions with Diarizations

Next, we will match each line of transcription with the corresponding diarization segments. This will be displayed in a structured format by generating an HTML file. To ensure accurate timing representation, we will also incorporate any audio sections where no speaker was identified.

Practical Applications of This Knowledge

One exciting application of these skills could involve developing a Whisper app during an AI hackathon, collaborating with like-minded innovators from around the globe. Following the development phase, participants can apply to New Native's Slingshot program for further project acceleration. Eventually, the goal would be to launch the solution commercially and address relevant global challenges with AI.

Alternatively, individuals might opt to internalize these skills and set their projects aside, letting others lead the charge for change. While this option is feasible, it is not particularly recommended.

Over 54,000 people have participated in lablab.ai's AI Hackathons globally across diverse fields, leading to the creation of over 900 prototypes. This community is continually growing, making it the perfect opportunity to connect with fellow AI builders and contribute to meaningful innovations!