AI Training - Tutorial - Compare models with W&B for audio classification task

Last updated 31st January, 2023.

Objective

The purpose of this tutorial is to compare two methods by running two jobs in parallel in order to classify audios. To see which model is better in terms of accuracy, resource consumption and training time, we will use the Weights & Biases tool.

Use case

The use case is Spoken Digit Database. It is available on Kaggle.

The database contains spoken digit files from zero to nine that have been recorded by different persons. All contain 1700 speech digits files.

Database License: Attribution 4.0 International (CC BY 4.0)

AI Models

To build these sound classifiers, we will use two methods.

Audio classification based on audios feature

The first method is to create an Artificial Intelligence (AI) model to classify audio files using the different features of sounds.

In order to do this, some data processing is required upstream. Each sound will constitute a line of a csv file thanks to its transformation into 26 parameters calculated by Librosa.

An Artificial Neural Network (ANN) is then built and trained on 100 epochs. It takes the 26 parameters calculated by Librosa as input and returns a probability for each class as output.

Image classification based on spectrograms

The second method is to create an image classification model using the spectrograms of each sound.

The data must be processed beforehand. From each sound a spectrogram (an image) is generated using the Python module Librosa. A Convolutional Neural Network (CNN) is then built and trained on 100 epochs.

It takes as input the spectrograms whose size is defined and returns as output a probability for each class.

Comparaison tool

Two Artificial Intelligence models of different natures are trained to perform the same task: to classify audio recordings of people speaking numbers from zero to nine.

To compare them, the Weights and Biases tool is used. It makes it easy to track and record the performance of deep learning models.

With Weights & Biases, it is possible to build better models faster through experiment tracking, dataset versioning and model management.

In our case, we will be able to track the evolution of different models based on the values of accuracies and losses. The tool also offers us the possibility to visualise the training times and the consumption of resources (GPU).

To know more about Weights & Biases, please refer to the documentation.

The basic principles for using Weights & Biases can be found here with AI Notebooks.

Requirements

• Access to the OVHcloud Control Panel;
• An AI Training project created inside a Public Cloud project in your OVHcloud account;
• A user for AI Training;
• Docker installed on your local computer;
• Make sure you have a Docker Hub account;
• Some knowledge about building image and Dockerfile;
• A Weights & Biases account, you can create it on their website. It’s Free for individuals.

Instructions

You will follow different steps to process your data and train your two models.

• More detailed data processing in this notebook concerning the classification of marine mammal sounds.
• Direct link to the full Python files can be found here.

The tutorial is as follows:

Here we will mainly discuss how to write the data processing and models training codes, the requirements.txt and packages.txt files and the Dockerfile. If you want to see the whole code, please refer to the GitHub repository.

Clone the GitHub repository

The first thing to do is to clone the GitHub repository.

git clone https://github.com/ovh/ai-training-examples

You can then place yourself in the dedicated directory.

cd ai-training-examples/jobs/weights-and-biases/audio-classification-models-comparaison

Uploading your dataset on Public Cloud Storage

It's a zip file (audio_files.zip)! We are going to push it into an object container named spoken-digit.

If you want to upload it from the OVHcloud Control Panel, go to the Object Storage section and create a new object container by clicking Object Storage > Create an object container.

If you want to run it with the CLI, just follow this guide. You have to choose the region, the name of your container and the path where your data is located and use the following command:

ovhai data upload <region> <container> <paths>

You should have:

├── spoken-digit
    └── audio_files.zip
    └── audio_files
        └── zero
        └── one
        └── two
        └── three
        └── four
        └── five
        └── six
        └── seven
        └── eight
        └── nine

Write the data processing Python files

For the data processing part, we distinguish two Python files.

Audio to csv file with features extraction

The first Python file is called data-processing-audio-files-csv.py. It allows to transform all the sounds into Librosa parameters and to make a csv file.

Refer to the comments of the code for more information.

The head of the csv file:

Audio to spectrogram with image generation

The first Python file is called data-processing-audio-files-spectrograms.py. It allows you to obtain a spectrogram (an image) corresponding to each sound.

Refer to the comments of the code for more information.

A sample spectrogram:

Once the processing of the data is complete, the AI models must be built.

Write the models training Python files

For the models training part, we distinguish two Python files.

ANN for audio classification based on sounds feature

An Artificial Neural Network is built to classify audios based on their features.

It takes as input the 26 Librosa parameters previously normalized.

The model returns as output a score between 0 and 1 for each class through a softmax activation function. The class with the highest score is likely to be the one corresponding to the pronounced number.

Refer to the comments of the code for more information.

CNN for image classification based on spectrograms

A Convolutional Neural Network is constructed to classify images that are in fact spectrograms.

The advantage of using CNNs is their ability to develop an internal representation of a two dimensional image. This allows the model to learn the position and scale in the different data structures, which is important when working with images.

It takes as input the spectrograms previously processed by the Keras data generator for image classification.

As previously, The model returns as output a score between 0 and 1 for each class through a softmax activation function.

Refer to the comments of the code for more information.

Write the requirements.txt and packages.txt files

The requirements.txt file will allow us to write all the modules needed to make our application work.

matplotlib==3.5.2
pandas==1.4.3
split-folders==0.5.1
opencv-python-headless==4.5.5.64
librosa==0.8.0
tensorflow==2.9.1
wandb==0.12.21

The packages.txt file will allow us to install and use the Librosa module and its dependencies.

libsndfile1-dev

These files will be useful when writing the Dockerfile.

Write the Dockerfile for the application

Your Dockerfile should start with the FROM instruction indicating the parent image to use. In our case we choose to start from a python:3.9 image:

FROM python:3.9

Create the home directory and add your files to it:

WORKDIR /workspace
ADD . /workspace

Install the packages.txt file which contains your needed Python modules using a apt-get install ... command:

RUN apt-get update
RUN xargs -a packages.txt apt-get install --yes

Install the requirements.txt file which contains your needed Python modules using a pip install ... command:

RUN pip install --no-cache-dir -r requirements.txt

Give correct access rights to the OVHcloud user (42420:42420):

RUN chown -R 42420:42420 /workspace
ENV HOME=/workspace

Build the Docker image from the Dockerfile

Launch the following command from the Dockerfile directory to build your application image:

docker build . -t <your-docker-id>/audio-classification-models:latest

Push the image into your Docker Hub

docker push <your-docker-id>/audio-classification-models:latest

Launch the jobs

Jobs are launched in two stages. First, the data processing jobs are launched. Once they are Done, the training jobs can be executed.

Data processing

• Audio to csv file with features extraction:

To run this job, you need to plug in a volume containing your sounds. Once the job is in Done status, your csv file will be synchronized to your Object Storage.

ovhai job run <your-docker-id>/audio-classification-models:latest \
      --cpu 12 \
      --volume <my-data>@<region>/:/workspace/data:RW:cache \
      -- bash -c 'python data-processing/data-processing-audio-files-csv.py'

• Audio to spectrogram with image generation:

To run this job, you need to plug in a volume containing your sounds. Once the job is in Done status, your csv file will be synchronized to your Object Storage.

ovhai job run <your-docker-id>/audio-classification-models:latest \
      --cpu 12 \
      --volume <my-data>@<region>/:/workspace/data:RW:cache \
      -- bash -c 'python data-processing/data-processing-audio-files-spectrograms.py'

At the end of the data processing, your Object Storage container should be as follows:

├── spoken-digit
    └── audio_files.zip
    └── audio_files
        └── zero
        └── one
        └── ...
        └── nine
    └── csv_files
        └── data_3_sec.csv
    └── spectrograms
        └── zero
        └── one
        └── ...
        └── nine
    └── spectrograms_split
        └── train
            └── zero
            └── one
            └── ...
            └── nine
        └── val
            └── zero
            └── one
            └── ...
            └── nine

To get the status of your jobs, run the following command:

ovhai job get <job-id>

Once your data has been pre-processed and both jobs are in Done status, you will be able to start your two training jobs.

Models training

• ANN for audio classification based on audios feature:

To run this job, you need to plug in a volume containing your sounds. Once the job is in Done status, your csv file will be synchronized to your Object Storage.

ovhai job run <your-docker-id>/audio-classification-models:latest \
      --gpu 1 \
      --volume <my-data>@<region>/:/workspace/data:RO:cache \
      -- bash -c 'python models-training/train-classification-audio_files_csv.py'

• CNN for image classification based on spectrograms:

To run this job, you need to plug in a volume containing your sounds. Once the job is in Done status, your csv file will be synchronized to your Object Storage.

ovhai job run <your-docker-id>/audio-classification-models:latest \
      --gpu 1 \
      --volume <my-data>@<region>/:/workspace/data:RO:cache \
      -- bash -c 'python models-training/train-image-classification-audio-files-spectrograms.py'

You can now compare your models with Weights & Biases.

Compare with Weights & Biases

You will be able to check your models training once your jobs are in running status. Run the following command:

ovhai job get <job-id>

Once the jobs are in running status, you can check the logs to obtain the Weight & Biases link. Run the command:

ovhai job logs <job-id>

Now, you can access the Weights & Biases panel. You will be able to check the accuracy and the loss values for the training and the validation sets.

• Training data:

Accuracy:

Loss:

• Validation data:

Accuracy:

Loss:

You can then observe which model is better in terms of speed, accuracy or resource consumption...

In this case, we see that the model classifying the spectrograms is better in terms of accuracy and loss on the validation set.

However, it takes longer to train and consumes more computing resources.

Go further

• To build an app to classify audios, refer to this tutorial.
• Do you want to know how to build and use custom Docker image with AI Training? Here it is.

Feedback

Please send us your questions, feedback and suggestions to improve the service:

• On the OVHcloud Discord server

Inne przewodniki, które mogą Cię zainteresować...