Librosa Tutorial: Audio Waveforms & Spectrograms in Python

Audio data augmentation is a technique that increases the number of training samples by applying transformations like reverb, pitch shifting, and time stretching to existing audio files, which helps address data scarcity, improve model robustness, reduce overfitting, and save resources; however, it should only be applied to the training set (not validation or test sets), and the augmented audio must remain credible to maintain meaningful learning signals.

Data augmentation is crucial for audio classification, especially when working with limited annotated data. Common techniques include: (1) Time shifting—shifting audio samples left or right to ensure the model can recognize sounds regardless of their position within analysis windows; (2) Time stretching—slowing down or speeding up audio to capture variations like speaking fast or slow; (3) Pitch shifting—changing pitch to account for variations in vocal tone; (4) Adding noise—mixing in random noise to improve robustness; (5) Mixup—linearly combining two audio samples and adjusting labels proportionally. Time shifting is typically the most important technique.

This video demonstrates how to implement five key audio data augmentation techniques in Python: (1) White noise addition using Gaussian distribution with a noise factor multiplier, (2) Time stretching using librosa.effects.time_stretch with a stretch rate parameter, (3) Pitch scaling using librosa.effects.pitch_shift with semitone parameters, (4) Polarity inversion by multiplying the signal by -1, and (5) Random gain using random.uniform to sample gain factors between min and max values. The implementation uses Python libraries including numpy, librosa, soundfile, and matplotlib for visualization, with practical examples showing how to load audio files, apply transformations, save augmented files, and plot waveforms to compare original and augmented signals.

Data augmentation techniques improve model robustness by artificially increasing training data diversity. For audio, common augmentations include adding noise, changing pitch, and applying time stretching. These transformations simulate real-world variations in recording conditions and speaker characteristics. The augmented data helps the model generalize better to unseen audio samples while maintaining the underlying semantic content.

Data augmentation is a technique to create more training data from existing data by applying transformations, which is essential for deep learning models that require large datasets. In the audio domain, augmentation techniques are divided into time-domain methods (such as stretching, pitch/loudness changes, and mixup) and frequency-domain methods (such as SpecAugment, which applies frequency and time masking to spectrograms). Python libraries like Audiomentations provide easy-to-use implementations for applying these augmentations, with visualization tools to inspect the transformed audio data. When applying augmentations, it is important to ensure that critical information in the audio is not destroyed.

Convolutional Neural Networks (CNNs) can be applied to audio classification by treating spectrograms as 2D images. Using a 3x3 convolutional filter with pooling layers, a CNN achieved over 90% accuracy (93-94%) on the test data. This demonstrates that CNN architectures developed for image processing can be directly transferred to audio classification tasks when audio data is represented as spectrograms.

Speech recognition can be transformed into an image recognition problem by converting audio to spectrograms, where the x-axis represents time, y-axis represents frequency, and color intensity represents amplitude. This allows applying CNN techniques to speech. The demonstration shows converting WAV files to spectrograms, where different digits produce distinct visual patterns. Isolated word recognition is simpler than continuous speech because word boundaries are clear. Large datasets like TIMIT contain recordings of 630 speakers reading sentences with all 43 English phonemes. The network learns to recognize digits from spectrogram images, achieving reasonable accuracy after training on labeled examples.

Spectrograms are generated using the Short-Time Fourier Transform (STFT) via tf.signal.stft(). Parameters include frame_length (320 samples) and frame_step (32 samples), controlling time-frequency resolution trade-offs. The output is converted to absolute values (removing negative amplitudes) and expanded to add a channel dimension (becoming shape [time, frequency, 1]). This format matches the expected input shape for convolutional neural networks, enabling image-based processing of audio data.

Spectrograms transform audio into visual representations showing time (x-axis), frequency/pitch (y-axis), and amplitude/gain (color). This allows convolutional neural networks to process audio as images. The AlexNet architecture, originally trained on ImageNet, uses convolutional layers for feature extraction, max pooling for dimensionality reduction, and fully connected layers for classification. The model uses cross-entropy loss with a soft-max classifier, training via backpropagation. Window shape affects feature extraction: 1xN windows capture time features (tempo, rhythm), while Nx1 windows capture frequency features. This approach is powerful for binary classification problems with nuanced differences between classes.

This video demonstrates how to implement a Convolutional Neural Network (CNN) in TensorFlow for music genre classification, covering the complete workflow from preparing train/validation/test datasets to building a CNN with three convolutional layers, max pooling, batch normalization, and dense layers, followed by model compilation, training, evaluation on the test set, and making predictions on single samples using inference.

Harmonic-Percussive Source Separation (HPSS) is an audio processing technique that separates mono sound signals into harmonic and percussive components by analyzing spectrograms. The algorithm applies median filters in two directions: vertically across time to enhance harmonic content by removing percussive transients, and horizontally across frequency bins to enhance percussive elements by dampening sustained harmonics. The filtered spectrograms are then compared to create binary or soft masks that determine which source dominates at each frequency-time point. The original spectrogram is multiplied by these masks, and inverse FFT is applied using original phases to reconstruct the separated audio components. Binary masks produce tighter separation with more artifacts, while soft masks provide smoother transitions with slightly more ringing but better overall sound quality.

This extensive section explores advanced audio processing techniques for creative performance systems. It covers three-way Harmonic-Percussive Source Separation (HPSS) extended to include transient components, comparing HPSS with NMF and finding HPSS produces more usable results for creative applications. The section details component-specific DSP routing: harmonic components through envelope shaping and overdrive, percussive components through wave folders, and transients through filters and wave shaping. Audio descriptors from each onset dynamically modulate parameters including duration and treble levels. The system implements real-time granular synthesis by writing grains to a three-channel buffer (harmonic, percussive, transient) with independent playheads, controlled by a blend parameter functioning like wet/dry mix. Practical limitations including minimum grain size constraints are addressed through clipping and parameter massaging. This demonstrates how source separation serves as a creative tool rather than an objective analysis method, enabling dynamic, context-aware sound manipulation.

There are two main approaches to onset detection: (1) DSP Methods - involve pre-processing audio to create an activation function (such as energy envelope, high-frequency components, or spectrogram changes) that makes onsets clearly visible, followed by peak picking to identify event starts; (2) Machine Learning Methods - use annotated data to train binary classifiers that scan short audio sections to detect whether an onset exists, creating a state-of-the-art activation function through learned features. The ML approach often achieves better accuracy while maintaining simplicity through threshold-based post-processing.

Transient Extract uses two algorithms: HPSS (Harmonic-Percussive Source Separation) and Transients. HPSS separates audio into harmonic and percussive components based on frequency content, providing larger transient-like sections. Transients algorithm captures sharp attack points. Both algorithms null-sum, meaning combining processed components reconstructs the original audio without loss. This enables selective processing of either harmonic or percussive components separately.

Librosa is a Python library for audio and music signal analysis that provides essential tools for extracting musically relevant features such as spectrograms, chroma, MFCCs, onset detection, beat tracking, and harmonic-percussive source separation, enabling researchers to analyze audio content for applications like genre classification, transcription, and structural analysis.