Contrastive Predictive Coding
The CPC-big model and k-means checkpoints used in Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing.
Contrastive predictive coding (CPC) aims to learn representations of speech by distinguishing future observations from a set of negative examples. Previous work has shown that linear classifiers trained on CPC features can accurately predict speaker and phone labels. However, it is unclear how the features actually capture speaker and phonetic information, and whether it is possible to normalize out the irrelevant details (depending on the downstream task). In this paper, we first show that the per-utterance mean of CPC features captures speaker information to a large extent. Concretely, we find that comparing means performs well on a speaker verification task. Next, probing experiments show that standardizing the features effectively removes speaker information. Based on this observation, we propose a speaker normalization step to improve acoustic unit discovery using K-means clustering of CPC features. Finally, we show that a language model trained on the resulting units achieves some of the best results in the ZeroSpeech2021~Challenge.
Basic Usage
import torch, torchaudio
from sklearn.preprocessing import StandardScaler
# Load model checkpoints
cpc = torch.hub.load("bshall/cpc:main", "cpc").cuda()
kmeans = torch.hub.load("bshall/cpc:main", "kmeans50")
# Load audio
wav, sr = torchaudio.load("path/to/wav")
assert sr == 16000
wav = wav.unsqueeze(0).cuda()
x = cpc.encode(wav).squeeze().cpu().numpy() # Encode
x = StandardScaler().fit_transform(x) # Speaker normalize
codes = kmeans.predict(x) # Discretize
Note that the encode function is stateful (keeps the hidden state of the LSTM from previous calls).
Encode an Audio Dataset
Clone the repo and use the encode.py script:
usage: encode.py [-h] in_dir out_dir
Encode an audio dataset using CPC-big (with speaker normalization and discretization).
positional arguments:
in_dir Path to the directory to encode.
out_dir Path to the output directory.
optional arguments:
-h, --help show this help message and exit
2.2k Jan 9, 2023
11 Nov 17, 2022
5 Dec 28, 2021
29 Oct 16, 2022
8.4k Dec 26, 2022
1.5k Dec 26, 2022
5.9k Jan 3, 2023
5k Jan 02, 2023
14 Jan 03, 2023
1 Jan 06, 2022
11 Sep 29, 2022
109 Dec 21, 2022
42 Mar 16, 2022
3 Apr 19, 2022
1 Nov 08, 2021
0 Dec 22, 2021
19 Nov 26, 2022
30 Dec 10, 2022
47 Sep 10, 2022
70 Dec 26, 2022
592 Dec 27, 2022
4.5k Dec 30, 2022
3 Nov 04, 2022
18 Nov 28, 2022
2 Oct 17, 2021
2 Nov 20, 2021
![[something hillarious here]](https://avatars.githubusercontent.com/u/20630248?v=4&s=60)
8 Jan 07, 2023