AudioCLIP
Extending CLIP to Image, Text and Audio
This repository contains implementation of the models described in the paper arXiv:2106.13043. This work based on our previous works:
- ESResNe(X)t-fbsp: Learning Robust Time-Frequency Transformation of Audio (2021).
- ESResNet: Environmental Sound Classification Based on Visual Domain Models (2020).
Abstract
In the past, the rapidly evolving field of sound classification greatly benefited from the application of methods from other domains. Today, we observe the trend to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models.
In this work, we present an extension of the CLIP model that handles audio in addition to text and images. Our proposed model incorporates the ESResNeXt audio-model into the CLIP framework using the AudioSet dataset. Such a combination enables the proposed model to perform bimodal and unimodal classification and querying, while keeping CLIP's ability to generalize to unseen datasets in a zero-shot inference fashion.
AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task, out-performing other approaches by reaching accuracies of 90.07% on the UrbanSound8K and 97.15% on the ESC-50 datasets. Further it sets new baselines in the zero-shot ESC-task on the same datasets (68.78% and 69.40%, respectively).
Finally, we also assess the cross-modal querying performance of the proposed model as well as the influence of full and partial training on the results. For the sake of reproducibility, our code is published.
How to Run the Model
The required Python version is >= 3.7.
AudioCLIP
On the ESC-50 dataset
python main.py --config protocols/audioclip-esc50.json --Dataset.args.root /path/to/ESC50
On the UrbanSound8K dataset
python main.py --config protocols/audioclip-us8k.json --Dataset.args.root /path/to/UrbanSound8K
Cite Us
@misc{guzhov2021audioclip,
title={AudioCLIP: Extending CLIP to Image, Text and Audio},
author={Andrey Guzhov and Federico Raue and Jörn Hees and Andreas Dengel},
year={2021},
eprint={2106.13043},
archivePrefix={arXiv},
primaryClass={cs.SD}
}
1 Aug 19, 2021
612 Jan 4, 2023
2 Mar 4, 2022
82 Dec 26, 2022
2.3k Dec 29, 2022
23 Nov 30, 2022
39 Oct 27, 2022
4.8k Dec 30, 2022
1 Dec 03, 2021
2 Oct 26, 2021
2 Feb 17, 2022
13 Oct 10, 2022
363 Dec 30, 2022
1.6k Dec 27, 2022
45 Sep 06, 2022
276 Dec 26, 2022
6 Oct 18, 2022
12 Dec 14, 2022
12 Aug 19, 2021
5.9k Jan 03, 2023
22 Nov 16, 2022
230 Nov 26, 2022
9 Nov 17, 2022
5 Oct 24, 2022
1 Feb 04, 2022
1 Feb 11, 2022
3 Apr 15, 2022
1 Mar 16, 2022