It’s Time for Artistic Correspondence in Music and Video

Columbia University
Adobe
CVPR 2022

Captions are available in the video options.

Abstract

Teaser figure

We present an approach for recommending a music track for a given video, and vice versa. We model the long-term temporal context of both signals, allowing our model to capture the high-level artistic correspondences between them. Our model learns a strong audiovisual representation that allows us to retrieve videos and music that look and sound natural to humans. On the left we show query video segments with the corresponding retrieved music segments, and on the right we show the opposite retrieval direction. Our model's audiovisual correspondence exploits artistic attributes such as music genre or rhythm.

Paper

PDF
@article{suris_musicforvideo,
  title={It’s Time for Artistic Correspondence in Music and Video},
  author={Sur\'is, D\'idac and Vondrick, Carl and Russell, Bryan and Salamon, Justin},
  journal={Computer Vision and Pattern Recognition (CVPR)},
  year={2022}
}

Contributions

Poster

Poster

Acknowledgements

The webpage template was inspired by this project page.