CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audio

Marco Pasini¹, Stefan Lattner², George Fazekas¹

1. Queen Mary University of London
2. Sony Computer Science Laboratories Paris

Paper: arXiv Github: Code

Abstract
Efficiently representing audio signals in a compressed latent space is critical for latent generative modelling. However, existing autoencoders often force a choice between continuous embeddings and discrete tokens. Furthermore, achieving high compression ratios while maintaining audio fidelity remains a challenge. This paper introduces a novel audio autoencoder that overcomes these limitations by both efficiently encoding global features via summary embeddings, and by producing both compressed continuous embeddings at ~11 Hz and discrete tokens at a rate of 2.38 kbps from the same trained model, offering unprecedented flexibility for different downstream generative tasks. This is achieved through Finite Scalar Quantization (FSQ) and a novel FSQ-dropout technique, and does not require additional loss terms beyond the single consistency loss used for end-to-end training. Our model supports both autoregressive decoding and a novel parallel decoding strategy, with the latter achieving superior audio quality and faster decoding. Our model outperforms existing continuous and discrete autoencoders at similar bitrates in terms of reconstruction audio quality. Our work enables a unified approach to audio compression, bridging the gap between continuous and discrete generative modelling paradigms.

Architecture

Training process of our autoencoder. Transformer modules are represented with T, audio embeddings with A, learned/summary embeddings with L, and mask embeddings with M. We represent chunked causal masking with a curved arrow.

Comparison Table

In the following table we present the evaluation results on pairwise (SI-SDR, ViSQOL) and distance-based (FAD-clap, FAD) metrics.

Audio Examples

We compare the reconstructions of our model against baselines for MusicCaps evaluation samples. We provide reconstructions of continuous embeddings and of discrete tokens at 2.38 kbps, using parallel decoding with 4 steps.

Stereo reconstructions are provided for stereo-capable models: Mousai, Music2Latent2, Stable Audio, and Ours.