Paper companion page
AudioCodex: Streaming Autoencoders
for Live Audio
1Sony Computer Science Laboratories Paris 2Queen Mary University of London
Abstract
We introduce AudioCodex, a fully causal autoencoder for live generative audio applications that compresses stereo audio into compact continuous latent sequences at ~10 Hz. It combines a Mean Flow magnitude autoencoder with latent hypersphere projection, bounded-noise augmentation, representation alignment, and adversarial phase recovery to learn representations that support both efficient downstream generation and high-quality reconstruction. Through extensive ablations, we show that AudioCodex outperforms the state-of-the-art in terms of generation-reconstruction trade-off, whilst remaining strictly causal in both encoding and decoding. This work paves the way towards truly real-time interactive and streaming generative audio applications.
Architecture
A causal encoder compresses the magnitude spectrogram into compact, sphere-normalized latents. The decoder upsamples the latents and generates the magnitude with a Mean Flow model, before a separate phase estimator recovers the waveform.
Baselines
We compare AudioCodex against continuous audio autoencoders, with a discrete codec as a reference. All models are evaluated on the same held-out clips. Please refer to the paper for details on the baselines and the evaluation datasets.
- DAC
- Descript Audio Codec: a convolutional codec with residual vector quantization, included as a codec reference.
- Music2Latent
- Consistency autoencoder operating on complex STFT representations.
- CoDiCodec
- STFT-domain consistency autoencoder, evaluated in its continuous mode.
- SAO
- Stable Audio Open: a convolutional waveform VAE.
- AudioCodex (ours)
- Fully causal Mean Flow magnitude autoencoder with adversarial phase recovery, at latent dimensions d = 128, 256 and 512.
Reconstruction Examples
Each row is one 10-second stereo clip from the evaluation set, encoded and decoded by every model. Within a row, playback position is carried over when you switch between models, so systems can be compared at the same point in the clip.
Full Evaluation Results
Due to space constraints, the main paper mostly reports results averaged over all audio categories. Here we provide the full evaluation results for every category, together with a fine-grained latency measurement for each component of the model.
