LiveBand: Live Accompaniment Generation in the Audio Domain

Marco Pasini², Javier Nistal¹, Mathias Rose Bjare³, Stefan Lattner¹, George Fazekas²

1. Sony Computer Science Laboratories Paris
2. Queen Mary University of London
3. Johannes Kepler University Linz

Paper: arXiv

Abstract
We present LiveBand, a real-time system that generates high-fidelity music accompaniments to live audio input, respecting strict causal constraints. Our method trains a causal transformer generator in the continuous latent space of a pre-trained causal audio autoencoder, using adversarial sequence-level supervision from a discriminator. At each timestep, the generator receives only the causally available mix context and Gaussian noise, and predicts accompaniment latents without access to future mix frames or ground-truth target latents. Training is performed in a single parallel forward pass under causal masking, while streaming inference proceeds autoregressively with a rolling attention state. The model’s training and inference computations are matched by design, eliminating teacher forcing and the associated exposure bias. On a multi-instrument music accompaniment benchmark, LiveBand improves over prior work on objective measures of audio quality, beat alignment, and mix adherence, while enabling real-time streaming generation without lookahead into the future on consumer hardware.

Framework

Train-Test Equivalence

Audio Examples

We highly recommend listening with headphones. All examples use Slakh2100 test samples.

Showcase: Internal LiveBand (0.1s)

We showcase some examples generated by the internal-data LiveBand model. Note how the model produces accompaniments that are initially incoherent (the model already starts generating with absolutely no context about the incoming mix), and then quickly adapts once it has enough context about the mix.

LiveBand Family Comparison

We compare the internal LiveBand model (0.1s anticipation) with Slakh2100-trained models at 0s, 0.1s, 1s, and the bidirectional variant. The bidirectional model outputs 10s, while all other samples are 20s.

StreamMusicGen Baselines

We show StreamMusicGen baselines with 0s and 1s anticipation. Note how the mix-adherence degrades over time because of compounding errors.