On-Device Piano Autocomplete: 125M Model Hits 108 Notes/Sec

TL;DR

Simon Edwardsson, a developer and researcher, has released RollTab, a free iOS app that turns a MIDI piano into a generative music instrument. The app runs a 125M-parameter transformer entirely on-device, predicting piano continuations at ~108 notes per second on an iPhone 15. The project, which hit the top of Hacker News with 434 points, demonstrates that with careful representation design and post-training, small models can deliver compelling real-time music generation.

The biggest performance leaps came not from scaling, but from three decisions: a compound note tokenization that reduces autoregressive steps by 5x, aggressive data cleaning over sheer volume, and Direct Preference Optimization (DPO) after pretraining. Edwardsson's detailed write-up offers rare transparency into the failures and iterative process behind a working on-device music AI.

The Challenge: Tokenizing Music for Transformers

MIDI files store music as discrete events—note-on, note-off, pitch, velocity, sustain pedal changes—rather than audio waveforms. The naive approach of tokenizing each event separately leads to vocabulary explosion (e.g., 128 pitches × 128 velocities) and models that 'drift,' forgetting to emit note-off events and leaving hanging notes.

Edwardsson's breakthrough was a compound note representation: each token represents a complete note with five categorical fields—event type, pitch, delta onset (time since previous note), duration, and velocity. Chords are encoded as multiple notes with delta=0, sorted by pitch. This means the transformer advances one full note per forward pass, rather than four or more passes for attributes, yielding a 5x speedup.

Internally, the note token is the sum of field-specific embeddings, with separate output heads for each field and a small nested decoder for field conditioning. Timing is quantized to 24 steps per quarter note, balancing resolution for live play with manageable vocabulary size.

Data: Quality Over Quantity

The training dataset comprised a few hundred thousand MIDI files (~300 million note events) from public domain classical sources. Edwardsson wrote extensive cleaning scripts to filter piano-focused material, deduplicate by transposition/tempo-invariant fingerprints, and group alternate versions of the same composition into the same split.

Scaling the dataset 5x actually hurt performance—a counterintuitive finding that underscores how noisy public MIDI collections are. 'Cleaning and selecting the data mattered more than simply adding more of it,' he notes.

Training and the DPO Leap

The base model was trained with cross-entropy over the five output heads, with scheduled sampling between fields (feeding the model's own predicted pitch up to 50% of the time) to mitigate train/inference mismatch. While this raised validation loss, it improved rollout quality significantly.

The most impactful phase was DPO (Direct Preference Optimization). Edwardsson used Gemini 3.5 Flash for pairwise evaluation—asking 'which continuation is better?' rather than absolute scores—and built a preference dataset. DPO with β=0.03 on a consensus-filtered dataset led to a 69% preference win rate over the base model, versus 24.55% for the base.

Key failures documented: note-on/off drift, grammar-masked streams being slow, Mirostat causing incoherence, and validation loss alone missing rollout quality differences.

On-Device Deployment and Future

The model (three sizes trained: small, medium, large) was exported to Core ML and quantized to INT8. The medium model is being tuned to approach large-model quality for reduced footprint. Context management uses a sliding window of 384 notes with KV cache rebuild.

RollTab is free on the App Store, with a pending update adding sampling options (top-k, top-p, min-p, XTC, top-h, Mirostat v2). Edwardsson's work offers a blueprint for others tackling on-device generative AI, showing that thoughtful representation and data curation can trump raw compute.

Related research cited: Aria, Moonbeam, MIDI-GPT, Anticipatory Music Transformer, PianoBART, MIDI-LLM.