Specialist audio-to-MIDI: the boundaries of GAPS, Filo, VioPTT and SwiftF0
Event CRNNs, F0 segmentation and technique-aware multi-task transcription, with dataset boundaries kept explicit.
AUTOMATIC MUSIC TRANSCRIPTION
GAPS, FiloSax, FiloBass, VioPTT and SwiftF0 all emit MIDI, but their input distributions, labels and matching tolerances differ. Route the instrument first; compare only inside the corresponding dataset.
Event F1
After matching notes inside an onset tolerance, P=TP/(TP+FP), R=TP/(TP+FN) and F1=2PR/(P+R). Offset-aware matching is stricter, so every F1 needs its tolerance and matching definition.
Three modelling families
GAPS/Filo predict onset, frame, offset and velocity events. VioPTT adds violin-technique heads. SwiftF0 segments a monophonic F0 contour with hysteresis, duration and pitch quantisation. None is a full-mix polyphonic AMT model.
Processing chain
A song should first be routed through an appropriate stem separator. Feeding the full mix directly to a single-instrument model can produce a MIDI file, but that output is outside the published evaluation distribution.
How transcription methods evolved
Audio-to-MIDI has no single model that covers every instrument and metric. GAPS targets guitar; FiloSax, FiloBass and VioPTT exploit instrument-specific range and articulation; SwiftF0 is closer to general pitch tracking. Shared file formats do not make their training distributions equivalent.
The field moved from framewise pitch detection to joint onset, offset, pitch and velocity prediction, then toward sequence decoding of complete note events. Progress is visible in repeated notes, sustain, slides and vibrato—not just a larger F1. Paper scores remain incompatible when tolerance windows and matching rules differ.
Event transcription must score both onset and offset
F1=2PR/(P+R), P=TP/(TP+FP), R=TP/(TP+FN)TPevents matching that paper's toleranceFPextra predicted eventsFNmissed events
Papers define true positives differently: some require onset tolerance only; others also require offset, pitch or technique. An F1 value is not comparable once its matching rule is removed.
GAPS, FiloSax, FiloBass, VioPTT and SwiftF0 target different instruments and annotation levels. Correct routing matters more than combining maxima from unrelated datasets.
Paper, code and weights can be followed through Unified specialist runtime / GAPS paper / VioPTT.
Compare only after event definitions match
Each row retains its source architecture, metric or task definition; results without a shared protocol remain separate.
| Model | Architecture or evidence | Scope |
|---|---|---|
| GAPS | piano notes in pedal context | dense chords, repeated notes, sustain |
| FiloSax / FiloBass / VioPTT | instrument-specific data and range | slides, soft attacks, accompaniment bleed |
| SwiftF0 | continuous fundamental-frequency track | requires a separate note segmentation step |
Aural review of note events
Piano needs sustain and repeated-note checks; saxophone and violin need slides, ornaments and vibrato; bass is easily confused with kick drums and low harmonics. A synthesizer can hide timing errors, so inspect the piano roll against the waveform.
Event F1 changes with the tolerance window. A 50 ms onset match, strict onset match, offset-aware note score and frame F1 answer different questions. The tool records model, threshold and quantization so results can be recomputed.
Musical information MIDI does not preserve
MIDI stores symbolic events, not the original timbre, room or touch. Instrument-identity errors in a mix propagate into arrangement and notation.
Without a shared dataset and matching tolerance rules, source-specific metrics remain separate rather than producing a synthetic cross-paper winner.
Papers and public files for Unified specialist runtime
Model names, numbers and limitations trace to the papers, repositories or model cards below.