MuScriptor-large: evidence and limits of a 1.366B full-mix-to-MIDI model
An evidence-led account of the flagship checkpoint, event modelling, private and cross-domain evaluation, Type-1 MIDI output, and gated non-commercial license.
AUTOMATIC MUSIC TRANSCRIPTION
MuScriptor-large is the largest public MuScriptor checkpoint and the model card's flagship, best-quality option. It converts 16 kHz mono audio into five-second segments, represents them as 512-bin mel features, and uses a 48-layer decoder-only Transformer to generate pitch, onset, offset, instrument, and drum events. The public checkpoint contains exactly 1,366,399,488 F32 parameters, with a 5,465,642,136-byte safetensors file.
Largest within a family is not the same claim as universal state of the art. The paper reports a large gain over YourMT3+ on its private D_Test and also publishes results on eight external datasets, where MuScriptor does not win every metric. D_Real, D_RL, and D_Test are not public, and MIROS has not been compared on the same audio, metric, and post-processing protocol. The defensible conclusion is narrower: MuScriptor-large is one of the strongest new open-weight candidates for general multi-instrument AMT, supported by author-reported leading results.
1. The flagship tier is an exact checkpoint, not a name inferred from marketing
MuScriptor publishes small, medium, and large checkpoints. Their repository metadata gives exact parameter counts of 102,968,832, 307,030,528, and 1,366,399,488. Large is about 4.45 times the size of medium and is explicitly described as the flagship, best-quality release. The paper's scale ablation uses the labels 60M, 100M, 300M, and 1.3B; those experiment labels must not be substituted for the exact parameter counts of the three public checkpoints.
At revision 8809fdfbed2affa7ade94a7059e746e3880720e7, the weight repository contains a 5,465,642,136-byte model.safetensors, plus a small set of configuration files. The source is pinned separately to commit 302343e8992bdfc619f77f1988168374ed5d675d. Together, these identifiers bind the architecture, tokenizer, inference defaults, and weights. A label that says “large” is not sufficient if an implementation can silently load medium or follow a moving main branch.
F32 storage does not mean every operation must use F32, and it does not establish that an unofficial low-bit derivative is quality-equivalent. The release provides no same-protocol quality study for an alternative quantized checkpoint. A quality-first path can therefore require the exact large identity and fail clearly when it cannot load, instead of switching to a smaller tier.
2. From an acoustic window to a conditional event sequence
Input is converted to 16 kHz mono. The short-time Fourier transform uses a 2,048-point window and a 160-sample hop, producing 100 frames per second, then projects them into 512 mel bins. A five-second segment therefore contains roughly 500 acoustic frames. These features enter a decoder-only Transformer with width 1,536, 24 attention heads, and 48 layers. The model does not first split a mix into a fixed set of stems and run an independent transcriber for each one; time, instrument, and note events are generated in a shared sequence.
p(y|x,c)=∏n=1N p(yn|y<n, Enc(x), c)xmel representation of a five-second audio segmentcoptional instrument condition; an empty set means automatic detectionyna time, instrument, pitch, or termination event token
The autoregressive factorisation lets previously generated instruments and chords inform later decisions, which is useful for cross-instrument context in a full mix. It also propagates errors. A wrong time-shift can displace later notes, a missing note-off can leave a hanging note, and an instrument-token drift can put correct pitches on the wrong track.
The current official example uses batch size 1, beam size 1, cfg_coef=1, and prelude forcing. Beam size 1 is a deterministic argmax path. A wider beam retains more prefixes and consumes more memory; without a controlled ablation, beam 4 cannot simply be relabelled “maximum quality.”
Classifier-free guidance sharpens a condition by combining conditional and unconditional logits: ℓ=ℓu+s(ℓc−ℓu). The final paper table reports Multi F1 of 48.2 at CFG=1 and 47.8 at CFG=2. The public default therefore remains 1 rather than assuming that a larger guidance number is better.
3. 1.45 million MIDI files, 11,000 hours of real audio, and 300 reviewed transcriptions
The paper separates training material into three layers. D_Synth contains about 1.45 million MIDI files from Lakh MIDI and a commercial data provider, rendered through more than 250 SoundFonts. Transposition, tempo, velocity, instrument randomisation, and detuning expose the same symbolic material to a wider timbral range. Synthetic rendering gives exact event boundaries but cannot fully reproduce microphone bleed, mastering, performer noise, or the long tail of real production.
D_Real contains roughly 170,000 internal recordings, about 11,000 hours in total. Their note labels come from audio-symbolic alignment rather than sample-level manual annotation. Real mixes broaden the production distribution while adding title matching, version differences, plug-in latency, and alignment error. D_RL then selects 300 high-quality transcriptions reviewed by people for GRPO/REINFORCE-style post-training. The three stages address coverage, domain shift, and sequence preference, but the real and reward datasets are not public.
Supervised learning minimises token-level negative log likelihood, while post-training uses a reward assembled from onset, offset, frame, drums, and multi-instrument measures. Sequence-level optimisation can penalise structural failures that independent token classification misses. Because the reviewed reward set has only 300 tracks, coverage of rare instruments, extreme tempi, and non-Western traditions still needs public external evaluation.
LSFT=−Σn log p(yn|y<n,x,c), J(θ)=E[R(y)]LSFTsequence cross-entropy during supervised trainingR(y)combined reward for a completed transcriptionD_RL300 reviewed transcriptions, not a public benchmark
4. A strict whitelist between 35 interface names and 36 learned groups
MuScriptor maps 128 General MIDI programs into MT3_FULL_PLUS instrument groups. The model learns 36 groups, while the public demo exposes 35 selectable names: acoustic and electric pianos, three guitar groups, acoustic and electric bass, strings, brass, woodwinds, synthesizers, voice, and drums among them. The difference reflects an internal-representation versus selectable-interface boundary; 35 interface choices should not be reported as the full number of learned categories.
With an empty instrument field, the model may detect any supported group. As soon as one or more names are selected, the decoder hard-masks all other instrument tokens. This is a strict whitelist, not a soft preference. It can reduce instrument leakage when an ensemble is known, but it can also erase a real part that the user failed to include. Piano, bass, and drums may be a good constraint for a known trio; automatic detection is safer when the arrangement is uncertain.
The constraint must travel from the interface to the tokenizer's token mask. Selected chips that are discarded by an API create a false control, while unrestricted strings postpone spelling errors until execution. A faithful contract accepts only the official 35 identifiers, rejects duplicates, and defines an empty array as automatic detection.
5. Reading the large gain on the private D_Test
Paper Table 1 compares YourMT3+ YPTF.MoE+Multi (noPS) with MuScriptor training stages on the same private D_Test. The final large model at CFG=1 reports Onset 60.4, Frame 73.3, Offset 49.0, Drums 50.2, and Multi 48.2. YourMT3+ reports 32.52, 45.54, 17.79, 41.4, and 21.9. The gap is substantial and supports the claim that the authors' larger data, model, and post-training recipe materially improved performance under their real-mix distribution and annotation protocol.
| Same private D_Test | Onset F1 | Frame F1 | Offset F1 | Drums F1 | Multi F1 |
|---|---|---|---|---|---|
| YourMT3+ | 32.52 | 45.54 | 17.79 | 41.4 | 21.9 |
| MuScriptor-large, CFG=1 | 60.4 | 73.3 | 49.0 | 50.2 | 48.2 |
Multi F1 requires instrument, pitch, onset, and offset to satisfy the matching rule together, so it is not interchangeable with a score that checks fewer fields. MIROS scored 0.5998 in the 2025 AMT Challenge on a different set of 76 roughly twenty-second synthetic excerpts and a program + pitch + onset Multi Onset F1. MuScriptor did not enter that same-protocol test. Ranking 48.2 beside 0.5998 would compare different denominators and different event definitions.
D_Test contains 372 multi-instrument works drawn from internal material, with title filtering intended to reduce training-test leakage. Title filtering is useful but not equivalent to a public, reproducible deduplication audit. The audio, annotations, version choices, and overlap-cleaning rules are unavailable. The headline must therefore remain labelled as an author-reported private-set result.
6. Cross-domain tables and the overlapping-note ablation expose failure regions
The eight external datasets show a more uneven profile. Against YourMT3+, MuScriptor wins Frame F1 on 7/8 datasets and Multi F1 on 6/8, but wins Onset F1 on only 4/8. It trails on onset, frame, offset, and multi for RWC-C; on offset, drums, and multi for RWC-R; and on onset and offset for Bach10. Strong sustained-pitch activity and overall track content do not imply dominance on every transient, instrument family, or recording distribution.
The tokenizer has a concrete representational limit: simultaneous overlapping notes with the same instrument and pitch cannot both be represented cleanly. The paper removes the shorter overlap for the main D_Test evaluation. Without that removal, Onset F1 falls from 60.4 to 51.8, Offset from 49.0 to 41.9, and Multi from 48.2 to 42.0. Repeated same-pitch piano notes under pedal, string unisons, and layered guitars can all encounter this collision.
Model-scale ablation belongs in a separate chart from the final comparison. Paper Table 4 trains on D_Real only with instrument conditioning. Multi F1 rises from 35.2 to 38.2, 39.7, and 40.5 across the 60M, 100M, 300M, and 1.3B labels. It demonstrates continuing scale benefit under that setup, but does not include the final synthetic pretraining and reinforcement stages and is not a direct benchmark of the public small, medium, and large files.
| Paper scale label | Onset | Frame | Offset | Drums | Multi |
|---|---|---|---|---|---|
| 60M | 47.7 | 65.7 | 35.3 | 39.8 | 35.2 |
| 100M | 51.2 | 67.2 | 38.7 | 41.5 | 38.2 |
| 300M | 52.4 | 68.0 | 40.3 | 42.0 | 39.7 |
| 1.3B | 53.2 | 68.7 | 41.0 | 42.5 | 40.5 |
Listening review should concentrate on dense unisons, shared drum-and-bass transients, distorted guitars, choirs, rare timbres, and heavy reverberation. Aggregate scores can remain good while a track develops short fragments, cross-track leakage, cut sustains, or multiple channels for one instrument. Averages describe population behaviour; they do not replace piano-roll inspection and resynthesised playback.
7. What Type-1 multi-track MIDI retains—and what the writer fixes
The current writer produces Type-1 MIDI: metadata occupies its own track, and each instrument track carries a name, program, channel, note-on, and note-off events. Drums follow the General MIDI channel-10 convention. Compared with flattening everything into Type-0, this structure is easier to mute, solo, and edit in a DAW, and makes instrument drift more visible.
The output does not recover performed dynamics. Public code writes note velocity as 100. A velocity field in the MIDI only satisfies the file format; it is not evidence that the model inferred expression from the recording. Drums provide onsets only. The model also does not infer the song's tempo map, time signature, or beat grid. The writer uses a fixed 120 BPM tempo event to map event seconds into ticks. “Fixed at 120 BPM” must never be presented as “detected 120 BPM.”
Continuous timing can be approximately retained while the notated values shown by score software depend on that fixed tempo and later quantisation. Rubato, swing, free-time passages, and tempo changes require a separate beat and structure analysis. Playable events, editable tracks, and publication-ready notation are three different layers; voicing, ties, pedalling, enharmonic spelling, repeats, and expression still require musical decisions.
When long audio is processed as overlapping segments, onset ownership must be assigned in global seconds while notes that remain open across a boundary are carried forward. Concatenating segment MIDI files can duplicate boundary attacks, cut sustains, or discard track names, programs, and the drum channel. Acceptance should check for hanging notes, preserved track metadata, and the absence of systematic timing drift across segment joins.
8. Licence information
MuScriptor code uses the MIT licence. The large weights use CC BY-NC 4.0 and are distributed through a gated Hugging Face page. Access requires a signed-in account, acceptance of the page terms, and confirmation that the user holds the necessary rights to the music being transcribed.
The code and weights have separate licences. Access to the weight files does not change their NonCommercial condition; the pinned model card and licence text are the primary references for permitted use.
9. Conclusion and primary sources
The value of MuScriptor-large is specific: it is the largest public family member at 1.366B parameters, targets real full mixes, supports a hard instrument whitelist, produces editable Type-1 multi-track MIDI, and has a large author-reported gain over YourMT3+ on private D_Test. Its limits are equally specific: the headline data are not public, external metrics are not a clean sweep, overlapping same-pitch notes collide in the tokenizer, velocity and a real tempo map are absent, and the weights are licensed for non-commercial use.
“Highest-quality MuScriptor checkpoint” is supported. “Independently confirmed universal audio-to-MIDI SOTA” is not. That assessment can change when a public reproduction compares MuScriptor-large, YourMT3+, and MIROS on the same recordings, event definitions, matching tolerances, and overlap policy.
- MuScriptor paper: method, training sources, Tables 1–4, and overlapping-note ablation
- Pinned source commit: inference, tokenizer, Type-1 MIDI writer, and demo behaviour
- Pinned MuScriptor-large revision: files, model card, and licence
- 2025 AMT Challenge: MIROS on a different dataset and Multi Onset F1 protocol