Audio to polyphonic MIDI: state of transcription
Turning a recording into polyphonic MIDI has moved from lab demos to shipping products, but it is still not a finished solution for guitar. Open models (Spotify Basic Pitch 2022, Google MT3 2021, YourMT3+ 2024) and commercial tools (Melodyne DNA, Ableton Live 12, AnthemScore 6, Klangio) now produce usable first drafts. The newest open release is MuScriptor from Kyutai and Mirelo (arXiv 10 July 2026), whose authors report onset F1 60.4 on their own test set versus 32.5 for YourMT3+; its weights are non-commercial. Clean solo guitar scores in the 80s to low 90s on GuitarSet, while mixes, distortion, bends and tab fingering remain weak, and the 2025 AMT Challenge winner scored only 0.5998 multi-instrument onset F1.
Latest developments
AnthemScore 6.3.0
AnthemScore 6.3.0 improved sung and lead melody detection, fitted rhythms across the whole song, added swing detection and wrote rolled chords as arpeggios.
AnthemScore 6.2.0
AnthemScore 6.2.0 added a new neural network for beat, downbeat and time-signature detection, compound meters such as 6/8 and 12/8, and chord symbols written above the staff.
Noise2Fret diffusion tab model
Simionato and Bigo submitted Noise2Fret, a diffusion model for string and fret assignment with five playability losses (including hand-span feasibility). They report gains over baselines on GuitarSet...
AnthemScore 6.1.0
AnthemScore 6.1.0 improved note detection accuracy, made the smallest note size a per-measure setting, and let batch processing detect percussion and save in several formats.
Key takeaways
Reference
Background5 points
- Piano is the best-solved case: the MAESTRO dataset (about 200 hours of paired piano audio and MIDI, v3.0.0 = 198.7 hours, CC BY-NC-SA 4.0) drove the field [magenta.tensorflow.org].
- GuitarSet (Xi, Bittner et al., ISMIR 2018) recorded 360 excerpts of about 30 s with a hexaphonic pickup so each string is annotated separately; v1.0.1 dated 2018-09-20, CC BY-NC 4.0 [zenodo.org].
- Slakh2100 (Manilow et al., announced 2019-09-20) is 2,100 synthesized songs, 145 hours, rendered from Lakh MIDI with isolated stems; it has no vocals [auditory.org].
- Google's MT3 (Gardner et al., submitted 2021-11-04) showed one Transformer could transcribe many instruments and datasets [arxiv.org].
- Spotify's Basic Pitch (paper 2022-03-18, blog 2022-06-01) put a small instrument-agnostic model, under 17,000 parameters, in everyone's hands for free [arxiv.org, engineering.atspotify.com].
Key players6 points
- Spotify Audio Intelligence Lab (Basic Pitch, Apache-2.0, Python 3.7-3.11, TensorFlow/CoreML/TFLite/ONNX backends) [github.com].
- Google Research / Magenta (MT3, Colab notebook, 'not an officially supported Google product') [research.google].
- Kong et al. (piano, code on GitHub) and Sony's Toyama et al. (hFT-Transformer, ISMIR 2023) [arxiv.org].
- Queen Mary University of London (GAPS classical guitar dataset; YourMT3+ with Huawei) [arxiv.org].
- Kyutai and Mirelo (MuScriptor: Rouard, Krause, Roebel, Simon-Gabriel, Defossez) [github.com].
- Commercial: Celemony Melodyne 5 (DNA in Editor and Studio only), Ableton Live 12, Lunaverus AnthemScore, Klangio (Transcription Studio, Guitar2Tabs), Samplab, Jam Origin MIDI Guitar 2, Band-in-a-Box 2026 [celemony.com, ableton.com, lunaverus.com, soundonsound.com, pgmusic.com].
Where it stands now7 points
- MuScriptor (arXiv 2607.08168, 2026-07-10): decoder-only Transformer, small/medium/large (about 103M/307M/1.4B parameters), trained on 1.45 million synthetic MIDI files plus about 170,000 real recordings (11,000+ hours) and reinforcement learning; code MIT, weights CC BY-NC 4.0 [huggingface.co, github.com].
- NeuralNote v2, an open plugin (VST3/AU/standalone), says it replaced Basic Pitch with MuScriptor; no release date was shown on the page [github.com].
- AnthemScore reached 6.0.0 on 2026-05-07 and 6.3.0 on 2026-09-29, adding drum detection, chord symbols, swing and arpeggiated rolled chords [lunaverus.com].
- Klangio Transcription Studio (Sound On Sound, 2026-03-23) has a Rock mode for electric guitar; the vendor claims accuracy, and the article reports no test [soundonsound.com].
- Ableton Live 12 has Convert Harmony, Convert Melody and Convert Drums; the manual says other instruments in the clip are also detected and swelling notes may be missed [ableton.com].
- Logic Pro 11.2 stem splitting added guitar and piano stems; the release notes we opened describe no audio-to-MIDI feature [support.apple.com].
- Demucs 4.1.0 (2026-07-11) is maintained but the author says he is no longer actively working on it; htdemucs_6s guitar is 'okay' and piano has bleeding [github.com].
Numbers and figures9 points
- MT3 onset F1 (mixture model) per its Table 2: MAESTRO 0.9455, GuitarSet 0.895, Slakh2100 0.7585, MusicNet 0.5016, URMP 0.77; Melodyne scored 0.2824 on GuitarSet in that paper's 30-track subset test [arxiv.org].
- Kong et al.: MAESTRO onset F1 96.72% versus 94.80% for Onsets and Frames; pedal onset F1 91.86% [arxiv.org].
- YourMT3+ (MLSP 2024): GuitarSet onset F1 91.65 / 88.87 (without / with pitch shift), Slakh 84.14, MAESTRO v3 96.98; best prior baseline on MAESTRO 97.44 [arxiv.org].
- GAPS paper (2024): GuitarSet note onset F1 91.2 for the authors' model, 90.0 MT3, 79.0 Basic Pitch; zero-shot GuitarSet 88.1 versus MT3 32.0 [arxiv.org].
- GOAT (2025): note F1 on GuitarSet 0.860 for a DI-trained model; tablature fine-tune of Whisper got 0.682 accuracy and the authors say it did not work well [arxiv.org].
- TART (2025): acoustic DadaGP pitch accuracy 94.9% but tab (string/fret) accuracy 42.1% [arxiv.org].
- 2025 AMT Challenge (76 synthesized pieces): MIROS 0.5998, YourMT3-MoE 0.5938, MT3 baseline 0.3932 multi onset F1; MIROS precision fell from 0.9067 on solo to 0.4643 on three-instrument excerpts [arxiv.org].
- MuScriptor (authors' D_Test, 372 tracks): onset F1 60.4 versus 32.5 for YourMT3+; the paper text says multi F1 48.2 and the model card says 47.8 [huggingface.co].
- Demucs HT fine-tuned: 9.00 dB SDR on MUSDB HQ [github.com].
Effects and evidence6 points
- Documented: dataset-trained models reach about 0.9 onset F1 on GuitarSet, which is clean solo acoustic playing [arxiv.org].
- Documented: accuracy collapses when the model sees unfamiliar audio (MT3 zero-shot GuitarSet 32.0) [arxiv.org].
- Documented: tone variation (amps, cabinets, pedals) hurts, which is why EGDB-PG and GOAT added amplifier-rendered data [arxiv.org].
- Documented: Songsterr AI users in 2025 called it a starting point, reporting rhythm, tempo-sync and key-signature errors [forum.bassbuzz.com].
- Documented: Jam Origin MIDI Guitar 2 was praised by Sound On Sound in 2017 for polyphonic tracking; KVR users in November 2022 disagreed about latency [soundonsound.com, kvraudio.com].
- Only claimed: Klangio Rock mode accuracy, Transkr V4 'no special pickup needed' and MuScriptor being 'the most accurate open-source transcription model' (vendor/author statements) [soundonsound.com, midi.org, github.com].
Competing views4 points
- The AMT Challenge authors say progress is real but polyphony and timbre variation remain hard [arxiv.org].
- The GAPS authors argue more data, not new architecture, drove their gains, and that encoder-decoder models like MT3 generalize poorly [arxiv.org].
- The MuScriptor authors argue synthetic pre-training plus real data and RL fixes the real-audio domain shift; the press summary cautions the figures may shrink under outside re-runs [aiweekly.co].
- Hardware pickups (Roland GK, Fishman TriplePlay) versus software: Jam Origin says no special pickup is needed, while forum users call software best for one to four notes at a time [jamorigin.com, kvraudio.com].
Key terms5 points
- Onset F1: share of notes whose start time is within a tolerance (usually 50 ms) and pitch match, balancing precision and recall.
- Multi F1: also requires the right instrument; much stricter than instrument-agnostic onset F1.
- Hexaphonic pickup: one sensor per string, giving clean per-string ground truth (GuitarSet) or MIDI output (Roland GK-3).
- Stem separation: splitting a mix into vocals, drums, bass and other before transcribing (Demucs).
- Zero-shot: tested on a dataset never seen in training.
Open questions4 points
- Will independent groups reproduce MuScriptor's gain on real guitar-heavy rock?
- Will a commercially licensed model reach the same accuracy?
- Can tab (string and fret) accuracy rise from the 42.1% reported by TART toward the 90s seen in pitch accuracy?
- Does stem separation before transcription help or hurt? Demucs's own notes say its guitar stem is only 'okay'.
Not yet established5 points
- Any controlled published accuracy for distorted, high-gain electric guitar mixes with bends on a commercial tool.
- Official current pricing for Melodyne 5, Jam Origin MIDI Guitar 2 and Band-in-a-Box 2026 AI-Notes; the Celemony and Jam Origin pages we opened listed none.
- Release dates for NeuralNote v2, Band-in-a-Box 2026, Transkr V4, MIDI Guitar 3.
- Whether Cubase or Studio One offer native polyphonic audio-to-MIDI; our searches found nothing confirming it.
- Whether Roland GR-55 or Fishman TriplePlay still match software on tracking; only anecdotal retailer reviews were found.
Full timeline
- September 29, 2026
AnthemScore 6.3.0
AnthemScore 6.3.0 improved sung and lead melody detection, fitted rhythms across the whole song, added swing detection and wrote rolled chords as arpeggios.
- September 4, 2026
AnthemScore 6.2.0
AnthemScore 6.2.0 added a new neural network for beat, downbeat and time-signature detection, compound meters such as 6/8 and 12/8, and chord symbols written above the staff.
- August 31, 2026
Noise2Fret diffusion tab model
Simionato and Bigo submitted Noise2Fret, a diffusion model for string and fret assignment with five playability losses (including hand-span feasibility). They report gains over baselines on GuitarSet and GOAT, with no figures on the abstract page.
- August 18, 2026
AnthemScore 6.1.0
AnthemScore 6.1.0 improved note detection accuracy, made the smallest note size a per-measure setting, and let batch processing detect percussion and save in several formats.
- July 15, 2026
MuScriptor model card limits
The large model card lists limits: no velocity output, cannot represent two simultaneous notes of the same pitch and instrument, drums onset-only, bias toward pop and Western classical, and 'not a substitute for a hand-annotated score'. It lists multi F1 47.8.
- July 15, 2026
MuScriptor open release coverage
AI Weekly reported Kyutai and Mirelo's release: three sizes (103M, 307M, 1.4B), code under MIT and weights CC BY-NC 4.0 (not commercially usable). Large model multi F1 48.2 versus 21.9 for YourMT3+; figures are the authors' own.
- July 11, 2026
Demucs 4.1.0
Demucs v4.1.0 modernized packaging and moved models to Hugging Face. The README says the maintainer is no longer actively working on it and that weight licensing is unclear.
- July 10, 2026
MuScriptor paper posted
Kyutai and Mirelo posted MuScriptor (arXiv 2607.08168). By the authors' tests, fine-tuning on real data adds about 20 points and the 1.3B model hits onset F1 60.4 on a 372-track test set. Weights and inference code are promised.
- May 7, 2026
AnthemScore 6.0.0
Lunaverus released AnthemScore 6.0.0 with improved note detection, automatic drum and percussion detection, and a 1-10 accuracy/speed slider. Editions were priced in CAD at 42.47 (Lite), 57.12 (Professional) and 144.99 (Studio).
- March 29, 2026
2025 AMT Challenge results
Chaturvedi et al. report 21 registered teams, 8 valid. MIROS won with 0.5998 multi onset F1 versus 0.3932 for MT3; precision fell from 0.9067 (solo) to 0.4643 (three instruments). Models hallucinated absent instruments.
Earlier updates19 more, back to 2017
- March 23, 2026
Klangio Transcription Studio reviewed
Sound On Sound described Transcription Studio (VST3, AU, standalone; cloud processing, under 30 seconds) with Universal, Classic and Rock modes, exporting PDF, MusicXML, Guitar Pro and MIDI, from 4 pounds a month. Rock mode accuracy is the vendor's claim only.
- October 2, 2025
TART technique-aware tab transcription
Gupta et al. proposed a four-part pipeline for tablature with fingerings and expressive labels (slides, bends). The abstract notes guitar lags piano and systems often miss techniques and assign wrong string and fret. The authors call it the first of its kind.
- July 22, 2025
GOAT guitar audio and tab dataset
The GOAT paper (ISMIR 2025) offers about 5.9 hours of direct-input electric guitar with tablature, plus amplifier rendering to 29.5 hours. Note F1 reached 0.860 on GuitarSet; a Whisper tab model got 0.682 accuracy and the authors say it did not work well. Access is by request.
- April 14, 2025
Songsterr AI tabs: user reports
Forum users in February to April 2025 called Songsterr's AI tabs a useful starting point but reported rhythm errors, a wrong key signature and poor tempo sync with the recording. Quality depended on source audio.
- April 10, 2025
Electric guitar tone-variation study
Chen et al. introduced EGDB-PG and the Tone-informed Transformer to cope with amplifier, cabinet and pedal variation in electric guitar audio, reporting gains over baselines across amplifier types. Specific scores were not on the abstract page.
- August 30, 2024
GAPS classical guitar dataset
Riley, Guo, Edwards and Dixon released GAPS: 300 performances, over 14 hours, 205 performers. Their model scored 91.2 note-onset F1 on GuitarSet and 88.1 zero-shot, where MT3 scored 32.0 zero-shot. Dataset is mostly nylon-string classical guitar.
- July 30, 2024
YourMT3+ revised
Chang, Benetos, Kirchhoff and Dixon released YourMT3+ (v2 dated 30 July 2024), using a hierarchical-attention encoder with mixture of experts. Reported onset F1: Slakh 84.14, GuitarSet 91.65, MAESTRO v3 96.98, versus 75.20 for MT3 on Slakh in their runs.
- July 10, 2023
Sony hFT-Transformer for piano
Toyama, Akama, Ikemiya, Takida, Liao and Mitsufuji submitted a two-level hierarchical frequency-time Transformer for piano. They report state of the art F1 on MAPS and MAESTRO v3.0.0; a later paper lists its MAESTRO onset F1 as 97.44.
- January 19, 2023
Samplab 2 announced
Samplab 2 launched as a Windows and macOS plugin and standalone app with polyphonic audio-to-MIDI, stem separation and note editing. Basic is free; Premium was announced at 9.99 USD or EUR per month.
- December 7, 2022
Demucs v4 and six-source model released
Per the Demucs README, v4 went to PyPI on 7 December 2022 and the htdemucs_6s model adding guitar and piano stems was released. The README rates its guitar stem 'okay' and says piano has a lot of bleeding and artifacts.
- November 17, 2022
KVR users split on MIDI Guitar 2
In a November 2022 KVR thread some users reported lag at the lowest settings and said it suits one to four notes at a time, while others reported no noticeable latency and rock-solid live use. Price was said to have risen from about 70 to 150 dollars.
- June 1, 2022
Spotify launches Basic Pitch
Spotify Engineering (Rachel Bittner) introduced Basic Pitch, a free open-source audio-to-MIDI tool built with Soundtrap. Spotify claims it is polyphonic, detects pitch bends, has under 17,000 parameters and runs faster than real time. The post itself says overlapping sounds are hard to separate.
- March 18, 2022
Basic Pitch paper on arXiv
Bittner, Bosch, Rubinstein, Meseguer-Brocal and Ewert submitted the small instrument-agnostic polyphonic note model later called Basic Pitch. The abstract says note estimation clearly beats a comparable baseline while frame accuracy is slightly below specialized systems.
- November 4, 2021
Google MT3 preprint
Gardner, Simon, Manilow, Hawthorne and Engel showed a general Transformer can transcribe multiple instruments across datasets. Mixture-model onset F1 was 0.895 on GuitarSet, 0.9455 on MAESTRO and 0.7585 on Slakh2100. Day taken from the v1 submission date shown on arXiv.
- June 1, 2021
Omnizart toolbox preprint
Academia Sinica authors led by Yu-Te Wu and Li Su submitted Omnizart, a Python library claimed as the first toolkit with models for solo instruments, ensembles, percussion and vocals plus chords and beats. The JOSS version was later listed by a search result as December 2021.
- October 5, 2020
Kong et al. high-resolution piano transcription
Kong, Li, Song, Wan and Wang submitted a system that regresses precise onset and offset times. It reports 96.72% note onset F1 on MAESTRO versus 94.80% for Onsets and Frames, and a pedal onset F1 of 91.86%. Revised to v3 on 31 July 2021.
- September 20, 2019
Slakh2100 dataset announced
Ethan Manilow announced Slakh2100 on the AUDITORY list: 2,100 songs, over 145 hours, synthesized with sample-based instruments from the Lakh MIDI Dataset, with isolated sources and aligned MIDI but no vocals.
- September 20, 2018
GuitarSet v1.0.1 published
GuitarSet, by Xi, Bittner, Pauwels, Ye and Bello, holds 360 guitar excerpts recorded with a hexaphonic pickup plus a reference microphone, annotated with per-string notes, chords and beats. Version 1.0.1 was published 20 September 2018 under CC BY-NC 4.0 and totals 7.5 GB.
- September 1, 2017
Sound On Sound reviews MIDI Guitar 2
Paul White reviewed Jam Origin MIDI Guitar 2 version 2.2.1 for Mac and Windows (standalone and VST/AU) at $99.95. He called polyphonic tracking very accurate even for fast playing and noted the standalone version adds 3 to 6 ms of latency compared with the plug-in.
huggingface.co · soundonsound.com · zenodo.org · auditory.org · arxiv.org · engineering.atspotify.com · kvraudio.com · github.com · rekkerd.org · forum.bassbuzz.com · lunaverus.com · aiweekly.co
Related stories
“Light strings are a bit of a fallacy”: String gauge does actually matter to tone, according to The Edge
The U2 guitarist once made his stance on string gauge crystal clear. B.B. King, Billy Gibbons and John Mayer might not agree with everything he says...
Steve Lukather on how he ended up getting a daily guitar lesson from one of his biggest heroes
When Lukather was a kid, he wanted to be Larry Carlton – and he ended up learning from the man himself years later
Session legend Carl Carlton on imposter syndrome, meeting his heroes, and the wild tale that gave him his name
Renowned as one of Europe’s great slide men, Carl Carlton might also be the rock ’n’ roll era’s best raconteur. He shares some war stories from the glory days