Turning Live MIDI Into Chord Events

MIDI does not contain chord boundaries

Press C, E, and G on a keyboard and the musical idea may be one C major chord. The MIDI connection reports three separate note-on messages. They may arrive a few milliseconds apart, or much farther apart if the chord is played as an arpeggio. Releases arrive separately too, while the sustain pedal can keep old notes sounding underneath the next harmony.

During one ordinary gesture, the sounding-note state might pass through C alone, then C-E, then C-E-G, then C-E-G-D, and finally C-E-G again. Once enough notes sound to form a chord, the chord recognizer analyzes each new snapshot. Some snapshots are meaningful chords. Others are just the route a pair of hands took between them.

Recognition and segmentation answer different questions:

• The recognizer asks, “What chord best explains the notes sounding right now?”
• The segmenter asks, “Did that reading last long enough to count as something the player meant?”

Without the second question, chord history would fill with brief in-between shapes. The key detector would treat every one as fresh harmonic evidence, and the chord name on screen would flicker through interpretations that no musician intended to hold.

From snapshots to events

The input path has one more stage than a static chord-recognition diagram usually shows:

A chord event is more than a name. It records when the chord began, how long it lasted, which pitch classes sounded, which one was in the bass, the exact MIDI voicing, the recognizer’s selected identity, and the nearby alternative readings available at that moment. It also preserves the tonality and playing context used for ranking, along with how far apart the candidates scored. Downstream code can therefore interpret the original decision and measure its ambiguity without rerunning recognition. The history stays in memory only. The streaming key detector consumes these events rather than the much noisier stream of raw note changes.

Single notes and intervals still appear in the interface, but they are not chord events. Capture begins only when the recognizer has a chord candidate, which currently requires at least three sounding notes.

Segment identities, not note changes

The segmenter follows the recognizer’s selected chord identity: the root, bass, quality, extensions, and which chord tones are actually present. Together, these make one reading distinct from another. The segmenter does not start a new event merely because the sounding MIDI notes changed.

Adding another C to a held C major chord changes the voicing but not its identity, so the event continues. The same is true when a doubled note is released. This keeps octave doubling above the bass from turning one held harmony into several artificial events. Changing the lowest note can instead change the chord’s inversion, which is a different identity and receives its own event.

An event snapshots its musical data when that identity first appears. Same-identity changes do not rewrite the snapshot later. That makes the record deterministic: its notes, candidate ranking, and analysis context all describe the same moment.

A challenger has to earn the change

When the selected identity changes, the segmenter does not immediately end the current chord. The new identity becomes a pending challenger.

If the original identity returns before the challenger survives the 200 ms stability window, the challenger is discarded. The original chord continues as one uninterrupted event. A brief added note can therefore produce a fleeting Cadd9 reading without splitting a held C major chord in two.

If the challenger does survive, it becomes the current chord and the previous one is committed if it lasted long enough. Crucially, the boundary is placed at the challenger’s original onset, not 200 ms later:

The decision arrives at 1,000 ms, but the stored history still says the musical change happened at 800 ms. Waiting for stability adds display and commit latency; it does not move the recorded timing or shorten the accepted chord.

A release, a drop below three notes, or another loss of a valid chord candidate closes capture immediately rather than waiting for another timer. Before handling any observation, including a release, the segmenter first promotes a challenger whose deadline has passed. A release after that deadline can therefore commit two events: the old chord through the challenger’s onset, then the newly accepted chord through the release. If the challenger is still inside its window, the old chord ends at the challenger’s onset and the unresolved challenger is discarded. Events that never lasted 200 ms are dropped.

One threshold, several jobs

For event capture, 200 ms serves both as the challenger window and the minimum duration of a committed chord. Those jobs could become separate settings if research ever gives them different answers, but so far one threshold keeps the model simple and the behavior easy to reason about.

The same 200 ms decision also governs the visible chord name. Its behavior and measured effect are covered below.

Why 200 milliseconds?

The original 200 ms value was an engineering judgment. It shipped with the first version of chord history, before there were any recorded performances to test it against: long enough to reject many finger rolls, short enough to feel like part of the gesture. The responsible next question was whether that plausible number held up once it could be measured.

It did, but not because 200 ms emerged as a clear optimum.

The segmenter was extracted into pure, clock-independent Dart so recorded MIDI performances could pass through the exact state machine used by the app. We then replayed 50 recorded piano performances with stability windows from 50 to 800 ms. This controlled comparison asks whether changing the window improves key detection, not how accurate the app is overall. Only events with an answer are included; either the signature’s major key or its relative minor counts as agreement. Reactive halved the weight of old evidence every second; Stable did so every 30 seconds.

Shortening the window from 200 to 50 ms doubled the number of events without helping the Stable result, while Reactive agreement fell from 55% to 52%. The extra events were mostly transition noise that the key detector did not need. Raising the window to 400 ms discarded nearly half the events for changes of only one or two points, neither of which was established as a real improvement. At 800 ms, only about a quarter as many events remained as at 200 ms. Stable agreement fell five points, while Reactive did no better than at 200 ms.

That is the useful conclusion: 200 ms sits comfortably on the flat part of the curve. It filters a large amount of transient input without pretending that waiting longer can clean up every ambiguity. The study also showed why the key detector still needs its own fading memory. A sustain-pedal blur can last much longer than any sensible segmentation window; it is sustained ambiguous evidence, not a quick mistake.

The display needed the same gate

History and key detection used the segmenter first. The main chord display originally did not. It showed the recognizer’s answer after every change to the sounding notes, even when that answer would never survive long enough to enter history.

That display now uses the same stability judgment. If a player builds a C major chord one note at a time, the display first shows the single C note, then a major third interval after E is added. When G completes the chord, that interval remains visible while C major proves itself. After 200 ms, the display changes to C major. The last useful label remains while any notes sound and clears only on silence, so it never blinks empty during that progression.

We measured flicker share, the portion of labeled display time occupied by names that lasted less than half a second, along with how often the name changed. We then simulated the display following the segmenter’s decisions over the same recorded performances.

In both sets, the segmenter-gated display sharply reduced flicker and name changes without missing any committed chords. That is why the display gate shipped.

What the segmenter deliberately gives up

Every stability policy chooses what not to represent. WhatChord will omit an intentional chord held for less than 200 ms. Fast ornaments and dense runs are more likely to disappear from chord history altogether. This trades recall for precision: the app would rather record fewer defensible chords than preserve every intermediate guess as though it were equally meaningful.

The segmenter also cannot separate a melody from an accompaniment, infer an unsounded harmony, or clean up a pedal wash that persists beyond the gate. Those are different problems. Recognition decides what a snapshot means, segmentation decides whether it lasted, and key detection decides how the accepted sequence fits together over time. Keeping those responsibilities separate is what makes each one measurable.

The codebase

The state machine lives in the pure-Dart ChordEventSegmenter. The app’s history provider owns live capture, timers, and retention, while the display provider runs its own copy for presentation.

Callers supply the clock rather than the segmenter reading wall time itself. That makes its behavior deterministic in unit tests and lets offline replay use the production state machine instead of an approximation. The same code that decides what counted while someone was playing is therefore the code the research measures later.

Watch each chord land.

WhatChord turns live MIDI into stable chord names and history, on-device. Free for iOS and Android, with no subscription and no ads.

Want the evidence? Read the performed-input research