Building a Streaming Key Detector

The problem is not a histogram

A pitch class is a note’s position within an octave, ignoring which octave it is in. Middle C, the C above it, and the C three octaves below therefore belong to the same pitch class. There are twelve pitch classes in all.

The textbook approach to key detection is a correlation. Count how much of each pitch class a piece contains, compare that histogram against a published template for each of the 24 major and minor keys, then return the closest match. It is a good algorithm and about fifty lines of code. For answering “what key is this song in” after the song has finished, it works well enough that several standard implementations ship it.

Three constraints make it insufficient on its own for our use case.

The answer has to arrive while the music is still happening. Our detector sees the past and only the past. It cannot read ahead to the musical resolution that would have explained an ambiguous opening, and it cannot go back to revise a past answer after later chords arrive. That rules out the entire family of offline techniques that decode a best path over a complete sequence.

The input is not raw MIDI. It is the output of the chord recognizer: a stream of committed chord events containing the pitch classes that sounded, the chord identity selected by our recognizer, and how long the chord lasted. The upstream stages have already made judgment calls, and they are sometimes wrong.

Sometimes there is no answer, and saying so is correct. A repeating chord pattern may have a clear home note without behaving like any of the 24 major and minor keys the detector knows. Musicians call that kind of pattern a modal vamp. Two chords into a piece there may not be enough evidence for any answer. A detector that must always name something will name something wrong, and a key indicator that confidently flickers between wrong answers is worse than one that waits for clearer evidence before changing its mind.

What that combination needs is not a better histogram. It is a model that carries a belief forward through time, updates it as evidence arrives, and knows how sure it is.

Overview: predict, observe, decide

The detector is a hidden Markov model over 24 states, one per key. It never observes the key directly; it observes chords, and maintains competing explanations for which key is generating them. Each explanation gets a share of the total probability. One chord event goes in, one updated distribution comes out.

The next sections unpack those boxes.

The state space

Twenty-four states: twelve tonics (home notes) times two modes (major and minor). The belief is a plain array of 24 probabilities that sums to 1, initialized uniformly at 1/24 each.

final List<double> _posterior = List.filled(24, 1 / 24);

Alongside it is a 12-number pitch-class histogram holding the decaying evidence described below. The detector also remembers the previous recognized chord, an event count, and enough timing information to fade old evidence. All of that state is fixed in size. The app separately keeps the 100 most recent chords for its history view, but the detector never reads that list, so the cost of an update is the same on the first chord and the thousandth.

Predict: what the key is likely to do next

Before looking at the new chord, the belief is carried forward through a transition model, which encodes one assumption: keys persist. Most of the predicted probability stays where it is, and the remainder spreads to other keys with nearer ones favored.

Here, “nearer” means having a similar key signature. C major is closer to G major than to F-sharp major because G major adds just one sharp while F-sharp major has six. This distance is conventionally arranged on the circle of fifths.

Three parameters turn that idea into a 24-by-24 matrix:

A high self-transition is what buys stability. An established belief in C major survives a bar of chords that lean elsewhere because moving probability to a new key has a cost. Only sustained contrary evidence pays it. This is the probability-based version of what a debounce or hysteresis rule approximates, rather than a separate filter added after the fact.

Cadences get a discount

One addition modifies the prediction step. Some chord pairs are the musical equivalent of a period at the end of a sentence: a two-chord move that says “we have landed here.” Musicians call this a cadence. When the detector sees one, it makes that destination temporarily easier to reach. In the matrix, that means increasing the transition weight into the arrival key, then scaling the row back down so all of its probabilities still add up to 1.

// Ordinary row: mass spreads by distance.
// Cadence row: the target's weight is boosted, then renormalized.
final rowNorm = 1 + (boost - 1) * row[cadenceKey];
for (var to = 0; to < 24; to++) {
  final weight = to == cadenceKey ? row[to] * boost : row[to];
  predicted[to] += mass * weight / rowNorm;
}

The important detail is where this happens. The cadence changes the prediction made before scoring the new chord, not the chord’s evidence score. It gives the model a credible reason to move at that moment, while ordinary drifting still pays the full switch cost. If the cadence lands in the key the model already favors, it reinforces that key instead.

The shipped pattern is deliberately narrow. A tense dominant-seventh chord or close relative must point toward the next chord’s root (the note its name starts from), and the destination must sound like a stable major or minor chord rather than another dominant. A plain major chord is not enough. These restrictions keep ordinary movement, especially blues, from looking like a key change.

Score the evidence

The observation side asks a narrower question: how well does what we have recently heard match each of the 24 keys?

The detector first turns the recent music into a 12-number pitch-class histogram. Each position collects how much of one pitch class has sounded recently. Two things shape that histogram.

Duration weighting. Each event is weighted by how long it was held, giving sustained harmonies more influence than passing chords.

A decaying window. Evidence does not accumulate forever; it fades exponentially on a half-life. This dial turns out to be the single most consequential number in the system. After one half-life, a chord contributes half as much as it originally did; after two, it contributes a quarter as much. A short half-life therefore makes each observation a snapshot of the immediate chords, which tracks brief detours closely. A long one makes it a summary of the current section, which absorbs those detours and reports the settled key.

Neither is more accurate in the abstract. Each is right about a different question, and which one you want is a design decision rather than a correctness one. The app ships a four-second half-life by default and exposes the choice as a setting.

For scoring, each key supplies a profile: a 12-number reference template describing how strongly each pitch class supports that key relative to its proposed home note. This engine uses profiles derived from a large collection of written music and published by Albrecht and Shanahan (2013).

Scoring aligns the recent pitch-class histogram with each possible home note in both major and minor, then takes the Pearson correlation, a standard measure of how closely two numerical patterns share the same shape. That gives 24 raw scores.

Turning scores into a distribution

The 24 correlations are not probabilities. A softmax converts them into an emission, the probability distribution representing the evidence at this event across all 24 keys. An emission temperature controls how strongly the best-matching profile can pull away from the rest:

final top = scores.reduce(math.max);
for (var k = 0; k < 24; k++) {
  // Subtract the max before exponentiating: same result,
  // no overflow.
  emission[k] = math.exp((scores[k] - top) / emissionTemperature);
  total += emission[k];
}
for (var k = 0; k < 24; k++) emission[k] /= total;

At the shipped temperature of 0.25 the distribution is fairly sharp: a well-matched key pulls hard. Raising it flattens the evidence so no single event can move the belief as much. The value is fixed rather than exposed as a user setting.

Choosing major or minor

A prominent residual error in this detector is reporting the wrong mode: C minor instead of C major. A single rule addresses it. When a clearly major or minor chord arrives, its root also names a pair of possible keys. A C major chord, for example, is direct evidence in the choice between the C major and C minor keys. The rule moves probability toward the member of that pair whose mode matches the chord.

The containment matters more than the rule. The shift keeps the pair’s total probability unchanged: whatever major gains, minor loses.

final pairSum = emission[majorK] + emission[minorK];
final factor = math.exp(modeTilt * direction);
final major = emission[majorK] * factor;
final minor = emission[minorK] / factor;
final rescale = pairSum / (major + minor);
emission[majorK] = major * rescale;
emission[minorK] = minor * rescale;

Because the pair sum is conserved, the rule can decide between C major and C minor but is structurally incapable of shifting support toward G major or any other home note.

Observe

The update itself is deliberately simple: multiply each key’s predicted probability by the evidence for that key, then scale all 24 results so they add back up to 1.

for (var k = 0; k < 24; k++) {
  predicted[k] *= emission[k];
  total += predicted[k];
}
for (var k = 0; k < 24; k++) predicted[k] /= total;

This is the forward algorithm run in streaming order. After each event, it gives the probability of each key using everything heard so far and nothing from the future. Its offline counterpart, Viterbi decoding, finds the single most probable key sequence over a complete piece and would let the ending explain the beginning. That can produce a more coherent retrospective analysis, but it is unusable here because the future does not exist yet.

The result is a posterior: a model probability rather than an arbitrary score. It still needs calibration before it can be displayed as real-world confidence.

Claim or abstain

At every event the detector ranks all 24 keys. It speaks only if the leader is ahead of the runner-up by at least a margin floor, shipped at 0.3 on the model’s zero-to-one scale.

This is not a confidence display threshold. It decides whether there is an answer at all. On a modal vamp, several major-or-minor interpretations may explain the notes similarly well, so the gap opens less often and the detector is more likely to stay quiet. That is preferable to forcing the music into a key the model cannot represent faithfully.

Abstention is treated as a first-class outcome throughout: it is never scored as an error, and the evaluation reports coverage (how often the detector speaks) alongside accuracy. A detector can make its accuracy look good by refusing every difficult case, or make its coverage look good by guessing every time, so neither number is meaningful alone.

Making the confidence number honest

The raw model is systematically overconfident. In the original held-out check, it reported roughly 91% confidence in situations where it was right about 72% of the time. Put plainly, a user seeing “91%” would expect about 91 correct answers out of 100, while the detector was actually producing about 72.

The fix is temperature scaling, a one-number correction. Raise every probability to 1/T, then scale the results so they add back up to 1. When T is above 1, this lowers the leader and shares more probability with the alternatives.

The value of T was fitted using development music with human-authored key labels. The transformation preserves the candidates’ order, so the first-place key remains first. More importantly, it is applied only after the detector has decided whether to claim or abstain. The detector’s internal arithmetic still runs on the raw values, so calibration changes the percentage shown to the user without changing any answer.

The behavior presets

The app exposes three key detection behaviors as a user setting: Stable, Balanced, and Reactive. The half-life is the one value that changes the detector’s arithmetic. Each preset also has its own confidence correction and a stale interval, which controls how long an idle result remains current-looking before the interface dims it.

The confidence correction is the T from the previous section. It is fitted separately for each timescale because raw confidence behaves differently when the detector remembers a few recent chords instead of a whole section.

The stale interval changes presentation, not inference: it stops an old answer from looking current after the player goes quiet. After two minutes without a chord, all three behaviors reset the detector entirely.

Every other detector setting is shared across the three. Transition persistence, the internal evidence-scoring temperature, and the margin floor can each affect responsiveness, but less cleanly than the half-life already does. The presets therefore leave those settings alone.

What the presets trade is responsiveness against steadiness, not accuracy against inaccuracy. Reactive catches substantially more real key changes and catches them sooner. The same short memory also lets a brief harmonic detour look like a new key, so it changes its answer more often when the music never truly left, and it abstains more often while the evidence is thin. Those are the same property seen from two sides. Picking a preset means choosing which side matters more to you.

Closing the loop

The detected key does not stay in the key indicator. In Auto mode, once the visible detector makes the same claim on two consecutive chord events, that key becomes the app’s current key. It is then passed to the chord recognizer, whose ranking rules can prefer readings that naturally belong to the key, prefer the chord built on its home note, and choose between two chord names that account for the same sounding notes.

The app uses a second, hidden detector for chord naming. It always runs in Reactive mode, so it can follow shorter local key changes even when the visible detector is set to Stable or Balanced. In Auto mode, Ensemble uses that internal key to choose among plausible chord names, including names whose root note was implied rather than played.

The app may also use the next chord to reconsider the previous history entry. Only that entry can change, and the correction is for display only. It never feeds either detector or influences later chord names.

The connection remains deliberately weak. Chord recognition supplies events to key detection, and key detection supplies context for chord ranking, but the live analyzer still judges each new chord independently. In practice, the displayed key changes only about 0.4% of chord names, and feeding those changes back does not measurably improve key detection. Keeping the connection weak reduces the risk that the engines reinforce each other’s mistakes.

How the numbers were chosen

The constants above were not chosen arbitrarily. They began as reasonable starting points from music theory and established modeling practice, then candidate values were compared on development music with human-authored key labels. Popular songs and classical scores were evaluated separately. We tracked not just whether claimed keys were correct, but how often the detector was willing to answer, how quickly it found real key changes, and how often it invented them. Focused fixtures supplied specific stress cases such as blues progressions, jazz progressions, and modal vamps.

Only after the choices were fixed did we run a separate set of held-out pieces reserved for the final check. Keeping the development and evaluation music separate matters: tuning and grading on the same pieces can make an improvement look far more general than it is. The research archive records the alternatives that failed as well as the settings that shipped.

What it does not handle

• Polytonality and atonality. The model assumes one key at a time from a fixed vocabulary of 24, so it cannot represent music that is genuinely in two keys at once, or in none.
• Diatonic modes beyond major and minor. Dorian, Mixolydian, and the other diatonic modes have no state of their own, so a modal passage is scored against whichever of the 24 major or minor keys it most resembles.
• Anything outside twelve-tone equal temperament. Microtonal intervals and fine pitch differences between tuning systems have no representation in the standard system of twelve equally spaced notes per octave used by this pitch-class model.

The codebase

The detector is written in Dart and lives in packages/whatkey/. It consumes the chord-event model from packages/whatchord/, the engine described in the companion article. Both are standalone packages with no framework dependencies; only the app around them is Flutter. The evaluation harness, music-dataset importers, and tools that compare detector versions on the same pieces live in tool/whatkey/.

The project is open source under the Zero Clause BSD License, so you are free to use, modify, and share it however you like.

Watch it follow along.

WhatChord names chords and tracks the key as you play, on-device. Free for iOS and Android, with no subscription and no ads.

Want the evidence? Read the research notes