Building a Real-Time Chord Recognizer

The problem is not a lookup

The first intuition when building a chord recognizer is to build a dictionary. There are only 12 pitch classes, which means there are only 2^12 = 4096 possible pitch-class sets. Store a name for each set, and when a user plays C-E-G, look up C-E-G and return “C major.”

The problem is not memory. Four thousand entries is trivial. The problem is meaning. A pitch-class set does not contain enough information to decide what musicians will call it.

Piano players often leave out notes that a dictionary entry might expect. Extended chords add notes that no fixed dictionary entry anticipates. And the same set of pitch classes, as discussed in the companion article, can legitimately be described as multiple different chords depending on musical context.

What you actually need is a cost model. It has to evaluate how well any given set of notes fits each chord type, rank all plausible interpretations, and apply musical judgment when costs are close.

Overview: a four-stage pipeline

Before diving into each component, here is the overall shape of the algorithm. A snapshot of sounding notes enters at the top; a ranked list of chord interpretations comes out at the bottom.

The rest of this article walks through each stage in detail, ending with a discussion of known limitations.

Pitch classes and bitmasks

WhatChord models the common 12-tone equal temperament (12-TET) pitch-class framework used by MIDI keyboards, which divides each octave into equal semitone positions. A pitch class is the note’s position within that octave, ignoring which octave it’s in, so middle C, the C above it, and the C three octaves below all share pitch class 0. In this engine, pitch classes are numbered 0 (C) through 11 (B).

For analysis, the engine collapses the sounding notes into a set of pitch classes plus the lowest sounding note as bass. The pitch-class set is represented as a 12-bit integer mask where bit n is set if pitch class n is present. C major (C=0, E=4, G=7) looks like this:

11	10	9	8	7	6	5	4	3	2	1	0
B	A♯	A	G♯	G	F♯	F	E	D♯	D	C♯	C
0	0	0	0	1	0	0	1	0	0	0	1

// Pitch classes: C=0, E=4, G=7
int pcMask = (1 << 0) | (1 << 4) | (1 << 7);
// pcMask == 0b000010010001 == 0x091

This representation is compact and fast. Checking whether a pitch class is present is a single bitwise AND. Counting present pitch classes is a popcount. Rotating the set relative to a candidate root is a loop over bits with modular arithmetic. All of these operations are cheap.

A key design decision: only pitch classes actually present in the voicing are tested as candidate roots. There are no “ghost roots” and the algorithm never proposes an interpretation where the chord is rooted on a note that is not being played. This keeps the candidate count small (bounded by the number of sounding notes, typically 3–7) and avoids obviously wrong readings.

This is a deliberate “solo keyboard” assumption. The current engine is optimized for the common case where the same MIDI stream contains both the harmony and the bass note. A future ensemble mode could relax that rule for settings where another instrument is carrying the bass, allowing rootless voicings to imply roots that are not literally present in the keyboard part.

Chord templates

Chord qualities are also defined as bitmask templates. Each one describes three sets of intervals relative to the root:

• Required: tones that must be present to identify this quality. Missing more than one required tone causes the template to be skipped entirely.
• Optional: tones frequently omitted in real voicings (almost always the perfect 5th). Present when played, unremarkable when absent.
• Penalty: tones that actively contradict this quality. Having a major 3rd present when you are trying to identify a minor chord raises the cost.

The 27 templates, organized by complexity:

Quality	Required intervals	Optional	Key penalties / constraints
Major	R, M3	P5	m3, m7, M7
Major (♭5)	R, M3, ♭5	—	P5, m3, m7, M7
Minor	R, m3	P5	M3, m7, M7
Minor ♯5	R, m3, ♯5	—	M3, P5, m7, M7
Diminished	R, m3, ♭5	—	M3, P5
Augmented	R, M3, ♯5	—	m3, P5
Power (5)	R, P5	—	m3, M3, ♭5, m6/M6, m7, M7
Sus2	R, M2, P5	—	m3, M3, m7, M7
Sus4	R, P4, P5	—	m3, M3, m7, M7
Double sus (Sus2sus4)	R, M2, P4, P5	—	Exact match only
Major 6	R, M3, M6	P5	m3, m7, M7
Minor 6	R, m3, M6	P5	M3, m7, M7
Dominant 7	R, M3, m7	P5	M7, m3
7sus2	R, M2, m7	P5	m3, M3, P4, M7
7sus4	R, P4, m7	P5	m3, M3, M7
7♭5	R, M3, ♭5, m7	—	P5, M7, m3
7♯5	R, M3, ♯5, m7	—	P5, M7, m3
Major 7	R, M3, M7	P5	m7, m3
Major 7sus2	R, M2, M7	P5	m3, M3, P4, m7
Major 7sus4	R, P4, M7	P5	m3, M3, m7
Major 7♭5	R, M3, ♭5, M7	—	P5, m7, m3
Major 7♯5	R, M3, ♯5, M7	—	P5, m7, m3
Minor 7	R, m3, m7	P5	M7, M3
Minor 7♯5	R, m3, ♯5, m7	—	P5, M7, M3
Minor-Major 7	R, m3, M7	P5	M3, m7
Half-Diminished 7	R, m3, ♭5, m7	—	P5, M3, M7
Fully Diminished 7	R, m3, ♭5, d7	—	m7, P5, M3, M7

Notice that the perfect 5th is optional for most chord families. Requiring it would cause the algorithm to miss many idiomatic voicings in common use. The power chord is the exception: a bare fifth is the whole chord, so its fifth is required, and the reading is discarded outright if any tone it cannot name as color is left over.

Penalty tones are not hard rejections. The template is still evaluated; it just pays an added price. This handles cases where a note might simultaneously belong to one chord and partially fit another, and lets the cost reflect the degree of fit rather than producing a binary yes/no.

Template pricing

For each candidate root (each pitch class present in the voicing), the analyzer rotates the pitch class mask relative to that root to get an interval mask. Then it assigns an explanation cost for that interval mask against all 27 templates.

// Rotate: compute intervals above rootPc for each sounding note
int rotateMaskToRoot(int pcMask, int rootPc) {
  var rel = 0;
  for (var pc = 0; pc < 12; pc++) {
    if ((pcMask & (1 << pc)) == 0) continue;
    final interval = (pc - rootPc) % 12;
    rel |= (1 << (interval < 0 ? interval + 12 : interval));
  }
  return rel;
}

Each surviving reading is then priced by how well it explains the input. That price is its explanation cost: core chord tones are free, and the name pays for everything else it asks a reader to accept. The lowest cost is the best fit.

Cost component	Price	Notes
Core chord tone present	0	The name itself carries these
Vocabulary rarity	0.1 / 0.4 / 1.0	How readily a musician reaches for the quality name: everyday names (major, minor, 7, m7, maj7, sus4, 6ths, and the bare power chord) are free; marked names (dim, aug, dim7, m7♭5, m(maj7), sus2, double-sus, 7sus4) cost 0.1; uncommon ones (7♭5, 7♯5, maj7sus4, maj7♯5) cost 0.4; names that almost always respell a more common chord (m♯5, m7♯5, maj♭5, maj7♭5, 7sus2, maj7sus2) cost 1.0
Natural extension (9, 11, 13)	0.30–0.35	Integrated stack members on seventh chords. A candidate that would promote 9, 11, or 13 while its required seventh is missing is rejected; those same tones are add tones or sixths. An 11 or 13 with no 9 under it pays a small surcharge (it is really an add-tone wearing a stack name), waived for an 11 in the bass (the sus-pedal idiom, Am7/D). On marked-vocabulary hosts (a 13 on a half-diminished or minor-major seventh) the price is multiplied by 1.75.
Add tone (add9, add11, add13)	0.30–0.40	Triad color. A natural 11 against a major third pays an avoid-tone surcharge of 0.5.
Altered color (♭9, ♯9, ♯11, ♭13)	0.45–0.55	The alt palette lives on dominant chords; hosting one of these on any other quality doubles its price. The discount requires the dominant's flat seventh to actually sound: a dominant name missing its seventh is a phantom host and pays the doubled rate too. The ♯11 stays unmultiplied where it is idiomatic Lydian color (major, dominant, minor, and sus4 hosts).
Split-degree surcharges	0.4 / 0.6 / 0.8	Two chromatic variants of one degree at once: a natural 2 plus ♭9/♯9, a natural 4 plus ♯11, a natural 6/13 plus ♭13, or a ♯5 plus natural 13
Stacked chromatic color	0.15 / 0.3 / 0.9	A chromatic color stacked among other colors reads as tension the name fails to integrate: an added ♭9 or ♯9 alongside other colors pays 0.15, and a ♭9 on a major sixth or seventh chord among other colors pays 0.3. A lone chromatic color (the harmonic-minor C6♭9, the Phrygian Cmadd♭9) is untouched. Fifthless minor-major ♭9+11 stacks pay 0.9 because they are usually chromatic bookkeeping rather than normal minor-major vocabulary.
Empty fifth slot	0.5 / 0.75	A ♯11 or ♭13 with no fifth-slot tone sounding reads as the altered fifth instead: C-E-G♭ is C(♭5), not a fifthless Cadd♯11, and G-B-F-A-D♯ is G7♯5(9), not G9♭13. Fifthless major-family Lydian stacks with a supporting ninth (D♭maj9♯11) and the harmonic-minor m(maj7)♭13 idiom are exempt, as are minor-third flat-five hosts like Em7(♭5,♭13), whose flat five already occupies the slot.
Missing essential tone	0.5–1.7	By the degree it would fill: a perfect fifth is routinely dropped (0.5), but a chord without its third (1.7), seventh (0.75), suspension (1.4), or defining altered fifth (0.9) is barely that chord. At most one may be missing.
Unexplained tone	2.0	A sounding pitch the name cannot account for at all
Bass placement	0–1.0	Root free; conventional inversion 0.15; integrated extension 0.3; altered fifth 0.3; altered color 0.5; bare add tone 0.65; suspended tone 0.7; unexplained 1.0. Diminished and augmented chords invert freely, so their fifth in the bass counts as a plain core tone rather than an altered fifth. A complete plain triad over its added-ninth bass (the D/E idiom) is treated as an upper-structure pedal instead.
Bare fifthless sixth chord	0.45	A three-tone root-third-sixth set is better read as the relative minor's triad (C-E-A as Am/C, not C6)

This input-centric accounting replaced an earlier template-centric cost model that rewarded each matched template slot and normalized by template size. That formulation systematically favored rare four-tone templates that booked every sounding note as a required tone over more common readings that treat one note as color, and it took a dozen hand-tuned counterweight bonuses to fight the bias. Pricing the input directly removes the bias at the source: a rare name can still win, but only when it explains the voicing decisively more cheaply than any common name.

Extension extraction

During template pricing, any tone not accounted for by the base template (required + optional + penalty) lands in the “extras” mask. A few context-specific penalty tones can be moved into that extras mask first when they function as chord color instead of true contradictions. These get converted to named extensions in the final chord identity, each priced by its role as described above:

• Alterations (from the extras mask): flat 9 (semitone 1), sharp 9 (semitone 3), sharp 11 (semitone 6), flat 13 (semitone 8)
• Split-third add tone: add sharp 9 (semitone 3) when a major-family triad already contains its major third
• Natural extensions: 9 (semitone 2), 11 (semitone 5), 13 (semitone 9)

Whether natural extensions become “9/11/13” or “add9/add11/add13” depends on whether the chord has a 7th. With a 7th present, a 9, 11, or 13 reads as a stacked extension regardless of which lower stack members are also sounding, matching common chord-symbol practice where the inner extensions are freely omitted. Without a 7th, the same pitch class is labeled as an add tone instead, with one hard exception: a bare major or minor triad plus a major sixth is a sixth chord (C6, Cm6), so its add13 relabeling is rejected outright rather than merely priced higher. The rejection spares the case where the sixth is the bass, since the add13 label folds into the slash and the reading survives as the conventional triad-over-sixth-bass symbol (A-C-E as C/A).

Interval 3 is normally a minor third, but the analyzer allows a few narrow musical exceptions where that pitch clearly functions as sharp-nine color instead: dominant 7th shells with ♯9 color, plain major seventh chords with both the major third and major seventh present, and major-family split-third voicings. These exceptions keep common blues, altered-dominant sounds, and explicit altered major-seventh colors from being misread as contradictions.

How the prices were tuned

The prices were not established arbitrarily. They started as musician-judged priors, were calibrated offline against a pool of every distinct 3–7 note pitch-class set, and were then tuned against a set of golden test cases: specific voicings where the expected output was chosen in advance. Most golden cases capture chords a musician would name unambiguously; ambiguous cases pin the intended primary reading for the current cost and ranking model.

The test suite covers major, minor, diminished, dominant, altered, and extended voicings across different inversions and ambiguous situations. The tuning loop looked like this:

1. Run the golden test suite.
2. For any case that failed, use the chord-debug CLI tool to inspect the full ranked candidate list with cost breakdowns.
3. Adjust prices or rules until the failing case passed.
4. Re-run the full suite to verify no regressions.

The chord-debug tool runs the full analysis pipeline on any set of notes and prints each candidate with its cost, individual cost contributions, and the ranking rule that decided its position relative to the previous candidate:

$ dart run tool/chord_debug.dart F# Bb C E

notes: F♯ B♭ C E  |  bass: F♯ (pc 6)  |  key: C major

 1) F♯7♭5          0.40
     members: root=F♯  major3=A♯  flat5=C  flat7=E
     cost: vocab+0.40

 2) C7♭5/G♭        0.70  Δ +0.30
     (vs prev: cost difference beyond tie-break range)
     members: root=C  major3=E  flat5=G♭  flat7=B♭
     cost: vocab+0.40  bass+0.30

 3) F♯7♯11         1.30  Δ +0.90
     (vs prev: cost difference beyond tie-break range)

The same diagnostic output also exposes enharmonic spelling decisions: MIDI provides pitch classes, and the engine chooses note names from the winning chord context.

That kind of diagnostic visibility was essential for understanding why the algorithm chose wrong answers and what needed to change. A weight that fixed one case would sometimes break another, and the only way to make progress without regressing was to have the full ranked list visible while making targeted adjustments.

The ranking problem

The debug output above shows why raw cost is only the first half of the problem. Once multiple readings are plausible, the analysis engine needs a separate ranking layer that encodes musical priorities more directly than a single numeric cost can.

This is not an isolated case. Several common note sets produce near-identical costs for multiple plausible interpretations, and the cost alone cannot distinguish which one a musician would name:

• C-E-G-A: C6 vs. Am7/C (identical costs; the 6th chord in root position should win)
• B-E-G with B in the bass: Em/B vs. G6/B (the complete triad should beat an inverted 6th-chord spelling whose fifth is absent)
• B-D-F-A♭: Bdim7 vs. G♯dim7/B vs. Ddim7/C♭ vs. Fdim7/C♭ (C♭ = B enharmonically; all four readings cost identically due to dim7 symmetry)

The analyzer handles these ambiguities with two ranking paths: narrow structural overrides for cases where the conventional name should win despite cost, and ordered tie-breakers for candidates whose costs are already close.

Hard rules

Hard rules are intentionally narrow guardrails for known failure modes in the cost model. They only fire when a pitch-class-valid but misleading interpretation looks cheaper than the name musicians would normally expect. Each rule is documented in code with the concrete voicing that motivated it, and covered by focused ranking tests so the exception stays bounded.

The near-tie window

The ordered list below applies only after those hard rules have had a chance to run. If none of them fire and the cost difference is greater than 0.25 (the nearTieWindow constant), the lower-cost candidate wins on cost alone.

When costs are within the near-tie window, tie-breaker rules are applied sequentially. The first rule that produces a non-tie result decides the ordering:

The displayed alternatives use the same cost window as a lower bound, then include every ranked candidate through the last cost-window match. This keeps hard-rule ordering coherent when a higher-ranked candidate sits just outside the raw numeric window.

1. Prefer a voicing-supported upper-structure slash: a complete chord stacked above an isolated bass note, when the input carries real octaves
2. Prefer root-position 6th over inverted 7th
3. Prefer a complete triad over an incomplete 6th chord missing its fifth
4. Prefer upper-structure dominant 7th slash
5. Prefer major-seventh upper-structure sus slash
6. Prefer root-position dominant sus, including flat-nine sus colors, over remote slash reinterpretations
7. Prefer flat-nine-bass dominant shells over remote minor-major or diminished reinterpretations
8. Prefer the cleaner-spelled reading of tritone-twin extended dominants (C7alt vs G♭9♯11 shapes), unless one side is a complete natural-thirteenth stack
9. Prefer stable extended dominant inversions over altered-fifth dominant slash
10. Prefer a complete altered dominant thirteenth over an altered minor-thirteenth reading with rarer color
11. Prefer a complete flat-nine flat-thirteen dominant over a remote diminished or seventh-family spelling
12. Prefer complete major-triad ♯11 inversions over sparse major-13-sus4 spellings
13. Prefer a complete major-triad inversion over a seventh-family chord where the bass is only an add-extension
14. Prefer root-position diminished 7th
15. Prefer dominant 7th slash over non-dominant seventh-family slash
16. Prefer a reading that names every tone over one that drops a tone, unless that would promote rarer altered bookkeeping above a lower-cost idiomatic shell
17. Prefer a lower-cost add-chord reading over an unusual seventh-family spelling that omits the third
18. Prefer a harmonic-minor tonic over a split-third major-triad inversion
19. Prefer a lower-cost major-seventh-bass inversion over a slash reading where the bass is only a remote color tone
20. Prefer fewer altered/tension colors
21. Prefer diatonic chords
22. Prefer a root-position relative-minor seventh over the equivalent major-sixth slash reading
23. Prefer the tonic chord
24. Prefer a complete triad with add-tone extensions over an unusual or sparse seventh-family reading that turns the same pitches into remote color
25. Prefer natural extensions (9/11/13) over add-tones, then fewer overall, unless that would reward an incomplete slash chord
26. Prefer root position
27. Prefer the more common name when the corpus shows a strong preference between otherwise equivalent spellings
28. Prefer cleaner spelling for otherwise tied tritone-related flat-five dominant readings
29. Prefer more conventional inversion, based on the bass tone's named role in the candidate rather than its raw interval alone
30. Prefer 7th chords over triads when both fit and the seventh is actually sounding, unless the seventh-family spelling is a suspended slash label with no third competing against a complete sixth-chord reading
31. Prefer fewer extensions
32. Avoid suspended chords
33. Prefer the reading whose members spell more cleanly in context

If all of these rules still have not produced a winner, there is a deterministic fallback: sort by root pitch class numerically. This ensures the output is always consistent for the same input, even for exotic voicings.

The ordering of these rules encodes musical priorities. Structural clarity (root position, shell tones) comes before contextual preferences (diatonic, tonic). Conventional naming (fewer alterations, natural extensions, and common corpus labels) comes before complexity. Suspended chords are deprioritized late because they are valid but easy to over-detect when a third is absent, so they should win only when the surrounding evidence supports them.

Turning the comparison into a stable order

Because hard rules and the near-tie window deliberately override raw cost, the candidate comparison is not guaranteed to be transitive: A can beat B, B can beat C, and yet C can beat A. A generic sort is undefined on a comparison like that and can bury a strong reading below a weaker one.

So the engine linearizes the candidates rather than sorting them directly: it repeatedly takes the one that nothing else outranks, breaking any cycle in a fixed, repeatable way. The result honors every rule above and always produces the same order for a given input.

That linearization is not free. To know which candidate nothing else outranks, the engine has to compare every candidate against every other, which is quadratic in the candidate count and dominates uncached analysis time. The ranking-performance deep dive covers the measurement, dead ends, and pruning work in detail.

Caching for real-time performance

Running the full pipeline (up to 12 candidate roots × 27 templates = 324 template evaluations) on every MIDI state change would be wasteful. In practice, a pianist tends to produce many repeated input states throughout a musical piece.

The engine uses a 512-entry Least Recently Used (LRU) cache implemented as a LinkedHashMap. The cache key is a hash of four inputs:

• The pitch class set and bass note
• The analysis context (key signature + tonality)
• The observed voicing's register signature, when one is supplied, because register evidence can nudge the ranking
• The take parameter (how many candidates to return, default 5)

The context is included in the key because diatonic preference rules depend on it; a different key signature can change which candidate ranks first even for identical voicings.

final key = Object.hash(input.cacheKey, voicing?.signature ?? 0, context, take);
final cached = _cache[key];
if (cached != null) {
  // Promote on hit so eviction removes LRU, not FIFO
  _cache
    ..remove(key)
    ..[key] = cached;
  return cached;
}

The LinkedHashMap preserves insertion order. On a cache hit, the entry is removed and re-inserted at the end (most recently used). On eviction, the first key is removed (least recently used). This is the standard LRU pattern in Dart without a separate doubly-linked list.

The 512-entry capacity was chosen from benchmarks across random inputs, exhaustive inputs, tonal progressions, and simulated live note transitions. Realistic playing showed high reuse, and larger caches produced no material improvement.

What the algorithm does not handle

A few things are known limitations or non-goals:

• Polychords. Two simultaneous independent sonorities (like Stravinsky’s Petrushka chord, an F♯ major triad over a C major triad) are not modeled. The algorithm will find the best single-chord description of the combined note set.
• Temporal context. Each snapshot of sounding notes is analyzed independently. The algorithm does not track what chord came before and does not use progression history to inform interpretation. Using temporal context to further increase accuracy is a natural direction for future improvement.
• Non-12-TET tuning. This engine is built around 12 pitch classes and standard MIDI note numbers. Microtonal intervals, quarter tones, and just-intonation distinctions have no representation in this model.

The cost heuristics are tuned from experience. They encode accumulated musical convention, but they are adjustable constants, not proven axioms. Edge cases and counterexamples help improve them.

The codebase

WhatChord is written in Dart using the Flutter framework. The chord analysis engine lives entirely in a handful of files with no platform dependencies and a unit test suite that verifies known-correct outputs across major, minor, dominant, altered, extended, and ambiguous chord types.

The project is open source and released under the Zero Clause BSD License, which means you are free to use, modify, and share the code however you like.

If you find a misidentified chord, the best way to report it is to long-press the chord card to open Analysis Details, copy the diagnostic output, and open a GitHub issue. The diagnostic output includes the exact pitch classes and context that produced the result, which makes it straightforward to reproduce and debug.