checksum: signal the method, weight by position, reduce mod 31 #21

Merged

hugooconnor merged 0 commits from refs/pull/21/head into main

2026-07-11 11:10:15 +00:00

hugooconnor commented

2026-07-10 03:46:26 +00:00

(Migrated from codeberg.org)

Fixes three defects in the terminal checksum, each of which let a corrupted phrase validate. Verified across the spec, the Guile reference, and the web app.

What was wrong

Dual acceptance. The spec required a validator to compute both the CRC-8 and the additive checksum and accept either. Two independent chances to pass roughly doubles the residual false-negative rate — measured 6.01% vs 3.03% over 200k random substitutions.

Transposition was never detected. The additive checksum was a plain letter sum, which is order-invariant, so every permutation of a phrase validated. Measured detection: 0.00%. This sat directly on the emergency-radio path the additive method exists to serve.

The modulus was wrong. With position weights 1..6, mod 32 leaves gcd(2,32)=2, gcd(4,32)=4, gcd(6,32)=2, so a substitution in the 4th word went undetected 12.65% of the time.

The fixes

Method signaling. The two methods now draw from disjoint vocabularies, so the trailing word identifies its algorithm and the validator computes exactly one checksum. MUST accept either is replaced by an explicit MUST NOT.
Position weighting. index = (Σ i·lettersum(wordᵢ)) mod M. Transposition detection goes 0% → ~97%.
Prime modulus. M = 31. Every weight 1..6 and every weight difference is invertible, so no hole exists to move. (All-odd weights fix substitution but make every difference even, relocating the hole onto transposition.)

Detection is now ~3.2%, uniform across substitution, transposition and omission, at every word position.

Vocabularies rebuilt

The old set was screened for spelling disjointness only. 23 of its 32 words sat within one phoneme of a BIP-39 location word — including gray/grape, and grape is the location word in every worked example — plus quail/whale and seal/sea.

BIP-39's four-letter identity rule is deliberately not applied. It bundles a written hazard (crow/crowd, which a phoneme test misses) with a spoken one (stork/story, which a prefix test misses), catches neither cleanly, and across six Latin-script wordlists rejects Romance cognates like emerald and crimson that are neither. SayWhere never truncates a word, so the rule buys nothing and costs most of the usable vocabulary. It's replaced by an explicit prefix rule and a phoneme-distance rule, plus spelling and familiarity criteria.

scripts/checksum_vocab.py (new) builds, audits and verifies the vocabularies against all ten official BIP-39 wordlists, the CMU Pronouncing Dictionary, and a spoken-English frequency corpus. It reports 0/32 and 0/31 failures for the adopted sets, and scores the superseded ones so the regression is visible.

Verification

Guile, Rust and the web app agree on all 144 phrases (24 locations × 6 precisions) under both methods.
Every worked value was re-parsed out of the spec and recomputed; the field card matches CHECKSUM_ADD_31 index-for-index.
Guile suites pass; the companion Rust PR passes 151 tests.

Known residual, documented on purpose

sum("column") − sum("grape") = 31 exactly, so swapping the first two words shifts the index by 0 mod 31 and column.grape.hip.tremor validates. It's the 1-in-31 residual landing on our own example. Rather than switch to mod 29 to make the demo look clean, the spec carries it as a labelled test vector and Rust pins it as a test. A terminal checksum reduces the chance of accepting a corrupted phrase; it does not eliminate it.

Breaking change

Location-only phrases are unaffected (grape.column.hip still resolves). Previously-shared checksummed phrases break:

phrase	before	after
`grape.column.hip`	ok	ok
`grape.column.hip.toad`	ok	rejected — no longer a checksum word
`kit.puzzle.ruby`	ok	rejected — `ruby` moved index; CRC is now `burgundy`

The kit.puzzle.ruby case surfaces as "checksum failed" rather than "unknown word", which reads like corruption rather than a spec revision. Worth a note in release comms.

Canonical example is now grape.column.hip.hermit (CRC-8) or grape.column.hip.tremor (additive).

Scope note

This branch also carries two pre-existing UI commits (171ee6a mobile wayfinding overlay, bf0a689 onboarding card) that predate the checksum work and were already on origin/geo-qr-pin. They touch web/index.html and the UI layer of web/js/app.js, not the checksum layer.

Companion PR: anuna/saywhere-rs — checksum-and-altitude-conformance.

🤖 Generated with Claude Code

https://claude.ai/code/session_017R8AwqxwAq8Av8GMZvLEPT

Fixes three defects in the terminal checksum, each of which let a corrupted phrase validate. Verified across the spec, the Guile reference, and the web app. ## What was wrong **Dual acceptance.** The spec required a validator to compute *both* the CRC-8 and the additive checksum and accept either. Two independent chances to pass roughly doubles the residual false-negative rate — measured **6.01% vs 3.03%** over 200k random substitutions. **Transposition was never detected.** The additive checksum was a plain letter sum, which is order-invariant, so *every permutation* of a phrase validated. Measured detection: **0.00%**. This sat directly on the emergency-radio path the additive method exists to serve. **The modulus was wrong.** With position weights `1..6`, `mod 32` leaves `gcd(2,32)=2`, `gcd(4,32)=4`, `gcd(6,32)=2`, so a substitution in the 4th word went undetected **12.65%** of the time. ## The fixes 1. **Method signaling.** The two methods now draw from disjoint vocabularies, so the trailing word identifies its algorithm and the validator computes exactly one checksum. `MUST accept either` is replaced by an explicit `MUST NOT`. 2. **Position weighting.** `index = (Σ i·lettersum(wordᵢ)) mod M`. Transposition detection goes 0% → ~97%. 3. **Prime modulus.** `M = 31`. Every weight `1..6` and every weight *difference* is invertible, so no hole exists to move. (All-odd weights fix substitution but make every difference even, relocating the hole onto transposition.) Detection is now **~3.2%, uniform** across substitution, transposition and omission, at every word position. ## Vocabularies rebuilt The old set was screened for spelling disjointness only. **23 of its 32 words sat within one phoneme of a BIP-39 location word** — including `gray`/`grape`, and `grape` is the location word in every worked example — plus `quail`/`whale` and `seal`/`sea`. BIP-39's four-letter identity rule is **deliberately not applied**. It bundles a written hazard (`crow`/`crowd`, which a phoneme test misses) with a spoken one (`stork`/`story`, which a prefix test misses), catches neither cleanly, and across six Latin-script wordlists rejects Romance cognates like `emerald` and `crimson` that are neither. SayWhere never truncates a word, so the rule buys nothing and costs most of the usable vocabulary. It's replaced by an explicit prefix rule and a phoneme-distance rule, plus spelling and familiarity criteria. `scripts/checksum_vocab.py` (new) builds, audits and verifies the vocabularies against all ten official BIP-39 wordlists, the CMU Pronouncing Dictionary, and a spoken-English frequency corpus. It reports **0/32 and 0/31 failures** for the adopted sets, and scores the superseded ones so the regression is visible. ## Verification - Guile, Rust and the web app agree on **all 144 phrases** (24 locations × 6 precisions) under **both** methods. - Every worked value was re-parsed *out of the spec* and recomputed; the field card matches `CHECKSUM_ADD_31` index-for-index. - Guile suites pass; the companion Rust PR passes 151 tests. ## Known residual, documented on purpose `sum("column") − sum("grape") = 31` exactly, so swapping the first two words shifts the index by `0 mod 31` and **`column.grape.hip.tremor` validates**. It's the 1-in-31 residual landing on our own example. Rather than switch to `mod 29` to make the demo look clean, the spec carries it as a labelled test vector and Rust pins it as a test. A terminal checksum reduces the chance of accepting a corrupted phrase; it does not eliminate it. ## Breaking change Location-only phrases are unaffected (`grape.column.hip` still resolves). **Previously-shared checksummed phrases break:** | phrase | before | after | |---|---|---| | `grape.column.hip` | ok | ok | | `grape.column.hip.toad` | ok | rejected — no longer a checksum word | | `kit.puzzle.ruby` | ok | rejected — `ruby` moved index; CRC is now `burgundy` | The `kit.puzzle.ruby` case surfaces as "checksum failed" rather than "unknown word", which reads like corruption rather than a spec revision. Worth a note in release comms. Canonical example is now `grape.column.hip.hermit` (CRC-8) or `grape.column.hip.tremor` (additive). ## Scope note This branch also carries two pre-existing UI commits (`171ee6a` mobile wayfinding overlay, `bf0a689` onboarding card) that predate the checksum work and were already on `origin/geo-qr-pin`. They touch `web/index.html` and the UI layer of `web/js/app.js`, not the checksum layer. Companion PR: `anuna/saywhere-rs` — `checksum-and-altitude-conformance`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_017R8AwqxwAq8Av8GMZvLEPT

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

anuna-research/saywhere!21

No description provided.

Rows
Columns