feat: honour the model's own silence, and spend it on recall - #84
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The vocabulary model has carried a trained
__NADA__class — "nobody is signing", cut byhealth_dataset.pyfrom the gaps between LSE-Health's translated sentences — since theco-articulated retrain. Nothing on this side read it:
classifytreated it as an ordinaryconcept and
createGlosshumanised it to the word__nada__, so the app wrote 131 of 1,477words as
__nada__on held-out signers, andhealth_words.pycounted every one of those as aword written. Over half the pause-babble figure this repo published was the model's own silence.
Nothing is retrained here and
lse-vocabulary.binis byte-identical.What it buys
Four seeds, floor 850, four held-out signers, 1,060 annotated instances and 211 windows landing
in gaps between annotated sentences.
__nada__reaching the transcriptBoth axes improve at once, which the floor/gate grid had never managed. The recall gain holds in
every one of the four seeds, so it is not the best of four.
The two moves are separate and in that order:
the runner-up, which would contradict the answer the model just gave. Measured, this alone
takes pause babble from 27.5% to 10.9% on shipped weights with recall unmoved at 38.1%. A
filtering policy that writes the runner-up scored identically to the word, because under an
abstention no real concept ever cleared the gate; the abstention is what ships and a test
pins the difference.
gate comes down from 0.60 to 0.45.
Notes for review
abstentionConcept, not hardcoded in TypeScript, andcheck_manifest.pynow fails on any__-prefixed concept that goes undeclared. This bug wastwo places having to agree with nothing checking, which is the shape of the handedness and
normalisation bugs already recorded in
tools/train/README.md.0.45, the
stabilizerveto becomes unreachable and a real diagnostic dies. The effectivethreshold a word must clear is 0.45 either way, which is exactly what the bench measured.
abstentionis a fourthWindowVeto. The model answering "nobody is signing" is an answer,not a threshold, and reporting it as
classifiersends anyone reading the panel after a floorthat was never involved.
check_manifest.pysubtracts the abstention so it cannot drift back.What this does not do
No learned abstention for the remaining babble, and
tools/train/README.mdsays why with themeasurement: the class is precise (argmax over a real sign 0.9%, never above 0.60) but of the
pauses where the app was writing a real word,
P(__NADA__) > 0.30in 0.0% of them. Abstentionand remaining babble are disjoint. The negatives that would fix the rest are not labellable in
this corpus — 101 annotated sign types against the 90-150 signs a minute of fluent discourse — and
with 23 of 211 windows left and ±6 points of sampling error, a further 3 or 4 points could not be
demonstrated here even if it worked.
No segmenter changes: floor, grace, window boundaries and
DominantHandTrackerare untouched.Verified:
npm run lint,npm run typecheck,npm test(205 tests, 11 new), the 33 Python benchtests,
check_manifest.py, andnpm run build.