Skip to content

Commit 1c2af6b

Browse files
norandomclaude
andcommitted
spec(model-seam): tasks approved; requirement 10, an indicator of whether a model is good enough
Owner decisions of 2026-11-13: three verdict levels (direct, with the repair loop, not as is), twenty trials with a quick mode of five, a generic measure with a small built-in task set. Requirements Amendment 2, design component Measure, tasks 4.3 and 5.3. Handover for the planning group. kiro-cli check: 73 criteria, 0 errors, 0 warnings. Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 7df01d4 commit 1c2af6b

6 files changed

Lines changed: 242 additions & 15 deletions

File tree

‎.kiro/handover.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Handover: the model seam is specified, its tasks wait for approval
1+
# Handover: the model seam is specified and approved; implementation starts with group 1
22

33
Reset 2026-11-12, 20:54 Europe/Berlin (from the clock), at the end of the day's session. This file is the
44
start for the next session. The file as it stood this morning, with the onboarding decisions in full, is
@@ -17,9 +17,10 @@ start for the next session. The file as it stood this morning, with the onboardi
1717
[R105](../docs/research/2026-11-12/12-model-seam-gate-p1.md),
1818
[R103](../docs/research/2026-11-12/10-model-seam-library-prototype.md),
1919
[R102](../docs/research/2026-11-12/09-product-path-two-models.md).
20-
3. If the owner has approved the tasks: set `approvals.tasks.approved` and `ready_for_implementation` in
21-
`specs/model-seam/spec.json`, write `task_44_handover.md` for this planning group, and start group 1
22-
(the crate `model/`) in the foreground, one commit per task.
20+
3. Update of 2026-11-13: the owner approved the tasks and asked for an indicator of whether a model is
21+
good enough, direct or with the repair loop (Requirement 10, tasks 4.3 and 5.3). The planning group is
22+
closed in [task_44_handover.md](task_44_handover.md); read it, then start group 1 (the crate `model/`)
23+
in the foreground, one commit per task.
2324

2425
## State
2526

@@ -82,7 +83,7 @@ if the transcript is shared.
8283

8384
## For the owner
8485

85-
1. Approve or change the tasks of `model-seam`.
86+
1. Done on 2026-11-13: the tasks of `model-seam` are approved.
8687
2. The tag `v1.2.0` stays yours to set; see `docs/releases/1.2.0.md`.
8788
3. Making the `Step guard` job required is a repository setting.
8889

‎.kiro/specs/model-seam/design.md‎

Lines changed: 90 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@ Generated 2026-11-12 (Europe/Berlin) from the approved [requirements](requiremen
44
Revised 2026-11-12 after the [prototype gate](prototype-gate.md) was assessed as `proceed` (R105, R106, R107).
55
Approved by the owner on 2026-11-12, 20:16. Discovery log: [research.md](research.md).
66

7+
**Amendment 1, 2026-11-13 (owner decision, requirements Amendment 2).** The component Measure is added:
8+
an indicator of a configured model against a task set, with a verdict on whether the model is good enough
9+
direct, good enough with the repair loop, or not good enough as is. Nothing else in the design changes.
10+
711
## Overview
812

913
The model seam is a second binary, `lesscode-model`, with its library. It is the only place in the
@@ -134,14 +138,18 @@ model/
134138
repair.rs the repair loop around a caller's check
135139
check.rs the fixed task of the check command and its judgement by the core's checks
136140
protocol.rs JSON forms of the call and check commands; exit codes
137-
main.rs command line: call, check
141+
measure.rs trials over a task set, the indicator, the verdict rule
142+
taskset.rs a task set and its check; the built-in set and its files
143+
main.rs command line: call, check, measure
144+
tasks/ the built-in task set: four small modules with their requests
138145
tests/
139146
stub_endpoint.rs a local stub service for refusals, stated waits, parts, cut-off, silence
140147
call.rs requirement 2 against the stub
141148
waiting.rs requirement 3 against the stub
142149
failures.rs requirement 4 and the key never shown
143150
repair.rs requirement 6 with scripted replies
144151
check_command.rs requirement 5 through the binary
152+
measure.rs requirement 10 with scripted replies and through the binary
145153
live.rs the first cases, run only when an endpoint file is named in the environment
146154
```
147155

@@ -229,6 +237,10 @@ to `parse_result` and the existing checks. The receipt is written by the core as
229237
| 8.3, 8.4 | the launcher returns `Sent`; the receipt's call fact is built from it |
230238
| 8.6 | the functions that verify earlier receipts (`request_kwargs`, `hash_set`) stay; the replay test stays |
231239
| 9.1, 9.2, 9.3, 9.4 | `record.rs` |
240+
| 10.1, 10.2, 10.3, 10.7, 10.9, 10.12, 10.13 | `measure.rs`: trials through `repair.rs`, counts, cost per accepted answer, trials not run |
241+
| 10.4, 10.5, 10.6, 10.8 | `measure.rs`: the verdict rule, one function over the counts |
242+
| 10.10, 10.11 | `taskset.rs`: the built-in set and the form a caller supplies |
243+
| 10.14, 10.15 | `main.rs measure`: exit codes; trials run in a temporary directory |
232244

233245
## Components and Interfaces
234246

@@ -243,6 +255,7 @@ to `parse_result` and the existing checks. The receipt is written by the core as
243255
| CheckCommand | seam, command line | Is this endpoint usable | 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 7.5 | model/src/check.rs, model/src/main.rs |
244256
| ProcessProtocol | seam, command line | One request in, one reply out, exit codes | 4.4 | model/src/protocol.rs, model/src/main.rs |
245257
| Launcher | core | Start the executable for a model request | 7.2, 8.1, 8.3, 8.4, 8.5 | native/src/seam.rs |
258+
| Measure | seam, command line | The indicator of a model against a task set, and its verdict | 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 10.10, 10.11, 10.12, 10.13, 10.14, 10.15 | model/src/measure.rs, model/src/taskset.rs, model/src/main.rs |
246259

247260
### Call data
248261

@@ -397,6 +410,76 @@ Files: `check.rs`, `main.rs`.
397410
- Exit codes: 0 usable; 3 the endpoint failed; 4 the reply was rejected by the check; 2 usage.
398411
- Writes nothing in the current directory.
399412

413+
### Measure
414+
415+
Files: `measure.rs`, `taskset.rs`, `main.rs`. Added 2026-11-13 with requirements Amendment 2.
416+
417+
The question it answers: is this model good enough for this kind of task, alone or with the repair loop,
418+
or does more have to be built around it. "Good enough" has no meaning without a task set and its check,
419+
so both are named in every report.
420+
421+
```rust
422+
pub struct Task { pub id: String, pub request: CallRequest }
423+
pub struct TaskSet {
424+
pub name: String,
425+
pub check_name: String,
426+
pub tasks: Vec<Task>,
427+
}
428+
429+
pub struct Counts {
430+
pub trials: u32,
431+
pub not_run: u32, // a call failed for a reason of the endpoint
432+
pub first_passed: u32, // the first reply passed the check
433+
pub passed: u32, // passed within the repairs allowed
434+
pub calls: u32,
435+
pub output_tokens: u64,
436+
pub seconds: f64, // waits included
437+
}
438+
439+
pub enum Verdict {
440+
GoodEnoughDirect, // first_passed / run >= 3/4
441+
GoodEnoughWithRepair, // else passed / run >= 9/10
442+
NotGoodEnoughAsIs, // else; an optimiser around the instruction is indicated
443+
// quick mode only
444+
GoodEnoughSoFar, // all five passed
445+
NotGoodEnough, // at most two passed
446+
Undecided,
447+
NoVerdict { why: String }, // more than one fifth of the trials not run
448+
}
449+
450+
pub fn verdict(counts: &Counts, quick: bool) -> Verdict;
451+
452+
pub fn measure(
453+
call: &mut dyn FnMut(&CallRequest) -> Result<Reply, CallError>,
454+
set: &TaskSet,
455+
check: &mut dyn FnMut(&Task, &str) -> crate::repair::Verdict,
456+
trials: u32, // spread evenly over the tasks of the set
457+
repairs: usize,
458+
) -> Counts;
459+
```
460+
461+
- A trial is `with_repairs` on one task. Rates are taken over the trials that ran.
462+
- `verdict` is a pure function of the counts; the thresholds are two constants beside it, with the runs
463+
they come from named in a comment.
464+
- The built-in set: four small Python modules, each with a request in the form of a regenerate turn and a
465+
stated target, checked by the core's `parse_result`, `prepare_candidate` and `evaluate_tier1_static`.
466+
Graded: one that every model tried so far passes at once (the function of R102), one with a public
467+
name that must survive, one with a statement target that a careless rewrite misses, one with state
468+
spread over two functions. Twenty trials are five of each.
469+
- A caller's set: the next spec supplies requests that ask for a simplification and the step flow as the
470+
check; `measure` is the same.
471+
- `lesscode-model measure --endpoint-env FILE [--trials N] [--quick] [--task-set DIR] [--out FILE]`
472+
prints the task set, the check, the model, the counts with their rates, calls, output tokens and time
473+
per accepted answer, and the verdict; `--out` writes the same as JSON. Exit codes: 0 for the three
474+
"good enough" verdicts; 4 for "not good enough", "not good enough as is" and "undecided"; 3 for no
475+
verdict; 2 usage. Trials run in a temporary directory.
476+
- What the verdict means for "a DSPy-style algorithm around the model": the repair loop is the smallest
477+
such algorithm and is always available. "Good enough direct" says it is rarely needed; "good enough
478+
with the repair loop" says it is needed and sufficient; "not good enough as is" says to try an optimiser
479+
around the instruction (`tools/guide-optimizer`) and to measure its result with this same command. The
480+
runs of 2026-11-10 to 2026-11-12 show both outcomes of that: it lifted one model and did nothing for
481+
another.
482+
400483
### Process protocol
401484

402485
Files: `protocol.rs`, `main.rs`.
@@ -470,6 +553,12 @@ Monitoring is the record of calls; nothing is sent anywhere else.
470553
- **Check command** (`tests/check_command.rs`): through the binary against the stub: usable, refused,
471554
rejected; exit codes; at most three requests seen by the stub (5.6); no file in the working directory
472555
(5.7).
556+
- **Measure** (`tests/measure.rs`): the verdict rule at and around each threshold (15 and 14 of 20 first
557+
replies; 18 and 17 of 20 passed; the three quick outcomes; five of twenty not run gives no verdict)
558+
(10.4 to 10.6, 10.8, 10.13); with scripted replies, counts and cost per accepted answer (10.1 to 10.3),
559+
an endpoint failure counted as not run (10.12); through the binary against the stub: the report names
560+
task set, check, model and trials (10.9), exit codes (10.14), a supplied task set (10.11), no file in
561+
the working directory (10.15).
473562
- **Core** (`native/tests/regenerator.rs`): `regenerate run` with a scripted executable in place of
474563
`lesscode-model`; the receipt states what the executable reported as sent and nothing else (8.3, 8.4);
475564
a failed turn's reason on standard error (8.5); the executable missing (7.2 message); the recorded

‎.kiro/specs/model-seam/requirements.md‎

Lines changed: 55 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,19 @@ meanwhile. Where the term "wait" below includes answering time, it is to be read
2424
owner also allowed at most three small calls to the Anthropic endpoint for the first case of 2.7, inside
2525
the existing spend stop.
2626

27+
**Amendment 2, 2026-11-13, 14:21 (owner decision).**
28+
29+
The owner asked for an indicator of how well the model behind the tool performs, for a measure of when
30+
inference is good enough, and for a way to tell whether a DSPy-style algorithm has to be wrapped around a
31+
model or not. Requirement 10 is added. Decided by the owner on the same day: three verdict levels (good
32+
enough direct, good enough with the repair loop, not good enough as is); twenty trials by default and a
33+
quick mode of five; a generic measure with one small built-in task set, so that the next spec can supply
34+
the real task and the step flow as the check. Thresholds come from the runs of 2026-11-10 to 2026-11-12:
35+
first replies accepted 15 of 20 for the model the tool was built against and 7 to 13 of 20 for GLM 5.3;
36+
accepted within two repairs 20 of 20 and 19 to 20 of 20; a model that ended at 15 of 20 with repairs was
37+
not improved by an optimiser (R74, R78, R99). "Which model is better or cheaper" stays out of scope: the
38+
indicator judges one configured model against a task set, it does not rank models.
39+
2740
## Introduction
2841

2942
An adopter who wants a model to do work behind lesscode can today use only the model the tool was built
@@ -44,13 +57,16 @@ describes it. A *call* is one request to the endpoint and its reply. A *wait* is
4457
endpoint accepts or answers a call. The *fence* is the set of existing checks (admission, control cases,
4558
the step flow's verdicts) that decide whether a model's answer is used. A *repair* is a further call that
4659
shows the model its rejected answer and the fence's reasons. The *check command* is what an adopter runs
47-
to find out whether a configured endpoint can be used.
60+
to find out whether a configured endpoint can be used. A *task set* is a fixed list of requests with the
61+
check that judges their replies. A *trial* is one request of a task set taken through the check with the
62+
repairs allowed. The *measuring command* runs trials and reports the indicator and its verdict.
4863

4964
## Boundary Context
5065

5166
- **In scope**: describing an endpoint; one call that behaves the same on every endpoint; waiting with a
5267
stated estimate; reasons for failures on the terminal; the check command; the repair loop around a
53-
caller's own check; the existing model-driven commands using the seam; a record of every call.
68+
caller's own check; the existing model-driven commands using the seam; a record of every call; an
69+
indicator, per task set, of whether a configured model is good enough alone or with the repair loop.
5470
- **Out of scope**: what the fence checks; the language packs and how they are written; which model is
5571
better or cheaper; prices; what a model is asked to do with a codebase; showing a model's answer as it
5672
is produced; a model calling tools.
@@ -202,3 +218,40 @@ compared afterwards without calling it again.
202218
2. Where the endpoint reports reasoning tokens or cached input tokens, the model seam shall record them.
203219
3. The model seam shall record no key and no part of one.
204220
4. The model seam shall write its records where the caller says, and nowhere else.
221+
222+
### Requirement 10: An indicator of whether a model is good enough
223+
**Objective:** As an adopter, I want a measured answer to whether my model is good enough for a kind of
224+
task, alone or with the repair loop, so that I know whether more has to be built around it before I rely
225+
on it.
226+
227+
#### Acceptance Criteria
228+
1. When the user runs the measuring command, the model seam shall take each request of the task set
229+
through the task set's check for the stated number of trials, with the repairs allowed.
230+
2. The measuring command shall report, with their counts, the share of trials whose first reply passed
231+
the check and the share of trials that passed within the repairs allowed.
232+
3. The measuring command shall report the calls, the output tokens and the time per accepted answer, waits
233+
included.
234+
4. When at least three quarters of the first replies passed, the measuring command shall report the
235+
verdict "good enough direct".
236+
5. When fewer than three quarters of the first replies passed and at least nine tenths of the trials
237+
passed within the repairs allowed, the measuring command shall report the verdict "good enough with
238+
the repair loop".
239+
6. When fewer than nine tenths of the trials passed within the repairs allowed, the measuring command
240+
shall report the verdict "not good enough as is" and say that an optimiser around the instruction is
241+
indicated and has to be measured in the same way.
242+
7. The measuring command shall run twenty trials unless the user states another number.
243+
8. Where the user asks for the quick mode, the measuring command shall run five trials and report "good
244+
enough so far" when all five passed within the repairs allowed, "not good enough" when at most two did,
245+
and "undecided" otherwise.
246+
9. The measuring command shall state the task set, the check, the model and the number of trials beside
247+
every verdict.
248+
10. The model seam shall ship one small built-in task set.
249+
11. The model seam shall accept a task set and a check supplied by a caller in place of the built-in one.
250+
12. If a call of a trial fails for a reason of the endpoint, then the measuring command shall count that
251+
trial as not run and not against the model.
252+
13. If more than one fifth of the trials were not run, then the measuring command shall report no verdict
253+
and say why.
254+
14. The measuring command shall end with exit code zero for a verdict of "good enough direct", "good
255+
enough with the repair loop" or "good enough so far", and with an exit code that is not zero
256+
otherwise.
257+
15. The measuring command shall change no file of the project it is run in.

‎.kiro/specs/model-seam/spec.json‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"feature_name": "model-seam",
33
"created_at": "2026-11-12T17:45:00+01:00",
4-
"updated_at": "2026-11-12T20:18:00+01:00",
4+
"updated_at": "2026-11-13T14:20:00+01:00",
55
"language": "en",
66
"phase": "tasks-generated",
77
"approvals": {
@@ -15,10 +15,10 @@
1515
},
1616
"tasks": {
1717
"generated": true,
18-
"approved": false
18+
"approved": true
1919
}
2020
},
21-
"ready_for_implementation": false,
21+
"ready_for_implementation": true,
2222
"prototype_gate": "prototype-gate.md",
2323
"prototype_status": "proceed",
2424
"related": [
@@ -35,5 +35,5 @@
3535
"docs/research/2026-11-12/13-model-seam-gate-p2-p3.md",
3636
"docs/research/2026-11-12/14-model-seam-gate-p4.md"
3737
],
38-
"notes": "2026-11-12: owner statements: language packs are published once and not regenerated; adopters must be able to run GLM or another model behind the tool; core and driver model agnostic, fence because inference is not deterministic; use an existing model library; preferred solution is horizontal compatibility with an estimated wait time; include the internal endpoint on localhost. Discovery brief written and the owner said 'go ahead' to requirements. Requirements generated 2026-11-12 and approved by the owner the same day (\"approved\"), including the four proposed points: the estimate is of a call's wait, not of a whole run; ten minutes total wait per call unless set otherwise; at most two repairs; the check command makes at most three calls. Provisional design, research log and prototype gate (P1 to P4) written the same day; design approval pending the gate; two open decisions for the owner in the design's contradiction table (rows 1 and 5), both decided as proposed. Prototype gate run the same day and assessed as proceed (R105 to R107), with two items carried into implementation. Revised design approved by the owner on 2026-11-12 (\"approved\")."
38+
"notes": "2026-11-12: owner statements: language packs are published once and not regenerated; adopters must be able to run GLM or another model behind the tool; core and driver model agnostic, fence because inference is not deterministic; use an existing model library; preferred solution is horizontal compatibility with an estimated wait time; include the internal endpoint on localhost. Discovery brief written and the owner said 'go ahead' to requirements. Requirements generated 2026-11-12 and approved by the owner the same day (\"approved\"), including the four proposed points: the estimate is of a call's wait, not of a whole run; ten minutes total wait per call unless set otherwise; at most two repairs; the check command makes at most three calls. Provisional design, research log and prototype gate (P1 to P4) written the same day; design approval pending the gate; two open decisions for the owner in the design's contradiction table (rows 1 and 5), both decided as proposed. Prototype gate run the same day and assessed as proceed (R105 to R107), with two items carried into implementation. Revised design approved by the owner on 2026-11-12 (\"approved\"). 2026-11-13: owner said \"we proceed with the model seam\" (tasks approved) and asked for an indicator of the driving model's performance, a measure of inference being good enough and a way to tell whether a DSPy-style algorithm is needed around a model; decided by menu: three verdict levels, twenty trials with a quick mode of five, a generic measure with a small built-in task set. Requirement 10, the design component Measure and tasks 4.3 and 5.3 added."
3939
}

0 commit comments

Comments
 (0)