You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1c2af6b
Browse filesBrowse the repository at this point in the historyBrowse files
spec(model-seam): tasks approved; requirement 10, an indicator of whether a model is good enough
Owner decisions of 2026-11-13: three verdict levels (direct, with the
repair loop, not as is), twenty trials with a quick mode of five, a
generic measure with a small built-in task set. Requirements Amendment 2,
design component Measure, tasks 4.3 and 5.3. Handover for the planning
group. kiro-cli check: 73 criteria, 0 errors, 0 warnings.
Co-Authored-By: Claude <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: .kiro/specs/model-seam/design.md
+90-1Lines changed: 90 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,6 +4,10 @@ Generated 2026-11-12 (Europe/Berlin) from the approved [requirements](requiremen
4
4
Revised 2026-11-12 after the [prototype gate](prototype-gate.md) was assessed as `proceed` (R105, R106, R107).
5
5
Approved by the owner on 2026-11-12, 20:16. Discovery log: [research.md](research.md).
6
6
7
+
**Amendment 1, 2026-11-13 (owner decision, requirements Amendment 2).** The component Measure is added:
8
+
an indicator of a configured model against a task set, with a verdict on whether the model is good enough
9
+
direct, good enough with the repair loop, or not good enough as is. Nothing else in the design changes.
10
+
7
11
## Overview
8
12
9
13
The model seam is a second binary, `lesscode-model`, with its library. It is the only place in the
@@ -134,14 +138,18 @@ model/
134
138
repair.rs the repair loop around a caller's check
135
139
check.rs the fixed task of the check command and its judgement by the core's checks
136
140
protocol.rs JSON forms of the call and check commands; exit codes
137
-
main.rs command line: call, check
141
+
measure.rs trials over a task set, the indicator, the verdict rule
142
+
taskset.rs a task set and its check; the built-in set and its files
143
+
main.rs command line: call, check, measure
144
+
tasks/ the built-in task set: four small modules with their requests
138
145
tests/
139
146
stub_endpoint.rs a local stub service for refusals, stated waits, parts, cut-off, silence
140
147
call.rs requirement 2 against the stub
141
148
waiting.rs requirement 3 against the stub
142
149
failures.rs requirement 4 and the key never shown
143
150
repair.rs requirement 6 with scripted replies
144
151
check_command.rs requirement 5 through the binary
152
+
measure.rs requirement 10 with scripted replies and through the binary
145
153
live.rs the first cases, run only when an endpoint file is named in the environment
146
154
```
147
155
@@ -229,6 +237,10 @@ to `parse_result` and the existing checks. The receipt is written by the core as
229
237
| 8.3, 8.4 | the launcher returns `Sent`; the receipt's call fact is built from it |
230
238
| 8.6 | the functions that verify earlier receipts (`request_kwargs`, `hash_set`) stay; the replay test stays |
231
239
| 9.1, 9.2, 9.3, 9.4 |`record.rs`|
240
+
| 10.1, 10.2, 10.3, 10.7, 10.9, 10.12, 10.13 |`measure.rs`: trials through `repair.rs`, counts, cost per accepted answer, trials not run |
241
+
| 10.4, 10.5, 10.6, 10.8 |`measure.rs`: the verdict rule, one function over the counts |
242
+
| 10.10, 10.11 |`taskset.rs`: the built-in set and the form a caller supplies |
243
+
| 10.14, 10.15 |`main.rs measure`: exit codes; trials run in a temporary directory |
232
244
233
245
## Components and Interfaces
234
246
@@ -243,6 +255,7 @@ to `parse_result` and the existing checks. The receipt is written by the core as
243
255
| CheckCommand | seam, command line | Is this endpoint usable | 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 7.5 | model/src/check.rs, model/src/main.rs |
244
256
| ProcessProtocol | seam, command line | One request in, one reply out, exit codes | 4.4 | model/src/protocol.rs, model/src/main.rs |
245
257
| Launcher | core | Start the executable for a model request | 7.2, 8.1, 8.3, 8.4, 8.5 | native/src/seam.rs |
258
+
| Measure | seam, command line | The indicator of a model against a task set, and its verdict | 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 10.10, 10.11, 10.12, 10.13, 10.14, 10.15 | model/src/measure.rs, model/src/taskset.rs, model/src/main.rs |
"notes": "2026-11-12: owner statements: language packs are published once and not regenerated; adopters must be able to run GLM or another model behind the tool; core and driver model agnostic, fence because inference is not deterministic; use an existing model library; preferred solution is horizontal compatibility with an estimated wait time; include the internal endpoint on localhost. Discovery brief written and the owner said 'go ahead' to requirements. Requirements generated 2026-11-12 and approved by the owner the same day (\"approved\"), including the four proposed points: the estimate is of a call's wait, not of a whole run; ten minutes total wait per call unless set otherwise; at most two repairs; the check command makes at most three calls. Provisional design, research log and prototype gate (P1 to P4) written the same day; design approval pending the gate; two open decisions for the owner in the design's contradiction table (rows 1 and 5), both decided as proposed. Prototype gate run the same day and assessed as proceed (R105 to R107), with two items carried into implementation. Revised design approved by the owner on 2026-11-12 (\"approved\")."
38
+
"notes": "2026-11-12: owner statements: language packs are published once and not regenerated; adopters must be able to run GLM or another model behind the tool; core and driver model agnostic, fence because inference is not deterministic; use an existing model library; preferred solution is horizontal compatibility with an estimated wait time; include the internal endpoint on localhost. Discovery brief written and the owner said 'go ahead' to requirements. Requirements generated 2026-11-12 and approved by the owner the same day (\"approved\"), including the four proposed points: the estimate is of a call's wait, not of a whole run; ten minutes total wait per call unless set otherwise; at most two repairs; the check command makes at most three calls. Provisional design, research log and prototype gate (P1 to P4) written the same day; design approval pending the gate; two open decisions for the owner in the design's contradiction table (rows 1 and 5), both decided as proposed. Prototype gate run the same day and assessed as proceed (R105 to R107), with two items carried into implementation. Revised design approved by the owner on 2026-11-12 (\"approved\"). 2026-11-13: owner said \"we proceed with the model seam\" (tasks approved) and asked for an indicator of the driving model's performance, a measure of inference being good enough and a way to tell whether a DSPy-style algorithm is needed around a model; decided by menu: three verdict levels, twenty trials with a quick mode of five, a generic measure with a small built-in task set. Requirement 10, the design component Measure and tasks 4.3 and 5.3 added."
0 commit comments