Skip to content

Commit aba5dd4

Browse files
norandomclaude
andcommitted
research(slice-b): majority and union of three annotation runs (R85)
Majority of three agrees more than single runs in four of five files and keeps as few sites; the 0.80 target in four files is missed. Union of three is steadier in four files and selects more. Index, roadmap and report updated. Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 6898023 commit aba5dd4

156 files changed

Lines changed: 11664 additions & 3 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.kiro/steering/roadmap.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,9 @@
2424
does not fix Rust admission where GEPA did
2525
([R79](../../docs/research/2026-11-11/05-slice-b-miprov2-opus.md)). Annotation-layer baseline run on both models
2626
([R83](../../docs/research/2026-11-11/09-slice-b-annotation-baseline.md)): Opus runs agree on 0.69 to 0.92 of their
27-
sites, three of five Mistral calls give no annotations. Recorded spend about 29.7 of 40 USD.
27+
sites, three of five Mistral calls give no annotations. Two selection rules over three runs tested
28+
([R85](../../docs/research/2026-11-11/11-slice-b-annotation-hardening.md)): the majority of three is more stable
29+
than one run in four of five files but misses the 0.80 target. Recorded spend about 32.9 of 40 USD.
2830
Status on 2026-11-11: measured for pack authoring and for annotation, between runs and between models, and
2931
under two optimisers for pack authoring; not measured: one model over time. Dated next step: the owner
3032
decides by the next session whether annotations get a hardening (sites named by at least two of three
Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
{
2+
"receipt": "slice-b-annotation-ensemble-v1",
3+
"preregistration": "R84",
4+
"scopes": {
5+
"rust-mmhmem-setup-update": {
6+
"runs": 6,
7+
"splits": 10,
8+
"eligible_sites": 71,
9+
"agreement": {
10+
"single": 0.75,
11+
"majority": 0.75,
12+
"union": 1.0
13+
},
14+
"agreement_range": {
15+
"single": [
16+
0.696,
17+
0.917
18+
],
19+
"majority": [
20+
0.708,
21+
0.792
22+
],
23+
"union": [
24+
0.875,
25+
1.0
26+
]
27+
},
28+
"median_sites": {
29+
"single": 21.0,
30+
"majority": 21.0,
31+
"union": 24.0
32+
}
33+
},
34+
"python-mvibe-ratelimit": {
35+
"runs": 6,
36+
"splits": 10,
37+
"eligible_sites": 211,
38+
"agreement": {
39+
"single": 0.818,
40+
"majority": 0.855,
41+
"union": 0.886
42+
},
43+
"agreement_range": {
44+
"single": [
45+
0.706,
46+
0.9
47+
],
48+
"majority": [
49+
0.781,
50+
0.898
51+
],
52+
"union": [
53+
0.814,
54+
0.914
55+
]
56+
},
57+
"median_sites": {
58+
"single": 57.0,
59+
"majority": 56.5,
60+
"union": 66.5
61+
}
62+
},
63+
"typescript-openultracode-state": {
64+
"runs": 6,
65+
"splits": 10,
66+
"eligible_sites": 70,
67+
"agreement": {
68+
"single": 0.842,
69+
"majority": 0.914,
70+
"union": 0.923
71+
},
72+
"agreement_range": {
73+
"single": [
74+
0.711,
75+
0.943
76+
],
77+
"majority": [
78+
0.838,
79+
0.971
80+
],
81+
"union": [
82+
0.872,
83+
0.949
84+
]
85+
},
86+
"median_sites": {
87+
"single": 34.0,
88+
"majority": 34.0,
89+
"union": 37.5
90+
}
91+
},
92+
"typescript-openultracode-verification": {
93+
"runs": 6,
94+
"splits": 10,
95+
"eligible_sites": 64,
96+
"agreement": {
97+
"single": 0.619,
98+
"majority": 0.744,
99+
"union": 0.6
100+
},
101+
"agreement_range": {
102+
"single": [
103+
0.458,
104+
0.867
105+
],
106+
"majority": [
107+
0.684,
108+
0.812
109+
],
110+
"union": [
111+
0.533,
112+
0.667
113+
]
114+
},
115+
"median_sites": {
116+
"single": 15.0,
117+
"majority": 15.0,
118+
"union": 24.0
119+
}
120+
},
121+
"java-commons-csv-lexer": {
122+
"runs": 6,
123+
"splits": 10,
124+
"eligible_sites": 184,
125+
"agreement": {
126+
"single": 0.857,
127+
"majority": 0.956,
128+
"union": 0.872
129+
},
130+
"agreement_range": {
131+
"single": [
132+
0.784,
133+
0.941
134+
],
135+
"majority": [
136+
0.882,
137+
1.0
138+
],
139+
"union": [
140+
0.846,
141+
0.872
142+
]
143+
},
144+
"median_sites": {
145+
"single": 32.5,
146+
"majority": 33.0,
147+
"union": 36.5
148+
}
149+
}
150+
}
151+
}
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
{"utc": "2026-11-11T09:27:39+00:00", "scope": "rust-mmhmem-setup-update", "out_dir": "run1-rust-mmhmem-setup-update", "model": "claude-opus-5-5", "input_tokens": 6754, "output_tokens": 2604, "attempts": 1, "exit_code": 0}
2+
{"utc": "2026-11-11T09:27:40+00:00", "scope": "typescript-openultracode-verification", "out_dir": "run1-typescript-openultracode-verification", "model": "claude-opus-5-5", "input_tokens": 4489, "output_tokens": 2754, "attempts": 1, "exit_code": 0}
3+
{"utc": "2026-11-11T09:27:40+00:00", "scope": "typescript-openultracode-state", "out_dir": "run1-typescript-openultracode-state", "model": "claude-opus-5-5", "input_tokens": 5105, "output_tokens": 3662, "attempts": 1, "exit_code": 0}
4+
{"utc": "2026-11-11T09:28:04+00:00", "scope": "rust-mmhmem-setup-update", "out_dir": "run2-rust-mmhmem-setup-update", "model": "claude-opus-5-5", "input_tokens": 6754, "output_tokens": 2234, "attempts": 1, "exit_code": 0}
5+
{"utc": "2026-11-11T09:28:01+00:00", "scope": "java-commons-csv-lexer", "out_dir": "run1-java-commons-csv-lexer", "model": "claude-opus-5-5", "input_tokens": 9910, "output_tokens": 3775, "attempts": 1, "exit_code": 0}
6+
{"utc": "2026-11-11T09:27:39+00:00", "scope": "python-mvibe-ratelimit", "out_dir": "run1-python-mvibe-ratelimit", "model": "claude-opus-5-5", "input_tokens": 12321, "output_tokens": 7925, "attempts": 1, "exit_code": 0}
7+
{"utc": "2026-11-11T09:28:22+00:00", "scope": "typescript-openultracode-state", "out_dir": "run2-typescript-openultracode-state", "model": "claude-opus-5-5", "input_tokens": 5105, "output_tokens": 3143, "attempts": 1, "exit_code": 0}
8+
{"utc": "2026-11-11T09:28:30+00:00", "scope": "typescript-openultracode-verification", "out_dir": "run2-typescript-openultracode-verification", "model": "claude-opus-5-5", "input_tokens": 4489, "output_tokens": 1941, "attempts": 1, "exit_code": 0}
9+
{"utc": "2026-11-11T09:28:09+00:00", "scope": "python-mvibe-ratelimit", "out_dir": "run2-python-mvibe-ratelimit", "model": "claude-opus-5-5", "input_tokens": 12321, "output_tokens": 5380, "attempts": 1, "exit_code": 0}
10+
{"utc": "2026-11-11T09:28:39+00:00", "scope": "java-commons-csv-lexer", "out_dir": "run2-java-commons-csv-lexer", "model": "claude-opus-5-5", "input_tokens": 9910, "output_tokens": 3564, "attempts": 1, "exit_code": 0}
11+
{"utc": "2026-11-11T09:28:48+00:00", "scope": "rust-mmhmem-setup-update", "out_dir": "run3-rust-mmhmem-setup-update", "model": "claude-opus-5-5", "input_tokens": 6754, "output_tokens": 2612, "attempts": 1, "exit_code": 0}
12+
{"utc": "2026-11-11T09:28:50+00:00", "scope": "typescript-openultracode-state", "out_dir": "run3-typescript-openultracode-state", "model": "claude-opus-5-5", "input_tokens": 5105, "output_tokens": 3219, "attempts": 1, "exit_code": 0}
13+
{"utc": "2026-11-11T09:28:48+00:00", "scope": "python-mvibe-ratelimit", "out_dir": "run3-python-mvibe-ratelimit", "model": "claude-opus-5-5", "input_tokens": 12321, "output_tokens": 5888, "attempts": 1, "exit_code": 0}
14+
{"utc": "2026-11-11T09:29:06+00:00", "scope": "typescript-openultracode-verification", "out_dir": "run3-typescript-openultracode-verification", "model": "claude-opus-5-5", "input_tokens": 4489, "output_tokens": 3448, "attempts": 1, "exit_code": 0}
15+
{"utc": "2026-11-11T09:29:09+00:00", "scope": "java-commons-csv-lexer", "out_dir": "run3-java-commons-csv-lexer", "model": "claude-opus-5-5", "input_tokens": 9910, "output_tokens": 3727, "attempts": 1, "exit_code": 0}
16+
{"utc": "2026-11-11T09:29:15+00:00", "scope": "rust-mmhmem-setup-update", "out_dir": "run4-rust-mmhmem-setup-update", "model": "claude-opus-5-5", "input_tokens": 6754, "output_tokens": 3018, "attempts": 1, "exit_code": 0}
17+
{"utc": "2026-11-11T09:29:36+00:00", "scope": "typescript-openultracode-verification", "out_dir": "run4-typescript-openultracode-verification", "model": "claude-opus-5-5", "input_tokens": 4489, "output_tokens": 2229, "attempts": 1, "exit_code": 0}
18+
{"utc": "2026-11-11T09:29:36+00:00", "scope": "typescript-openultracode-state", "out_dir": "run4-typescript-openultracode-state", "model": "claude-opus-5-5", "input_tokens": 5105, "output_tokens": 3251, "attempts": 1, "exit_code": 0}
19+
{"utc": "2026-11-11T09:29:41+00:00", "scope": "java-commons-csv-lexer", "out_dir": "run4-java-commons-csv-lexer", "model": "claude-opus-5-5", "input_tokens": 9910, "output_tokens": 3638, "attempts": 1, "exit_code": 0}
20+
{"utc": "2026-11-11T09:29:55+00:00", "scope": "rust-mmhmem-setup-update", "out_dir": "run5-rust-mmhmem-setup-update", "model": "claude-opus-5-5", "input_tokens": 6754, "output_tokens": 2976, "attempts": 1, "exit_code": 0}
21+
{"utc": "2026-11-11T09:29:32+00:00", "scope": "python-mvibe-ratelimit", "out_dir": "run4-python-mvibe-ratelimit", "model": "claude-opus-5-5", "input_tokens": 12321, "output_tokens": 7702, "attempts": 1, "exit_code": 0}
22+
{"utc": "2026-11-11T09:30:09+00:00", "scope": "typescript-openultracode-state", "out_dir": "run5-typescript-openultracode-state", "model": "claude-opus-5-5", "input_tokens": 5105, "output_tokens": 3374, "attempts": 1, "exit_code": 0}
23+
{"utc": "2026-11-11T09:30:20+00:00", "scope": "typescript-openultracode-verification", "out_dir": "run5-typescript-openultracode-verification", "model": "claude-opus-5-5", "input_tokens": 4489, "output_tokens": 2308, "attempts": 1, "exit_code": 0}
24+
{"utc": "2026-11-11T09:30:30+00:00", "scope": "java-commons-csv-lexer", "out_dir": "run5-java-commons-csv-lexer", "model": "claude-opus-5-5", "input_tokens": 9910, "output_tokens": 3408, "attempts": 1, "exit_code": 0}
25+
{"utc": "2026-11-11T09:30:00+00:00", "scope": "python-mvibe-ratelimit", "out_dir": "run5-python-mvibe-ratelimit", "model": "claude-opus-5-5", "input_tokens": 12321, "output_tokens": 7869, "attempts": 1, "exit_code": 0}
26+
{"utc": "2026-11-11T09:30:35+00:00", "scope": "rust-mmhmem-setup-update", "out_dir": "run6-rust-mmhmem-setup-update", "model": "claude-opus-5-5", "input_tokens": 6754, "output_tokens": 2802, "attempts": 1, "exit_code": 0}
27+
{"utc": "2026-11-11T09:30:59+00:00", "scope": "typescript-openultracode-verification", "out_dir": "run6-typescript-openultracode-verification", "model": "claude-opus-5-5", "input_tokens": 4489, "output_tokens": 2563, "attempts": 1, "exit_code": 0}
28+
{"utc": "2026-11-11T09:30:56+00:00", "scope": "typescript-openultracode-state", "out_dir": "run6-typescript-openultracode-state", "model": "claude-opus-5-5", "input_tokens": 5105, "output_tokens": 2958, "attempts": 1, "exit_code": 0}
29+
{"utc": "2026-11-11T09:31:00+00:00", "scope": "java-commons-csv-lexer", "out_dir": "run6-java-commons-csv-lexer", "model": "claude-opus-5-5", "input_tokens": 9910, "output_tokens": 3809, "attempts": 1, "exit_code": 0}
30+
{"utc": "2026-11-11T09:30:46+00:00", "scope": "python-mvibe-ratelimit", "out_dir": "run6-python-mvibe-ratelimit", "model": "claude-opus-5-5", "input_tokens": 12321, "output_tokens": 8314, "attempts": 1, "exit_code": 0}

0 commit comments

Comments
 (0)