diff --git a/.claude/skills/blog-next/SKILL.md b/.claude/skills/blog-next/SKILL.md index 70cc4794f..ff5ec481d 100644 --- a/.claude/skills/blog-next/SKILL.md +++ b/.claude/skills/blog-next/SKILL.md @@ -47,6 +47,43 @@ everywhere:** - §13 is the live queue (§12 retired). §13a/§13d record the traps, including the ones that killed rows in this same file. +## Stage A0 - the calendar, before anything else + +**Run this first. A topic is not "next" in isolation - it is next relative to +what shipped, when, and on what.** Skipping it is how the blog got 11 posts in +three days (2026-08-20 x6, 08-21 x2, 08-22 x3) against a stated capacity of ~6 +per month, then nothing for the twelve days before that. + +```sh +# what shipped, when - the whole calendar in one line +for f in content/blog/*/index.md; do + d=$(grep -m1 '^date:' "$f" | sed -E 's/date: *.?([0-9-]+).*/\1/') + echo "$d $(dirname $f | sed 's|content/blog/||')" +done | sort -r | head -25 +``` + +Check three things, and record each in the topic row: + +1. **Spacing.** How many posts in the last 7 days? Capacity is ~6/month + (`20.09`, three streams). If the last week already holds three, the correct + output is a SCHEDULED row with a future date, not another same-day post. +2. **Theme recency.** Does a post in the last 14 days already carry this + thesis or this proof-signal? If yes the topic is not dead - it is + **deferred**. See the exit list. +3. **Stream and stack balance.** Which of the three streams (Rails Technical / + Snippet Hygiene / Founder-ICP-E) has gone quiet, and which stack? Measured + 2026-08-22 across 620 posts: `rails` 114, `ruby` 100, `css` 16, `llm` 13, + `react` 12 - against `postgres` 3, `laravel` 3, `tailwind` 2, `python` 2. + **Thin does not mean unwanted.** The single highest-impression uncited guide + on the property is Laravel (19,841), and `langchain-python-tutorial` earns + the best CTR of any high-impression page (15 clicks / 7,329). Under-covered + stacks that already rank are the cheapest expansion available; a fourth + Rails post in a week is the most expensive. + +**Publishing three posts on one day is a calendar failure even when all three +are good.** They compete with each other for the same reader and collapse into +one impression of the blog. + ## Stage A - find candidates, then audit them **Search demand cannot generate topics on this property; it can only veto them.** @@ -72,7 +109,31 @@ three sources below, then use search to kill the bad ones. 2. **Paul's raw material**, when he supplies it - one sentence about something that happened is a topic ("the agents cancelled half their backlog"). Treat this as the highest-value source when present; he may not always have it. -3. **Our real work** - this repo and `~/dev/elital`. The 2026-08-20 batch's best +3. **Reddit** - where practitioners complain in their own words, which is the + phrasing a search-purposed post needs. **The `.json` endpoint is BLOCKED** + from this host - `reddit.com/r//top.json` returns HTTP 403 with an HTML + block page and `old.reddit.com` returns a 302 (both verified 2026-08-22). + Do not put that curl in a script. Use `WebSearch` instead, which surfaces + `old.reddit.com` threads WITH their comment text: + ``` + web_search(search_queries=["reddit r/rails discussion this week", + "reddit ExperiencedDevs complaint"]) + ``` + Worth sweeping: `r/rails`, `r/ruby`, `r/laravel`, `r/webdev`, + `r/ExperiencedDevs`, `r/LocalLLaMA`, `r/devops`. Read the COMMENTS, not the + title - the top comment on a complaint thread is usually the real topic, and + a thread with 200 comments and no accepted answer is an unwritten post. + Quote the practitioner phrasing verbatim into the topic row; it is the one + thing this source has that HN and our own work do not. +4. **X/Twitter** - no free API. Use `WebSearch` scoped to the site, or read the + accounts that set the agenda for our stacks via `WebFetch` on nitter-style + mirrors when reachable. If neither works, say so and lean on the other four + sources rather than inventing a trend. +5. **Changelogs and release notes as a trend source** - a framework's own + release is a dated, citable event with a built-in audience: Rails/Ruby + releases, Laravel releases, and the security advisories for both. This is + the highest-signal source for the under-covered stacks in Stage A0. +6. **Our real work** - this repo and `~/dev/elital`. The 2026-08-20 batch's best material was a commit-documented outage. Sanitize: shapes and lessons yes; prompts, model IDs, proprietary numbers no. @@ -153,25 +214,100 @@ reviewer verdict), a rebuilt plan section, or a HOLD. **Fetch sources; do not recall them.** Training memory is not a citation. -1. **Web search** per `blog-pipeline.md` STEP 3 - official docs, release notes, +**Start the slow one first.** NotebookLM deep research takes ~5 minutes and runs +server-side, so `research_start` it BEFORE the web searches and collect it at the +end. Running them in sequence wastes the whole five minutes; running them in +parallel makes the deep sweep free. This is the default, not an optimisation. + +1. **NotebookLM deep research - fire this FIRST** (`notebooklm-mcp`). It searches + the open web and returns ~40 sources with a synthesised report, which is a + different instrument from `web_search`: it goes wider and returns things a + 3-6 word query never surfaces. + + ``` + server_info() # gate - see auth below + research_start(query=..., mode="deep", title=..., source="web") + # mode: "fast" ~30s / ~10 sources · "deep" ~5min / ~40 sources, web only + # ... do the web searches and the code mining while this runs ... + research_status(notebook_id=..., task_id=..., max_wait=600) + research_import(notebook_id=..., task_id=..., cited_only=True) + notebook_query(notebook_id=..., ...) + ``` + + **`research_import` is not optional** - without it the sources are discovered + but never enter the notebook, and `notebook_query` then answers from nothing. + Prefer `cited_only=True`: the report's own citations are the sources that + earned their place, and importing all 40 buries them. + + **Auth gate.** Check `server_info` first and read `auth_status` precisely: + `configured` go · `not_configured` first-time setup · `stale` means expired, + ask the user to run `nlm login` · `unverified` means THE CHECK failed, not the + credentials - try the call anyway rather than sending them to re-auth. + +### NotebookLM also makes VISUALS, not just text + +`studio_create` builds artifacts from the notebook's sources and +`download_artifact` saves them. Verified signatures 2026-08-23: + +``` +studio_create(notebook_id=..., artifact_type="infographic", + infographic_style=..., orientation="landscape") +studio_status(notebook_id=...) # poll until complete +download_artifact(notebook_id=..., artifact_type="infographic", + output_path="content/blog//figure.png") +``` + +Types: `infographic` (PNG) · `mind_map` (JSON) · `slide_deck` (PDF/PPTX) · +`data_table` (CSV) · `report` · `audio` · `video` · `flashcards` · `quiz`. +Poll `studio_status` after creating - generation is asynchronous. + +**Where these earn their place:** + +- `mind_map` BEFORE outlining. It shows how the sources cluster, which is the + fastest way to see that your six planned H2s are really three. +- `data_table` to pull every number the sources state into one CSV, so the + claim-verification pass has a checklist instead of a memory. +- `infographic` as a STRUCTURE draft - what a diagram of this argument wants to + contain - not as the shipped asset. + +**Two limits, and both matter.** + +A generated infographic does not use the house palette +(`#0e0e14` ground, ruby `#cc342d`, purple `#a855f7`, labels INSIDE the diagram) +and will not match `.stitch/design.md`. Shipped covers and in-post figures are +still hand-built SVG exported to PNG with `rsvg-convert`, which is also what the +standing rule requires: the PNG is the artifact, not the source. + +And the numbers inside a generated visual are **generated**. They carry exactly +the same burden as prose and are harder to notice, because a chart reads as a +measurement rather than as a sentence. Verify every figure in a generated visual +at its primary before shipping it, or do not ship it. + +2. **To interrogate sources you already have** (rather than find new ones): + `notebook_create` → `source_add` (`source_type: "url"`, `urls` takes a list) + → `notebook_query`. + +3. **Web search** per `blog-pipeline.md` STEP 3 - official docs, release notes, primary reports. This is Stage B's job; the coordinator starts at the writer and never re-runs research. -2. **NotebookLM** to interrogate a body of sources: `notebook_create` → - `source_add` (`source_type: "url"`, `urls` takes a list) → `notebook_query`. - To *find* sources: `research_start` → `research_status` → - **`research_import`**; without the import nothing enters the notebook. Check - `server_info` first - `stale` means ask the user to run `nlm login`; - `unverified` means the check failed, not that credentials are bad. -3. Ask what a draft needs: what changed and when, the official position, where + +**A NotebookLM report is a LEAD, never a citation.** It is a synthesis over +sources it chose, and the same rule that governs a `web_search` excerpt governs +it: open the primary and quote from there. On 2026-08-22 two figures reached a +topic row through search excerpts - GitHub's "more than one in five code reviews" +and an ISSRE study's findings - and BOTH had to be re-fetched at source before +they could be written down. One of them, the study, turned out to say something +more interesting than the summary implied. +4. Ask what a draft needs: what changed and when, the official position, where practitioners disagree, the strongest counter-argument. Not "summarize this." -4. **Mine our real code** (this repo, `~/dev/elital`) for first-hand material. +5. **Mine our real code** (this repo, `~/dev/elital`) for first-hand material. Sanitize: shapes and lessons yes; prompts, model IDs, proprietary numbers no. -5. Verify every statistic against its source - **and every mechanism too.** How +6. Verify every statistic against its source - **and every mechanism too.** How a tool behaves is a claim, not context; it just reads as reasoning, so it gets waved through where a number would be challenged. Fetch the README or the release notes. Zero fabricated clients, stats, quotes or personas - `.okf/content/claims-canon.md` records "Sarah" as banned. -6. Internal links per `blog-pipeline.md` STEP 3b. +7. Internal links per `blog-pipeline.md` STEP 3b. Stage B output: a sourced digest - every claim with its URL, links verified. @@ -200,11 +336,25 @@ it straight on and let the gates decide. Paul asked for delivery without a human in the loop (2026-08-22), and the stop list below is the whole of what he still owns. -**One post** → the **`blog-write` skill**, with the topic row, research digest, -**the approved outline**, and `premise audited: yes`. That skill owns STEP 4 -onward and is what the user can also invoke directly; it delegates to +**One post** → **INVOKE the `blog-write` skill NOW.** Not "recommend it", not +"name it in the handback" - call it, in this same run, via the Skill tool: + +``` +Skill(skill="blog-write", args=" - topic row 20.09 §, premise audited: yes") +``` + +Pass the topic row, the research digest, the approved outline, and +`premise audited: yes`. That skill owns STEP 4 onward and delegates to `blog-post-coordinator` when agent spawning is available. +**Ending a run by telling the user to run `/blog-write` themselves is a FAILED +run, not a handback** (Paul 2026-08-23). Someone who asked for a post and +received a topic row reasonably concludes the pipeline is broken. The ONLY +reasons not to invoke it are the exits below - HOLD, SCHEDULED-and-parked, or +BLOCKED - and each needs its evidence. "The outline gate could not run" is not +one of them: say so and invoke anyway, because `blog-write` runs its own gates +and a stated gap is worth more than a stalled pipeline. + **Several posts** → `blog-batch-orchestrator` with N; it runs Stages A-C per row. Session dev server, once, never 1313: `PORT=$((20000 + RANDOM % 20000)) bin/dev`. @@ -216,12 +366,25 @@ Full unattended contract, including the completion promise and why it must be about gates rather than quality: `docs/workflows/autonomous-delivery-prompt.md` §"Running CONTENT unattended". -## Three exits, and only three +## Four exits, and only four A run ends in exactly one of these. **Two exits are not enough** - a loop whose -only outcomes are "shipped" or "try again" will always ship something. +only outcomes are "shipped" or "try again" will always ship something. But +three were not enough either: with only SHIP / HOLD / BLOCKED, a good topic that +merely collides with this week's calendar gets recorded as DO-NOT-WRITE and is +lost. That happened on 2026-08-22 and Paul corrected it - **a spacing conflict +is a scheduling decision, not a verdict on the topic.** - **SHIPPED** - gates green, committed, PR open, verdicts quoted. +- **SCHEDULED** - the topic is good and the slot is not. Hand it to + `blog-write` NOW with an explicit future `date:` in the frontmatter; a + future-dated post is normal scheduling, and production skips future content + until the date arrives (`bin/hugo-build` builds it, `rake test:links` does + not). Pre-writing is the point: the research is hot today and cold in a + fortnight. Pick the date from Stage A0 - the next gap of >=3 days on a stream + that has gone quiet. Record the date and the reason for it in the topic row. + **Use this whenever the only objection is "we just published something like + this"** - that is exactly the case it exists for. - **HOLD, with evidence** - a terminal success, not a failure. Every candidate failed its floor, or the queue is dry after rescopes. Record the numbers that killed each one so nobody re-proposes them from intuition next quarter. Never diff --git a/.okf/content/fabrication-ratchet.md b/.okf/content/fabrication-ratchet.md index 44c4d0bbd..67d4d688a 100644 --- a/.okf/content/fabrication-ratchet.md +++ b/.okf/content/fabrication-ratchet.md @@ -80,6 +80,35 @@ disagreement their job - and by the sourcing rules in [claims-canon](/content/claims-canon.md), especially **a replacement is a new claim**, which is the rule this gate's own creation violated. +## Two true quotes can compose into a false claim (2026-08-23) + +The hardest defect a claim pass has found here was built entirely from +verbatim quotes. `dev-shop-ai-code-review-what-to-ask` set Uber's "only 51% of +human-written comments are considered as bugs by the author" against uReview +getting "over 65% of its posted comments addressed", and concluded the machine +reviewer beats the humans. Both quotes exact. Both figures real. **They are +different instruments** - the 65% is Uber re-running uReview five times on the +final commit and checking whether the comment still fires, with no person +involved, while the 51% is a human author's verdict. + +Nothing in the sentence is fabricated and no ratchet can see it, because the +defect lives in the JOIN, not in either half. So the check is not "is this +quote real" but **"were these two numbers produced by the same instrument?"** - +ask it every time a sentence puts two figures in a comparison, especially when +one of them flatters the argument. + +**The prose fix is not the whole fix.** That comparison was also rendered as a +bar chart. Deleting the sentence left `reviewers.svg` asserting it in a form +nobody re-reads, still shipped in the page bundle and still reachable at its +own URL. When a claim is corrected, grep the bundle for a diagram carrying it. + +Related instrument errors from the same pass, all invisible to shape-counting: +a figure scoped "on the OWASP Benchmark positives" quoted as if it spanned the +real-world dataset too; a paper measuring SAST-alert triage cited as if it +measured AI-reviewer comment filtering; and `+15%` attached to "vs 2022 levels" +when that clause governed the OTHER half of the source sentence and the chart +was indexed to 2023. + # Prioritise by impressions, never by indignation When clearing survivors, rank by live GSC impressions. The first sweep diff --git a/.okf/content/voice-rules.md b/.okf/content/voice-rules.md index 82ddb6321..3ebb913df 100644 --- a/.okf/content/voice-rules.md +++ b/.okf/content/voice-rules.md @@ -48,6 +48,15 @@ Full table in 90.11 §1b. - **Progressive disclosure**: orientation blocks orient; thresholds and mechanics belong where the reader acts on them. - **Callout rhythm**: no two adjacent same-form callouts. +- ⚠️ **Measure the brick PER PARAGRAPH, not per run between headings + (2026-08-23).** An agent invented a stricter metric - words between structural + elements (heading, image, table, list) - and it reported 208-word "bricks" in + three posts whose longest ACTUAL paragraphs were 361, 427 and 485 chars, all + comfortably under the 700 line below. Prose was rewritten to satisfy an + instrument nobody had calibrated. Split on blank lines, take the longest + paragraph, compare to 700 chars. This is the second time a self-invented + measurement drove edits to correct work; the rule is the same as everywhere + else - calibrate the instrument before believing its verdict. - **No text bricks (Paul 2026-08-17; threshold calibrated 2026-08-17 after a course-wide audit)**: a brick is a paragraph over **~700 source chars** (~8+ rendered lines). An earlier ~400-char draft of this rule flagged ~300 diff --git a/.okf/log.md b/.okf/log.md index d8b432f92..e5d646157 100644 --- a/.okf/log.md +++ b/.okf/log.md @@ -3530,3 +3530,53 @@ synthesized; contrast walk 122 elements / 0 failures at 1440 and 390 (worst 4.63:1), proven live by an injected 1.67:1 probe; all four buttons clear 3:1 edge / 4.5:1 label; zero console messages, zero non-2xx, zero third-party hosts. + +## 2026-08-23 - Two true quotes can compose into a false claim + +A four-lens cold-eyes panel over four scheduled posts found three defects that +change what a post asserts, plus a section-level rhythm defect. Concept touched: +[fabrication-ratchet](/content/fabrication-ratchet.md), new section "Two true +quotes can compose into a false claim". + +The headline finding is a new defect CLASS, not another instance of an old one. +`dev-shop-ai-code-review-what-to-ask` compared Uber's 51% (human authors judging +comments) with uReview's 65% (addressed comments) and concluded the machine +wins. Both quotes verbatim, both figures real, and the comparison unsupported: +the 65% is uReview re-running itself five times on the final commit, no person +involved. Nothing is fabricated, so no ratchet can see it - the defect is in the +join between two true sentences. The operational check is "were these produced +by the same instrument?", asked whenever two figures appear in a comparison. + +Second-order lesson with teeth: **correcting the prose did not correct the +claim.** The same comparison was drawn as `reviewers.svg`, which survived the +sentence edit, stayed in the page bundle and stayed reachable at its own URL. A +claim fix now includes grepping the bundle for a diagram carrying it. That file +is deleted. + +Also caught: `+15%` two-week code churn attributed to a 2022 baseline when the +source's "vs 2022 levels" governed the reuse signals and the chart was indexed +to 2023; an ISSTA paper measuring SAST-alert triage cited as measuring +AI-reviewer comment filtering; 22.25% quoted without its "on the OWASP +Benchmark positives" scope; an Anthropic non-ranking ("not yet great at") +promoted to a superlative ("the thing these systems are still worst at"); and +an Anthropic/Cognition equivalence asserted as a restatement of Anthropic's, +when their research subagents write no code and so have no writes to serialise. + +The voice half is why the panel is not optional. A single agent counted section +closers: **20 of 26 end on a short declarative verdict generalising the evidence +just given, 13 of those on an is/are copula**, plus ten sentences ranking the +evidence for the reader before showing it (four literally "the third one is the +important one"). That is measurable only across a set, which is exactly what +per-post self-review and any regex cannot reach - and it is what Paul was +reacting to when he called the batch AI slop. + +One panel claim was wrong and was caught by checking it: the ICP lens reported +the hedge landing at ~80% in all four posts; measured body-line positions are +75 / 79 / 64 / 15%. Brief-and-verify applies to reviewers too. + +Gates: hugo-build clean; test:critical 38 runs / 126 assertions / 0 failures; +snap_diff 55 screenshots no failures; test:links 0 errors over 31,948 unique +links - which covered NONE of the four posts, since all are future-dated and +production skips future content. Their internal links were checked by hand. +A green link run over a build that excludes the changed pages is vacuous +evidence; say so rather than quoting the number. diff --git a/STATUS.md b/STATUS.md index 63e7a4ea8..5fd24fdfc 100644 --- a/STATUS.md +++ b/STATUS.md @@ -21,7 +21,7 @@ | LinkedIn (primary demand lane) | **LIVE** — 3 posts published (2026-08-13/18/19), first metrics read 2026-08-20; ICP-E lane on disk: 1 posted + 4 approved (of 10 planned), validation clock running | Post an approved draft (week1-tue / week1-wed) at Stream 0 cadence (3-4/wk) | [`metrics-ledger`](linkedin-posts/metrics-ledger.md) · [`plan`](docs/workflows/linkedin-icp-validation-plan.md) | | 2608 site design system (v2 `/next/` rail) | Paused 2026-08-22 mid-flight; 3 pilots built + voted | Apply the [repositioned pilot copy](docs/projects/2608-site-design-system/20-29-strategy/20.09-repositioned-pilot-copy.md) (2608's own 20.09 — not the content plan); then Paul's 5 decisions | [`2608 README`](docs/projects/2608-site-design-system/README.md) | | Positioning / homepage offer (2608 niche research + [ADR-0007](docs/adr/0007-homepage-main-offer-and-copy.md)) | **Paused 2026-08-22** — 8 research lanes complete, copy drafted + panel-voted, nothing shipped to the live site; landed on master via [#606](https://github.com/jetthoughts/jetthoughts.github.io/pull/606) | Resume via the HANDOFF — read its corrections first: the staffing constraint was briefed backwards, so every lane's discards need re-reading before acting on a shortlist | [`HANDOFF`](docs/projects/2608-niche-research/HANDOFF.md) | -| Blog / SEO (2510) | **Pipeline rebuilt and 3 posts shipped 2026-08-22.** It previously had no invokable way to produce a post - `/blog-next` never writes and the delivery half existed only as agents. Now three skills: `/blog-operator` (the door, REPAIR > UPGRADE > WRITE > RESTOCK), `/blog-next`, `/blog-write`. Live: [what-senior-developers-catch-that-ai-misses](content/blog/what-senior-developers-catch-that-ai-misses/index.md), [how-to-audit-content-you-didnt-write](content/blog/how-to-audit-content-you-didnt-write/index.md), [when-did-a-test-last-fail-on-purpose](content/blog/when-did-a-test-last-fail-on-purpose/index.md), plus an upgrade to claude-code-xp-team-workflow. Fabricated case studies purged from 14 posts; two ratchets live in `marketing_copy_test.rb` (fabrication baseline 9, uncited baseline 38). §1 outreach constraint still stands - Paul drove these directly. | **§13h**: verify the uncited technical guides, highest-traffic first (crewai 6,458 impr, propshaft 6,194, solid-cache 4,891) - re-rank before starting, the table decays. **N9** still parked: the AI code-search benchmark, design owed before any runs. | [`20.09 §13h`](docs/projects/2510-seo-content-strategy/20-29-strategy/20.09-content-plan-revision-aug-2026.md) | +| Blog / SEO (2510) | **Pipeline rebuilt and 3 posts shipped 2026-08-22.** It previously had no invokable way to produce a post - `/blog-next` never writes and the delivery half existed only as agents. Now three skills: `/blog-operator` (the door, REPAIR > UPGRADE > WRITE > RESTOCK), `/blog-next`, `/blog-write`. Live: [what-senior-developers-catch-that-ai-misses](content/blog/what-senior-developers-catch-that-ai-misses/index.md), [how-to-audit-content-you-didnt-write](content/blog/how-to-audit-content-you-didnt-write/index.md), [when-did-a-test-last-fail-on-purpose](content/blog/when-did-a-test-last-fail-on-purpose/index.md), plus an upgrade to claude-code-xp-team-workflow. Fabricated case studies purged from 14 posts; two ratchets live in `marketing_copy_test.rb` (fabrication baseline 9, uncited baseline 38). §1 outreach constraint still stands - Paul drove these directly. **2026-08-23: four more posts written and SCHEDULED (08-26, 09-02, 09-09, 09-16) on [#613](https://github.com/jetthoughts/jetthoughts.github.io/pull/613)**, then put through a four-lens cold-eyes panel that found 3 wrong claims, 2 internal contradictions and a set-level voice tic (20 of 26 section closers ended on a declarative verdict). All fixed except one: `delegate-the-goal-not-the-task` is HELD on a fabricated example (see Blocked on Paul). New defect class recorded in [`fabrication-ratchet`](.okf/content/fabrication-ratchet.md) — two verbatim quotes can compose into a false comparison, and the chart drawing it survives the prose fix. | **§13h**: verify the uncited technical guides, highest-traffic first (crewai 6,458 impr, propshaft 6,194, solid-cache 4,891) - re-rank before starting, the table decays. **N9** still parked: the AI code-search benchmark, design owed before any runs. **Panel gaps NOT closed on #613**: 0 of 4 posts carry a runnable code block (against `blog-writer-reference-samples.md`), and 0 carry a first-party number. | [`20.09 §13h`](docs/projects/2510-seo-content-strategy/20-29-strategy/20.09-content-plan-revision-aug-2026.md) | | 2605 course | v2 live; measuring. Diagnosis: **arrival, not content**; course SEO/AEO **closed** (Paul 2026-08-21) | LinkedIn arrival-test cards **LI-0…LI-D** in [`content-plan`](linkedin-posts/content-plan.md); no new funnel posts on the unproven bridge | [`TASK-TRACKER`](docs/projects/2605-tech-for-non-technical-founders/TASK-TRACKER.md) | | 2607 campaign tasks | Cold-public-sourcing premise tested and **failed** (3 sweeps, 4 venues, 0 verified-fresh rows; Reddit still un-openable) | Sept-restart Paul decision: retire the cold lane or buy Reddit API access (backlog §c) | [`2607 backlog`](docs/projects/2607-vibe-code-rescue/backlog.md) | | Test/CI hygiene | **Gates rebuilt 2026-08-22.** Fault-injecting 8 realistic defects caught **3**; now catches **8** ([#576](https://github.com/jetthoughts/jetthoughts.github.io/pull/576), audit in [`20.11`](docs/20-29-testing-qa/20.11-gate-fault-injection-2026-08-22-reference.md)). Link job was excluding 90% of links — 114,050 checked now vs 15,642, and it found 5 real site defects ([#574](https://github.com/jetthoughts/jetthoughts.github.io/pull/574)). `bin/dtest` was comparing **nothing** from a worktree ([#578](https://github.com/jetthoughts/jetthoughts.github.io/pull/578)). CI Linux screenshot job green (run 32565008850) | **Paul decides the dtest arch policy** (below); otherwise nothing queued | [`test-gates`](.okf/build/test-gates.md) | @@ -37,5 +37,7 @@ | **Joy Adamson override** (1 min; Paul's one override candidate from outreach batch 1, still publicly unanswered) | [`20.09 §1`](docs/projects/2510-seo-content-strategy/20-29-strategy/20.09-content-plan-revision-aug-2026.md) desk table + 2607 backlog card #12 | | Three 2605 fabricated-fact findings (five-tech-words client claim, $78K/$400 story, SVG chart stats) | [`2605 TASK-TRACKER`](docs/projects/2605-tech-for-non-technical-founders/TASK-TRACKER.md) §Aug-20 sweep | | 2607 T3 (Gmail warm-source consent) + T10 (split strategy docs, [#449](https://github.com/jetthoughts/jetthoughts.github.io/issues/449)) | [`2607 backlog`](docs/projects/2607-vibe-code-rescue/backlog.md) | +| **The real reviewer anecdote** for `delegate-the-goal-not-the-task` (scheduled 09-16, on [#613](https://github.com/jetthoughts/jetthoughts.github.io/pull/613)). Its before/after prompt pair describes a tenant-scoped caching layer that does not exist in this repo (`git grep tenant` → nothing), and the closing section asserts it as a real instruction Paul sent. Post is HELD until the actual one replaces it | [`#613`](https://github.com/jetthoughts/jetthoughts.github.io/pull/613) + panel findings in the PR body | +| Optional: regenerate the `dev-shop-ai-code-review-what-to-ask` cover — its chip reads "22.25% of real bugs suppressed as noise" while the body now scopes that figure to the OWASP synthetic benchmark | [`#613`](https://github.com/jetthoughts/jetthoughts.github.io/pull/613) | -Last updated: 2026-08-22 +Last updated: 2026-08-23 diff --git a/content/blog/agent-team-success-rate/cover.png b/content/blog/agent-team-success-rate/cover.png new file mode 100644 index 000000000..fc620dd9e Binary files /dev/null and b/content/blog/agent-team-success-rate/cover.png differ diff --git a/content/blog/agent-team-success-rate/index.md b/content/blog/agent-team-success-rate/index.md new file mode 100644 index 000000000..af6b1f165 --- /dev/null +++ b/content/blog/agent-team-success-rate/index.md @@ -0,0 +1,101 @@ +--- +title: "Autonomous Agent Teams Are Real. Measured Success Runs 13% to 59%." +description: "Six multi-agent frameworks, measured across coding and reasoning benchmarks. Then the pattern both the loudest sceptic and the loudest proponent ended up agreeing on: many agents may think, one agent writes." +date: 2026-09-09 +draft: false +author: "Paul Keen" +slug: agent-team-success-rate +keywords: 'multi agent success rate, autonomous ai agents team, multi agent llm failure, ai agent orchestration, coding agents production' +tags: ['ai', 'agents', 'engineering', 'architecture'] +categories: ['Engineering'] +cover_image: "cover.png" +cover_image_alt: 'Obsidian-dark cover reading Autonomous agent teams are real, measured success runs 13 to 59 percent, with a faceted ruby gem and three chips: six frameworks measured, 15x the token bill, writes stay single-threaded' +metatags: + image: cover.png +canonical_url: 'https://jetthoughts.com/blog/agent-team-success-rate/' +related_posts: false +--- + +You can stand up a team of autonomous agents this afternoon. Whether it produces anything you would ship is a separate question. + +A group at UC Berkeley collected execution traces from seven multi-agent frameworks and published how often each one finished its task correctly. Six of them, on their own benchmarks: + +| Framework | Success on its own benchmark | +|---|---:| +| AG2 | 59.0% | +| MetaGPT | 40.0% | +| Magentic-One | 38.0% | +| ChatDev | 33.3% | +| HyperAgent | 25.3% | +| AppWorld | 13.3% | + +If you go and check that figure, it is captioned "Failure rates" - I have flipped it, because the chart's own legend labels those segments Success. The caption warns against reading it as a ranking: "Performances are measured on different benchmarks, therefore they are not directly comparable." AppWorld and AG2 are not attempting the same thing. The caption also names the models, GPT-4o and Claude-3.7-Sonnet, which is not what you would wire up today. Whether the failures got better when the models did is unmeasured. + +Even allowing for all of that, most runs failed. The paper's first line is "Despite enthusiasm for Multi-Agent LLM Systems (MAS), their performance gains on popular benchmarks are often minimal." + +## The failures look like a bad org chart + +The same team sorted those traces into fourteen failure modes and three groups: system design, inter-agent misalignment, and task verification. In plainer words, the specification was wrong, the agents talked past each other, or nobody checked the result. + +None of the three is "the model could not do it." Every one of them is a management problem, and you can find all three in a company that has never used an agent. + +## The other side has numbers too + +Anthropic runs a multi-agent research system, a lead agent directing subagents, and reports it "outperformed single-agent Claude Opus 4 by 90.2% on our internal research eval." For the right shape of work, the architecture wins by a lot. + +They are equally plain about the price. "Agents typically use about 4× more tokens than chat interactions, and multi-agent systems use about 15× more tokens than chats." Spending is also most of what separates a good configuration from a bad one: across their browsing evaluations "token usage by itself explains 80% of the variance," with tool-call count and model choice as the other two factors. + +Then they say who it is not for: + +> most coding tasks involve fewer truly parallelizable tasks than research, and LLM agents are not yet great at coordinating and delegating to other agents in real time. + +## Cognition changed its mind in public + +In June 2025 Cognition, who build Devin, published *Don't Build Multi-Agents*. Their complaint was not about model quality. Split a job between two subagents and each one makes reasonable local choices the other never sees, so you get two coherent halves that do not fit together. + +Ten months later Walden Yan published the follow-up. He did not recant and he did not double down. He narrowed the claim: + +> we've found a narrower class of patterns that do [work]: setups where multiple agents contribute intelligence to a task while writes stay single-threaded. + +Anthropic's system reaches the same shape from the other end - many subagents searching and reading, one lead agent holding the result - though theirs is a research system whose subagents never touch code, so it has no writes to serialise in the first place. Yan's rule is the version that applies when they do: any number of agents may read, reason and argue, and one of them is allowed to change the code. + +![Diagram contrasting two agents both editing a codebase and producing incompatible halves, against several agents reading and checking while a single agent owns every change](writers.svg) + +When two agents edit the same code, their disagreement shows up at integration, long after either of them made the choice that caused it. Send every change through one writer and the same disagreement arrives as a review comment, while there is still time to do something about it. + +## What we do + +Ask first whether the job has parts that can genuinely happen at once. Reading twenty sources does. Building a feature usually does not, because step four depends on something decided in step two, and an agent that never saw step two will contradict it with total confidence. + +We lost review findings between agents for months before we accepted that and stopped letting more than one of them write. What replaced it is four lines: + +```text +One author owns the diff. Nobody else writes. +Reviewers get the artifact and the goal, never my conclusion. +Every reviewer must name one thing they would cut. +Author never reviews their own change. +``` + +Line one is the topology the papers describe. Lines two and three exist because once the topology is fixed, the next failure is agreement: a reviewer told what you already concluded will agree with you. Making it name a cut forces it to have an opinion of its own. + +On our own projects that runs the whole loop. Agents take work off a queue and open pull requests, each change written by one of them and reviewed by others - the rule is one writer per diff, not one writer overall. A short written list says what comes back to a human: pricing, anything published outward, whether a claimed number is real, and anything both split and irreversible. They settle the rest. + +Two limits on that. It is our codebases, not client delivery. And that list of things which come back to a human is a component of the system rather than an apology for it. + +The implementation, if you want that rather than the argument, is [our multi-agent pipeline in Rails](/blog/multi-agent-llm-rails-rubyllm/), including the part where one agent turned out to be enough. + +Budget for the 15× before you start. A system that is right somewhat more often and costs ten times more per run is only worth it if a wrong answer is expensive, and how expensive a wrong answer is depends on your business rather than on agents. + +## So how many of them work? + +Between 13% and 59% on the published benchmarks, running the wide-open shape most of those frameworks attempted. + +Ours works, and I am not going to put a number on it. The loop runs without a human in it between those checkpoints and produces work we ship, which is a description rather than a measurement. Printing a percentage next to Cemri's would be the exact thing this post is complaining about. + +Which leaves the question worth pointing at yourself. What share of your agent runs produced something you shipped without rework? Nobody I have asked can answer it, ourselves included, and in the absence of that number everyone has been going on impressions. + +## Sources + +- Mert Cemri et al., [Why Do Multi-Agent LLM Systems Fail?](https://arxiv.org/abs/2503.13657), arXiv 2503.13657, NeurIPS 2025 Datasets & Benchmarks track. +- Anthropic, [How we built our multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system). +- Walden Yan, Cognition, [Don't Build Multi-Agents](https://cognition.com/blog/dont-build-multi-agents) (June 2025) and [Multi-Agents: What's Actually Working](https://cognition.com/blog/multi-agents-working) (22 April 2026). diff --git a/content/blog/agent-team-success-rate/writers.svg b/content/blog/agent-team-success-rate/writers.svg new file mode 100644 index 000000000..57835b979 --- /dev/null +++ b/content/blog/agent-team-success-rate/writers.svg @@ -0,0 +1,46 @@ + + The shape that fails and the shape both labs converged on + Parallel writers fail: two agents each edit the codebase without seeing the other's decisions, producing incompatible halves. The pattern that works keeps writes single-threaded: several agents read and check, and one agent makes every change. + + + + Many agents may think. One agent writes. + + + PARALLEL WRITERS - THE SHAPE THAT KEEPS FAILING + + + Agent A + + Agent B + + → + → + + + both edit + the codebase + + Neither saw the other's decisions. + Two reasonable halves that disagree. + + + + + SINGLE-THREADED WRITES - WHAT BOTH LABS CONVERGED ON + + + reads + + checks + + → + → + + + ONE writer + owns every change + + The extra agents contribute + intelligence, not actions. + diff --git a/content/blog/delegate-the-goal-not-the-task/cover.png b/content/blog/delegate-the-goal-not-the-task/cover.png new file mode 100644 index 000000000..17025700f Binary files /dev/null and b/content/blog/delegate-the-goal-not-the-task/cover.png differ diff --git a/content/blog/delegate-the-goal-not-the-task/index.md b/content/blog/delegate-the-goal-not-the-task/index.md new file mode 100644 index 000000000..b78264be8 --- /dev/null +++ b/content/blog/delegate-the-goal-not-the-task/index.md @@ -0,0 +1,102 @@ +--- +title: "Stop Handing Out Tasks. Hand Out the Goal." +description: "A task list caps the result at your own understanding of the problem. Here is the one clause we added to our reviewer instructions that turned approvals into findings." +date: 2026-09-16 +# HELD 2026-08-23: the before/after prompt pair below describes a tenant-scoped +# caching layer that does not exist in this repo, and "Try it on the next thing +# you send" asserts it as a real instruction. Flip to false once the real pair +# replaces it. See STATUS.md > Blocked on Paul. +draft: true +author: "Paul Keen" +slug: delegate-the-goal-not-the-task +keywords: 'delegation, engineering management, commander intent, ai agents delegation, briefing teams, outcome delegation' +tags: ['management', 'engineering', 'ai', 'teams'] +categories: ['Engineering'] +cover_image: "cover.png" +cover_image_alt: 'Obsidian-dark cover reading Stop handing out tasks, hand out the goal, with a faceted ruby gem and three chips: a task list caps at your own understanding, name one thing you would cut, goals need constraints' +metatags: + image: cover.png +canonical_url: 'https://jetthoughts.com/blog/delegate-the-goal-not-the-task/' +related_posts: false +--- + +We had an AI code reviewer that approved everything. + +Not carelessly. It read the diff, it applied the criteria, and it came back with some version of "this looks good, here are two small suggestions." Every time. The work was not that good, and I knew it was not that good, which is the only reason I noticed. + +The instruction was the problem. Here is roughly what we had been sending: + +```text +Review this change. I think the caching layer is wrong - +check whether the cache key handles the tenant scope, and +confirm the tests cover the invalidation path. +``` + +Read it as the reviewer. Three decisions are already made: that the caching layer is the risk, that tenant scope is the specific worry, and that invalidation is what tests should cover. The only work left is confirmation. So it confirmed, and handed my own opinion back to me with a second signature on it. + +## What we changed was one clause + +```text +Review this change. Goal: I need to be able to change the +caching layer in six months without breaking tenant isolation. + +Tell me what you would cut, and name at least one thing. +``` + +No mention of cache keys. No mention of invalidation. The reviewer came back about a race in the warm-up path that I had not been looking at, because I had not thought of it, because if I had thought of it I would have put it in the prompt and capped the answer again. + +That is the whole mechanism. "Check whether X is wrong" is a task, and completing it faithfully produces a confirmation. "Here is what I need to be true, tell me what should go" is a goal, and it cannot be completed without forming an opinion. + +Our own written instructions now carry the general form: + +> Brief others with evidence, never with your conclusions - a panel handed your inference will return it wearing independent-sounding confidence. + +## Why a task list caps the result + +When you assign a task, you have already made the decisions that matter. You decided what the problem is, what approach fits, and what the steps are. Whoever executes it can do that faithfully and cannot do better, because the ceiling is your understanding of the problem at the moment you wrote the ticket. + +That is fine when your understanding is complete. Most of the time it is not, and the person doing the work is about to learn things you did not know when you assigned it. A task list gives them nowhere to put that. + +Delegating the goal inverts it. You supply purpose, what success looks like, and the constraints that must hold. They supply the route. When the situation turns out different from your assumption - and it will - they can adapt without waiting for you, because they know what the point was. + +Military doctrine has a name for this: mission command. + +## The same instruction, both ways + +| Handed out as a task | Handed out as a goal | +|---|---| +| "Add caching to the search endpoint" | "Search feels slow on the dashboard. Get it under 200ms without stale results." | +| "Write tests for the payment module" | "I want to be able to change the payment module without fear. Show me what you would trust." | +| "Review this PR for SQL injection" | "This touches auth. Tell me what you would not ship, and what you would cut." | + +The left column is easier to write and easier to measure. It is also the version where you find out on delivery that caching was the wrong fix, the tests cover the code rather than the risk, and the reviewer found no SQL injection because there was none - while missing the session handling that was actually broken. + +## Where this goes wrong + +Goals without constraints produce creative compliance. "Make the dashboard faster" gets you a dashboard that loads instantly and shows yesterday's numbers. The constraint was in your head, it never made it into the brief, and what came back satisfied every word you said. + +So the goal has to carry its boundaries. Not steps - boundaries. What must remain true, what you will not accept, and where the edges are. "Under 200ms" is a goal. "Under 200ms without stale results" is a goal somebody can actually pursue without accidentally destroying something you cared about. + +The second failure is subtler. Goal delegation needs the person or the agent to have enough context to make the decisions you are handing over. Give a goal to someone who does not know the system and you have not empowered them, you have abandoned them. That is not an argument for task lists. It is an argument for the context arriving first. + +This bites harder with agents than with people, and Anthropic's engineers name the reason in their write-up of their own multi-agent system: LLM agents are "not yet great at coordinating and delegating to other agents in real time." A human team repairs a bad brief by asking each other what you meant. Agents mostly do not ask, so whatever you left out of the instruction stays out. + +## It is the same rule for people and for agents + +I did not arrive at this from management books. I arrived at it because that reviewer was producing agreeable nonsense, and the clause that fixed it turned out to be the thing I should have been doing with people for years. + +The mechanism is identical. A junior developer told exactly what to do will do it and not tell you the requirement was wrong. A senior developer given the goal will come back and say the feature should not exist. You hired the second one for that, and then you brief them like the first one. + +Structuring a team of agents - who may write, who only reads - is a separate problem, and I went through [the measured success rates](/blog/agent-team-success-rate/) elsewhere. + +## Try it on the next thing you send + +Open the last instruction you wrote, to a person or a model, and look for your own conclusion in it. Mine was hiding in the words "I think the caching layer is wrong." Take it out, put the evidence in its place, and add the clause. + +Two warnings. The first week is unpleasant, because you find out what people were not telling you. And you have to take the cuts: ask for them twice, override them twice, and you have taught everyone the question was rhetorical, which leaves you worse off than before you asked. + +And if what comes back is alarming enough that you want another pair of eyes on it, [that is a thing we do](/services/vibe-code-rescue/). + +## Sources + +- Anthropic, [How we built our multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system) - on the current limits of agents delegating to each other. diff --git a/content/blog/dev-shop-ai-code-review-what-to-ask/cover.png b/content/blog/dev-shop-ai-code-review-what-to-ask/cover.png new file mode 100644 index 000000000..344a9b9e1 Binary files /dev/null and b/content/blog/dev-shop-ai-code-review-what-to-ask/cover.png differ diff --git a/content/blog/dev-shop-ai-code-review-what-to-ask/index.md b/content/blog/dev-shop-ai-code-review-what-to-ask/index.md new file mode 100644 index 000000000..7e8360e2f --- /dev/null +++ b/content/blog/dev-shop-ai-code-review-what-to-ask/index.md @@ -0,0 +1,111 @@ +--- +title: "Your Dev Shop Says the Code Was Reviewed. By What?" +description: "Copilot now takes part in more than one in five code reviews on GitHub. The research says AI reviewers earn their keep - and that the tuning which makes them usable suppressed about a fifth of real vulnerabilities on a standard benchmark." +date: 2026-08-26 +draft: false +author: "Paul Keen" +slug: dev-shop-ai-code-review-what-to-ask +keywords: 'ai code review, dev shop code quality, non-technical founder, vendor due diligence, ai generated code security, outsourced development' +tags: ['ai', 'startup', 'code-review', 'hiring', 'security'] +categories: ['Startups'] +cover_image: "cover.png" +cover_image_alt: 'Obsidian-dark cover reading It was reviewed. By what? Ask what it ignored, with a faceted ruby gem and three chips: 22.25% of real bugs suppressed as noise, over 50% missed in crypto categories, 1 in 5 reviews on GitHub are Copilot' +metatags: + image: cover.png +canonical_url: 'https://jetthoughts.com/blog/dev-shop-ai-code-review-what-to-ask/' +related_posts: false +--- + +"It's been reviewed" used to mean a person read it. + +That is no longer a safe assumption, and the change happened fast enough that most founders never got told. GitHub published the number in March: Copilot code review usage "has grown 10X, now accounting for more than one in five code reviews on GitHub." If your shop works on GitHub, there is a reasonable chance the reviewer on your last release was software. + +## The case for the machine is stronger than the case against + +Uber built an AI reviewer called uReview and published what it does. The system now analyses over 90% of the roughly 65,000 diffs that land at Uber each week, inside a median of four minutes. + +They publish three numbers about comment quality. By Uber's measurement, "only 51% of human-written comments are considered as bugs by the author and addressed in the same changeset." uReview gets "over 65% of its posted comments addressed." And "engineers who interact with the tool" mark 75% of its comments useful. + +It is tempting to read the first two as a scoreboard, and they are not one. The 51% is a human author's verdict. The 65% is the system scoring itself - Uber checks whether a comment was addressed by re-running uReview five times on the final commit and seeing whether it still fires. Two different instruments, and only one of them involves a person deciding the comment was right. + +What the numbers do support is narrower and still worth knowing: at Uber's scale, nobody has produced evidence that the machine reviewer is the weak link. Anyone who tells you AI review is uniformly sloppy is arguing against a measurement they have not read. + +A separate benchmark on C# code found language models beating the established scanners at finding real vulnerabilities. The researchers were careful about what that means: "their noisier output and imprecise localisation limit their standalone use in safety-critical audits." + +## Noise is the problem everyone actually solved for + +An AI reviewer that comments on everything gets ignored, then switched off. So deployments filter. Uber's own write-up describes pruning low-confidence alerts, merging duplicate comments, and automatically suppressing whole categories with "historically low developer value." + +That is sensible engineering. It is also where the risk moves, and a paper accepted to ISSTA 2026 measured what that kind of filtering costs. Their subject is a neighbouring tool - LLM agents triaging security-scanner alerts rather than an AI reviewer's comments - so read it as evidence about the filtering decision, not about uReview. + +Yunpeng Xiong and Ting Zhang at Monash University tested three agent frameworks against the OWASP Benchmark and real Java vulnerabilities. The filtering works: an initial false-positive rate above 92% fell to as low as 6.3% in the best configuration. Then they checked what went missing along with the noise. Their third stated lesson: + +> Aggressive FP suppression risks hiding real vulnerabilities and should not be fully automated. + +The measurement behind it: even the best-performing configuration "incorrectly suppresses 22.25% of real vulnerabilities on the OWASP Benchmark positives." Roughly one in five genuine findings on a synthetic benchmark, quietly reclassified as noise. + +It gets worse in a specific place. The miss rate is close to nothing for the injection-style bugs everyone knows about, and it "exceeds 50% for cryptography- and policy-related categories" - weak encryption, weak password hashing, trust boundaries. Those are the failures a founder would care most about, and they are the ones the filter is worst at keeping. + +![Diagram showing the filter cutting false alarms from over 92 percent to 6.3 percent, while also suppressing 22.25 percent of real vulnerabilities and over 50 percent in cryptography and policy categories](suppressed.svg) + +## Why this lands harder on AI-written code + +A study at ISSRE 2025 compared more than 500,000 code samples, human-written against output from ChatGPT, DeepSeek-Coder and Qwen-Coder. Machine code and human code turn out to fail in different directions. AI-generated code is "generally simpler and more repetitive," while human-written code "exhibits greater structural complexity and a higher concentration of maintainability issues." + +And then: "AI-generated code also contains more high-risk security vulnerabilities." + +Put the two results next to each other. AI writes code that carries more high-risk security problems. AI review is weakest at exactly the security categories where suppression is worst. If your shop uses AI to write and AI to review, the gap in the reviewer lines up with the weakness in the writer. + +## Four things you can check without reading code + +None of these require you to evaluate a line of anything. They are artifacts - either they exist or they do not, and asking for them tells you something either way. + +| Ask for | What a good answer looks like | +|---|---| +| The review configuration | Which categories are suppressed, and who decided. **Somebody chose the filter deliberately instead of accepting a default.** | +| A recent review with comments | Real comments, some disagreed with, some acted on. **Reviews are read rather than rubber-stamped.** | +| The security scan, separately | A named tool, run on a schedule, with its own output. **Security is not delegated to the same filter that optimises for quiet.** | +| Who signs off | A person's name, and what they check that the tool does not. **Accountability did not evaporate into the pipeline.** | + +Run something deterministic for the security categories, and do not let the tool that was tuned for developer patience be the only thing standing between you and a weak hashing bug. + +If you want it as something to paste into an email, this is the whole thing: + +```text +Four questions about our review process, no rush: + +1. What categories does our AI reviewer suppress, and who chose them? +2. Can you send me a recent review with its comments, including + any the author disagreed with? +3. Do we run a separate security scan that is not the same tool? + Which one, and on what schedule? +4. Who signs off on a merge, by name, and what do they check that + the tooling does not? +``` + +## What this does not tell you + +The suppression study is Java, against a benchmark and one real-world dataset. The scanner comparison is C#. Neither is Rails, or Python, or whatever your product is built in, and a number measured on one stack does not transfer to another just because it is inconvenient to re-measure. + +The headline numbers also look like they disagree. An industry study at Tencent found hybrid static-analysis-plus-LLM methods eliminating 94-98% of false positives "while maintaining high recall," which sounds like the suppression problem solved. It is a smaller result than that phrasing suggests - 433 alarms, three bug types, one company's advertising software. + +Read to the end of that paper and it lands where the Monash one does: "recall remains below the enterprise-expected threshold of 90%, indicating that some degree of manual review is still necessary to guarantee absolute safety." Do not run this unattended. + +The filter is a decision, somebody made it, and you are allowed to ask what it was. + +## The question for your next status call + +Not "was the code reviewed." You will get a yes, it will be true, and it will not mean what you wanted it to mean. + +Ask what the review was configured to ignore, and who decided that. + +I have been on the other side of this question, and the honest thing to say is that a good shop will not be offended by it. The ones who bristle are usually the ones who set the filter to default and never looked at it again. If the answers send you looking for the exit, [there is a way to leave without losing the codebase](/blog/switch-dev-shops-safely-transition-guide/). + +## Sources + +- GitHub, [60 million Copilot code reviews and counting](https://github.blog/ai-and-ml/github-copilot/60-million-copilot-code-reviews-and-counting/), 5 March 2026. +- Uber Engineering, [uReview: Scalable, Trustworthy GenAI for Code Review at Uber](https://www.uber.com/us/en/blog/ureview/). +- Yunpeng Xiong and Ting Zhang, [Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering](https://arxiv.org/abs/2601.22952), Proc. ACM Softw. Eng. (ISSTA 2026). +- [Large Language Models Versus Static Code Analysis Tools: A Systematic Benchmark for Vulnerability Detection](https://arxiv.org/abs/2508.04448), arXiv 2508.04448. +- Domenico Cotroneo, Cristina Improta and Pietro Liguori, [Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity](https://arxiv.org/abs/2508.21634), IEEE ISSRE 2025. +- Xueying Du et al., [Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry](https://arxiv.org/abs/2601.18844), arXiv 2601.18844. diff --git a/content/blog/dev-shop-ai-code-review-what-to-ask/suppressed.svg b/content/blog/dev-shop-ai-code-review-what-to-ask/suppressed.svg new file mode 100644 index 000000000..26fc76379 --- /dev/null +++ b/content/blog/dev-shop-ai-code-review-what-to-ask/suppressed.svg @@ -0,0 +1,38 @@ + + What the noise filter removes along with the noise + Filtering cuts static-analysis false positives from over 92 percent to as low as 6.3 percent. The same configuration also suppresses 22.25 percent of real vulnerabilities, and the miss rate exceeds 50 percent for cryptography and policy categories such as weak encryption and weak password hashing. + + + + The filter works. That is the problem. + Xiong & Zhang, ISSTA 2026 - best-performing configuration tested + + + WHAT IT WAS BUILT TO DO + + False alarms: over 92% + → + + 6.3% + noise gone + + + WHAT IT ALSO DOES + + + + 22.25% + of REAL vulnerabilities suppressed + + + + over 50% + missed in crypto & policy categories + + roughly one + in five findings + weak encryption, + weak password hashing + + The categories a founder would care most about are the ones it keeps worst. + diff --git a/content/blog/generate-then-verify-moved-the-work/cover.png b/content/blog/generate-then-verify-moved-the-work/cover.png new file mode 100644 index 000000000..296f61602 Binary files /dev/null and b/content/blog/generate-then-verify-moved-the-work/cover.png differ diff --git a/content/blog/generate-then-verify-moved-the-work/index.md b/content/blog/generate-then-verify-moved-the-work/index.md new file mode 100644 index 000000000..e3dd3fece --- /dev/null +++ b/content/blog/generate-then-verify-moved-the-work/index.md @@ -0,0 +1,93 @@ +--- +title: "\"Generate Then Verify\" Moved Your Work. It Didn't Remove It." +description: "81% of engineering teams report spending more time in code review since adopting AI tools. The advice assumes checking is the cheap half. Measurement says that is where the hours went." +date: 2026-09-02 +draft: false +author: "Paul Keen" +slug: generate-then-verify-moved-the-work +keywords: 'ai code review time, ai generated code verification, code churn ai, developer productivity ai, engineering workflow ai, code review bottleneck' +tags: ['ai', 'engineering', 'code-review', 'productivity'] +categories: ['Engineering'] +cover_image: "cover.png" +cover_image_alt: 'Obsidian-dark cover reading Generate then verify moved the work, with a faceted ruby gem and three chips: 81% report more review time, moved code fell 21% to 3.8%, verification has no automation path' +metatags: + image: cover.png +canonical_url: 'https://jetthoughts.com/blog/generate-then-verify-moved-the-work/' +related_posts: false +--- + +Let the model write the code, then you check it. That is the advice, and four out of five teams who took it now spend more time in review than they did before. + +## Somebody measured where the hours went + +Harness surveyed 700 engineering practitioners and managers at large enterprises across five countries this April, through Sapio Research: + +> 81% say developers spend more time in code review since adopting AI coding tools, with 28% reporting a significant increase of more than 30%. + +The step everyone assumed was the light one is where the time landed. It is vendor-commissioned and self-reported, and enterprise teams are not your four-person startup - discount it as you see fit, and the number still points this direction rather than the other one. + +## Reading is not the cheap half, and never was + +Writing a function means holding one intention in your head and making the machine agree with it. Reading a function means reconstructing somebody else's intention from the residue, without being sure there was one. + +AI output fails this way too. It is plausible. It compiles, it follows a pattern, it looks like something a competent person wrote, and none of those properties tell you whether it is right for your system. + +GitClear has been measuring the residue across 623 million analyzed changes from 2023 to 2026. Moved code - their proxy for the reorganising work that keeps a codebase coherent - was 21% of changed lines in 2022. It fell to 13% in 2023. It is 3.8% so far in 2026. + +![Chart showing moved code as a share of changed lines falling from 21 percent in 2022 to 13 percent in 2023 to 3.8 percent in 2026](refactoring.svg) + +Duplication you can find later. The habit of going back and tidying is harder to restart once a team stops doing it, and generation does not encourage it: the model is very good at adding a thing and has no opinion about whether the thing should have been added next to the four like it. + +## The obvious fix is the one that does not work + +If verification is now the expensive step, automate verification. Everyone arrives here, and the tooling exists, and some of it is genuinely good. + +It is also the one place where handing the work back to a machine fails in a specific and quiet way. I went through the evidence separately in [what to ask when your dev shop says the code was reviewed](/blog/dev-shop-ai-code-review-what-to-ask/), so I will not re-run it here. The short version is that making an automated reviewer quiet enough for developers to tolerate is the same operation as making it miss things, and it misses them worst in the categories you would least like. + +So the expensive half of this workflow has no automation path, which is why "generate then verify" is a description of where the work went rather than a plan for handling it. + +## What we do instead + +None of this is an argument against using the tools. We use them daily. It is an argument against the sentence, and the first thing it changes is what we hand to a model in the first place. + +The question we ask is not "can it write this" but "how fast can I tell whether it did": + +| Generate freely | Type it yourself | +|---|---| +| A migration you can run against a copy of the database | Business logic where "correct" lives in a stakeholder's head | +| A test you can watch fail before you trust it passing | Anything touching money, auth, or permissions | +| A transformation with a known-correct output to diff against | Code whose failure mode is silent and shows up next quarter | + +The left column is not the easy work. It is the work where a wrong answer announces itself quickly and cannot reach far when it does. + +Two habits carry the rest of it. The first is saying the review number out loud during planning, which in practice means estimating like this: + +```text +Task: add rate limiting to the public API + + write it yourself ~2h, and I understand it afterwards + generate + verify ~20m generating + ~90m reading it properly + = 1h50m, and I understand it less + +Decision: generate. The check is fast here - there is a test +that fails when the limit is wrong. Would not generate the +billing reconciliation on the same trade. +``` + +Quoting the twenty minutes and discovering the ninety is how teams end up feeling slower while shipping more. The second habit is keeping the tidying in the same pull request rather than in a someday ticket, since a someday ticket is where that 3.8% went. + +Then somebody owns the merge by name - a person who read it and would be embarrassed by it later, rather than a bot's approval. That rule predates any of this. It just got much easier to skip. + +## What I actually think + +I have watched a lot of code get produced very quickly and then get read very slowly, including in this repository. The generating is not the part that takes the day. + +If your team has adopted AI tooling and velocity has not moved the way you expected, you are not doing it wrong and the tools are not broken. The work moved somewhere nobody was measuring. + +So measure there. Put the review estimate in the ticket next to the build estimate, for two weeks, and see which one you keep getting wrong. + +## Sources + +- Harness, [State of Engineering Excellence 2026](https://prnewswire.com/news-releases/harness-report-reveals-ai-has-outpaced-how-engineering-organizations-measure-developer-productivity-302770521.html) - 700 practitioners and managers across the US, UK, India, France and Germany, fielded by Sapio Research, April 2026. +- GitClear, [The Maintainability Gap: 2026 AI Code Quality Research](https://www.gitclear.com/the_ai_code_quality_maintainability_gap) - 623 million analyzed changes, 2023-2026. diff --git a/content/blog/generate-then-verify-moved-the-work/refactoring.svg b/content/blog/generate-then-verify-moved-the-work/refactoring.svg new file mode 100644 index 000000000..1dde41422 --- /dev/null +++ b/content/blog/generate-then-verify-moved-the-work/refactoring.svg @@ -0,0 +1,29 @@ + + Moved code as a share of changed lines, 2022 to 2026 + GitClear measured moved code - their proxy for reorganising work - at 21 percent of changed lines in 2022, 13 percent in 2023, and 3.8 percent year to date in 2026, across 623 million analyzed changes. + + + + The tidying stopped + Moved code as a share of changed lines - GitClear, 623M analyzed changes + + + + + + + 21% + 2022 + + + + 13% + 2023 + + + + 3.8% + 2026 so far + + Duplication you can find later. The habit is harder to restart. + diff --git a/docs/projects/2510-seo-content-strategy/20-29-strategy/20.09-content-plan-revision-aug-2026.md b/docs/projects/2510-seo-content-strategy/20-29-strategy/20.09-content-plan-revision-aug-2026.md index ee3e6d52d..ed4119a7b 100644 --- a/docs/projects/2510-seo-content-strategy/20-29-strategy/20.09-content-plan-revision-aug-2026.md +++ b/docs/projects/2510-seo-content-strategy/20-29-strategy/20.09-content-plan-revision-aug-2026.md @@ -617,6 +617,541 @@ must frame as decision guide linking the two ranking LangChain guides, not a par comparison page. Stopped at 3 new posts + 1 refresh for the day - a 5th same-day post compounds the cluster-fingerprint risk voice-rules now documents. +### 13i. 13h's ranking is wrong: it orders repair work by bot traffic (2026-08-22, fifth run) + +**Verdict: 13h stays SCHEDULED, but re-ordered. `crewai` moves from first to last.** +The trend candidate is **SCHEDULED for a later date**, not killed. + +> **Corrected 2026-08-22 by Paul.** This section first recorded the trend +> candidate as DO-NOT-WRITE on dedup. That was wrong: a same-week collision is a +> SPACING problem, and the fix is to pre-write and future-date, not to discard a +> good topic. `blog-next` gained a fourth exit (SCHEDULED) and a Stage A0 +> calendar check as a result. The rest of this section stands. + +**The trend candidate, and why it dies.** HN's last three weeks carry a real +cluster - "LLMs reward expertise" (1,416 pts), "AI is removing the middle class +of software engineering?" (1,012), "'Code was never the hard part' is an insult +to all programmers" (946), "Don't paste the AI, please" (1,041), "AI;DR" (1,107). +Eight front-page posts arguing about what AI does to engineering judgment, and +this session had first-hand evidence for it. It is still DO-NOT-WRITE: its best +angle IS `what-senior-developers-catch-that-ai-misses` and its second-best IS +`when-did-a-test-last-fail-on-purpose`, both published 2026-08-22. A third post +this week on one proof-signal reads as one post three times +(`blog-pipeline.md`, cross-post repetition). + +**So schedule it.** The calendar is the constraint, not the topic. Aug 20-22 holds +11 posts; the 12 days before hold none. The distinct angle Reddit supplies and our +three posts do not is **AI code-review tools as a product category**, in +practitioner language - r/ExperiencedDevs "What is your honest take on AI code +review tools?" carries "produced plausible, inaccurate comments about one-third of +the time", "reviews stuff that is not in the PR", and the reframe that lands it: +they are "a more robust version of rule-based linters", not a reviewer. That is a +different post from `what-senior-developers-catch-that-ai-misses`, which argues +about human judgment rather than about a tool category. Verify each quote at the +thread before use. + +**The ranking defect.** 13h ranks the repair queue by 90-day impressions. On this +property that metric is mostly synthetic, so the queue points at bot-magnet +pages. Worked on the row 13h put FIRST: + +| Measure | `crewai-multi-agent-systems-orchestration` | +|---|---| +| Page-dimension impressions (90d) | 6,039 | +| Sum of NAMED query impressions | **67** | +| Clicks | **0** | +| Page-level position | 10.8 (an artifact - Trap C) | + +`named (67) << page (6,039)` is Trap B, which voids the page CTR; Trap C then +voids the position. What survives in the named rows is the synthetic fingerprint +§13a describes - combinatorial permutations of ONE stem, `crewai hierarchical +process manager agent` + {docs, documentation, example, official docs, worker +reviewer documentation}, 15 rows, zero clicks throughout. + +**Corrected order, by human arrivals (clicks, 90d, 2026-05-25 → 2026-08-22, +`sc-domain:jetthoughts.com`, 300 rows, elital subdomain excluded):** + +| Clicks | Impr | Pos | Post | Was | +|---:|---:|---:|---|---| +| 7 | 4,793 | 9.5 | `rails-8-solid-cache-performance-redis-migration` | 3rd | +| 5 | 6,028 | 10.1 | `propshaft-vs-sprockets-rails-8-asset-pipeline-migration` | 2nd | +| 0 | 6,039 | 10.8 | `crewai-multi-agent-systems-orchestration` | **1st** | + +The top three invert. Start with **solid-cache**, not crewai. + +**Membership confirmed, ranking not.** All three are genuinely uncited per the +instrument that owns the definition - `MarketingCopyTest#uncited_posts`, called +directly rather than reimplemented, returns **37** posts (ratchet baseline 38) +and contains all three. A `grep` for external links disagreed - it showed crewai +with 1 link - and the instrument wins. `langchain-python-tutorial` (15 clicks), +`ruby-3-4-yjit` (10) and `rails-virtual-attributes` (7) are NOT in the uncited +set, so they are not candidates however good their traffic. + +**Two traps fired on the way, both worth repeating.** `sort_by: "impressions"` +on the page dimension returned CLICKS order - the pull must be re-sorted +locally. And the first pull came back `row_count 25` against a `row_limit` of 25 +with `has_more: true`: a truncated denominator (Trap A), which is exactly how +`langgraph` (39,413 impressions) stayed invisible until the wider pull. It is +cited (3 external links) and correctly absent from 13h. + +**Owed:** this section is measurements, so `blog-next` requires a `core-reviewer` +re-derivation before the numbers are quoted onward. Not run this session - agent +spawning was not authorised. Treat the figures as author-verified only. + + +### 13j. WRITE (scheduled 2026-08-26): "your shop says the code was reviewed - by what?" + +First run of `blog-next` with Stage A0. The calendar picked the date and the +stream before any topic was considered, which is the point of the change. + +**Stage A0 - calendar.** Last 7 days hold **11 posts** (08-20 x6, 08-21 x2, +08-22 x3) against ~6/month capacity, and the twelve days before them hold none. +Nothing may publish today. Stream balance: Rails Technical took all 11; the +**Founder/ICP-E stream last shipped 2026-08-08** (`migrate-lovable-replit-app-to-rails`), +two weeks quiet. The slot belongs to ICP-E, scheduled for **Wednesday +2026-08-26** - the next weekday leaving >=3 days of spacing. + +**Row** + +| Field | Value | +|---|---| +| Slug | `dev-shop-said-ai-reviewed-it` (working) | +| Stream | Founder / ICP-E | +| Job | **Arrival-purposed.** The search floor does not apply - quoting it here would be the category error §13 warns about | +| Date | 2026-08-26 (future-dated; production skips it until then) | +| Dedup | Nearest is `code-quality-evaluation-non-technical-founders` (2025-10-14, 6,065w), which is general code-quality evaluation written before AI delivery was the norm. It never asks who or what did the reviewing. No collision | + +**Why it is not a fourth AI-verification post.** The three posts of 2026-08-22 +argue about human judgment and about gates that do not gate. This one is about a +**purchase decision a non-technical founder actually faces**, and its evidence is +external rather than our own tooling - which is also what stops it repeating the +proof-signal those three shared. + +**Stage B - sources, both verified at the primary and quoted verbatim:** + +- GitHub, *60 million Copilot code reviews and counting*, 2026-03-05: + "Since our initial launch of Copilot code review (CCR) last April, usage has + grown 10X, now accounting for **more than one in five code reviews on + GitHub**." First-party, and the fact the whole post rests on. +- Cotroneo, Improta & Liguori, *Human-Written vs. AI-Generated Code: A + Large-Scale Study of Defects, Vulnerabilities, and Complexity*, IEEE ISSRE + 2025 (DOI 10.1109/ISSRE66568.2025.00035, arXiv 2508.21634). >500k samples, + Python and Java, ChatGPT / DeepSeek-Coder / Qwen-Coder. Findings, verbatim: + AI code is "generally simpler and more repetitive, yet more prone to unused + constructs and hardcoded debugging"; human code "exhibits greater structural + complexity and a higher concentration of maintainability issues"; and + "AI-generated code also contains **more high-risk security vulnerabilities**." +- r/ExperiencedDevs, *What's your honest take on AI code review tools?* - + practitioner phrasing only, and **every quote must be re-verified at the + thread before use**; they reached this row through a search excerpt, not a + read of the source. + +**REVISED after NotebookLM deep research (63 sources found, 44 cited-only +imported).** The rebuttal sweep contradicted the first thesis, which is what it +is for. Everything in this block is a LEAD from a synthesis report and **not one +figure may be published before it is re-fetched at its primary** - the same rule +that caught two numbers earlier in this row. + +*The case AGAINST "AI review is unreliable" is stronger than expected:* + +- **Uber, uReview** (first-party engineering blog): only **51%** of HUMAN review + comments are judged valid and addressed in the same changeset; uReview reaches + a **65%** address rate, with **75%** of inline comments rated useful. It covers + >90% of ~65,000 weekly changes at a 4-minute median. AI review is not the weak + link in that comparison - human review is. +- **LLMs vs SAST**, systematic benchmark: mean F1 **0.797** (GPT-4.1) against + **0.546** Snyk Code, **0.386** CodeQL, **0.260** SonarQube. + +*And the finding that becomes the post:* + +- **Xiong & Zhang, "Sifting the Noise"** - SWE-agent + Claude Sonnet 4 removed + >92% of OWASP false positives and, doing so, **incorrectly suppressed 22.25% + of REAL vulnerabilities** - rising to **77.17%** for CWE-327 (weak + cryptography) and **84.50%** for CWE-328 (weak hashing). +- A practitioner counter-case calls LLM analysis "random number generators with + opinions" (+-1.8 points across runs on a 10-point scale; missed a blatant SQL + injection on run 3 that it caught on other runs) and reports better results + after replacing it with deterministic static analysis. +- Tencent's LLM4PFA claims ~98% false-positive reduction "while maintaining high + recall", which **directly contradicts** the suppression finding. Say so in the + post rather than picking the convenient one. + +**REVISED THESIS - and it is not "AI review is bad".** The mechanism that makes +an AI reviewer trustworthy to developers is filtering: suppress the nits, or +nobody reads it. That same filter is what drops real findings, and it drops them +hardest in the least visible category. So the reassurance a founder is offered - +*"it was reviewed and came back clean"* - is partly manufactured by the tuning +that made the tool usable. **The question is not whether it was reviewed. It is +what the filter was allowed to throw away.** + +This is a better post than the row's first version and it survives its own +counter-evidence, which the first version had not been tested against. + +**Outline change:** section 2 now states the case FOR AI review first, using the +Uber numbers, because a reader who thinks this is anti-AI stops reading. Section +3 turns on the suppression figures. Section 5 (the honest limit) must carry the +Tencent contradiction. + +**Superseded below - kept because the row's earlier reasoning is still the dedup +and calendar argument:** + +**The thesis the sources originally supported** - note it is not "AI code is worse": +AI-written and human-written code **fail differently**. AI code is simpler but +carries more high-risk security vulnerabilities; human code is more tangled but +its problems are maintainability. That asymmetry is the reason "we reviewed it" +stopped being a complete answer, because an AI reviewer is weakest in the same +places AI code is. + +**Stage C - outline** + +1. Hook: the founder's question is not "was it reviewed" but "by what". >1 in 5 + reviews on GitHub are now Copilot's - the odds it was a human are no longer + obvious. *(reader does differently: asks a second question)* +2. What the peer-reviewed comparison found, stated fairly, both directions. + *(stops the reader treating this as anti-AI)* +3. Why the asymmetry matters for a buyer: the failure mode that survives AI + review is the one AI code produces most. +4. Four things a non-technical founder can verify without reading code - each + an artifact they can ask for and check the existence of, not a judgement call. +5. What this does NOT tell you. The honest limit; the ISSRE study is Python and + Java, not every stack. +6. Close: the question to put in the next status call, in one sentence. + +Internal links: `code-quality-evaluation-non-technical-founders` (the general +version of the question), `switch-dev-shops-safely-transition-guide` (what to do +if the answers are bad). CTA: the rescue context call, per canon. + +**Cut test applied:** an earlier section 4 listed six checks; four survive +because two were restatements of "ask for the test suite". + +**Owed before drafting:** Stage C requires a `core-reviewer` gate (author != +verifier) and Stage A requires re-derivation of the calendar figures. Neither +ran - agent spawning is not authorised this session. Both are gates on the +DRAFT, not on this row, so `blog-write` must not skip them. + + +### 13k. WRITE (scheduled 2026-09-02): "generate then verify" moved the work, it did not remove it + +Paul's row, supplied verbatim: *"AI should generate code, but verify - NOPE!"* +A1 treats his raw material as the highest-value source, so this is taken rather +than audited for whether it deserves a slot. What follows is the audit of HOW to +write it without it becoming the fifth post in a week on one theme. + +**Stage A0 - calendar.** Aug 20-22 hold 11 posts; 2026-08-26 now holds the +scheduled ICP-E post (§13j). Theme recency is the binding constraint here, not +spacing: FOUR posts in the last six days already touch AI verification. Date is +therefore **Wednesday 2026-09-02**, a week clear of §13j, and the evidence is +durable rather than news-pegged so nothing decays by waiting. + +**The dedup problem, stated honestly.** This is the fifth candidate in the +family. It survives because the four siblings all argue about the TOOLING or the +REVIEWER, and this one argues the WORKFLOW ADVICE is wrong: + +| Post | Argues | +|---|---| +| `what-senior-developers-catch-that-ai-misses` (08-22) | what human judgment catches | +| `when-did-a-test-last-fail-on-purpose` (08-22) | gates that report green over nothing | +| `how-to-audit-content-you-didnt-write` (08-22) | auditing text you did not write | +| `dev-shop-ai-code-review-what-to-ask` (08-26) | what an AI reviewer suppresses | +| **13k** | **"generate then verify" is not a division of labour, it is a transfer of work to the expensive half** | + +⚠️ **METR IS SPENT AND IS BANNED FROM THIS POST.** It appears **5 times** in +`what-senior-developers-catch-that-ai-misses`. Reaching for it again is the +cross-post repetition that made an ICP reviewer say she had "read one post three +times". This row's evidence deliberately shares nothing with its siblings. + +**Stage B - sources. The load-bearing one is verified at the primary:** + +- Harness, *State of Engineering Excellence 2026* (via its press release): + "**81% say developers spend more time in code review since adopting AI coding + tools, with 28% reporting a significant increase of more than 30%**." 700 + software engineering practitioners and managers at large enterprises across + US/UK/India/France/Germany, fielded by Sapio Research, April 2026. VERIFIED. +- GitClear, 211M lines of changed code: churn ~3.3% pre-2023 to ~7.1% in 2025; + duplication up sharply; refactoring down from roughly 25% of changed lines in + 2021 to under 10%. **UNVERIFIED at primary** - reached this row via secondary + write-ups. Fetch `gitclear.com` research pages before any figure is written. +- DORA 2025, *State of AI-assisted Software Development*, ~5,000 respondents - + AI as "amplifier" of existing strengths. UNVERIFIED at primary. +- A widely-repeated "63% of developers spent more time debugging AI code than + writing it themselves" is attributed to Stack Overflow 2026 / Octoverse / + SlashData by a vendor blog. **DO NOT USE** until located in one of those + reports directly; the attribution chain is exactly the shape that produced a + wrong sentence on 2026-08-22. + +**Thesis.** The advice sounds like a division of labour and is not. Generating +became nearly free; verifying did not, and the measurement says that is where +the time went. The sting is that verification is the half you cannot hand back +to a machine - §13j's own sources show an AI reviewer suppressing 22.25% of real +findings - so "generate then verify" is a workflow whose expensive half has no +automation path. That is why Paul's answer is NOPE rather than "yes, but". + +**Stage C - outline** + +1. The advice, quoted as everyone states it, and the one word doing the damage: + *then*. It implies sequence and cheapness. +2. What actually happened to review time - Harness, with the sample stated so a + reader can discount it. +3. Why reading unfamiliar code is not the cheap half. Churn and the refactoring + collapse as the visible residue. +4. The trap: the obvious fix is to automate verification, and that is precisely + where it fails (link §13j rather than re-arguing it). +5. What actually works instead - written as things a team does differently on + Monday, not principles. +6. Close: what Paul does instead, in one paragraph, first person. + +Internal links: `dev-shop-ai-code-review-what-to-ask` (the automation trap), +`what-senior-developers-catch-that-ai-misses` (judgment) - ONE reference each, +no re-argument. CTA: rescue context call. + +**Owed:** Stage C outline gate (`core-reviewer`) and Stage A re-derivation did +not run; agent spawning is unavailable this session. Stated, and `blog-write` +invoked anyway per the skill - a stated gap beats a stalled pipeline. + + +### 13l. WRITE (scheduled 2026-09-09): the setup is real, the success rate is 13-59% + +Paul's row, verbatim: *"setup highly-autonomus team of agents is real already +today, but how many got success?"* Taken as his ordering preference. The +question turns out to have a measured answer, which is the whole reason it is +worth a slot. + +**Stage A0.** Aug 20-22 hold 11 posts; 08-26 and 09-02 are scheduled. Next slot +at >=3 days clear is **Wednesday 2026-09-09**, a week after 09-02, keeping this +family on a weekly cadence rather than a cluster. + +⚠️ **Flagging the pattern rather than silently continuing it: this is the SIXTH +consecutive AI post.** Rails Technical has 114 posts and the blog's recent +output is entirely AI commentary. That is a positioning drift worth a decision, +not a dedup problem, so it is Paul's call and the row is taken. If the answer is +"vary the mix", the cheapest correction is `20.09 §13h` - the uncited technical +guides, which are Rails and Laravel and already rank. + +**Dedup - the near-miss is real and it survives.** `multi-agent-llm-rails-rubyllm` +(2026-08-20) shares the words and not the subject: it is a HOW-TO for a +multi-agent LLM pipeline as a PRODUCT FEATURE in Rails (eight RubyLLM agents, a +21-line base class), and it already owns the "when one agent is enough" caveat. +This row is about agent teams doing the ENGINEERING WORK, and its content is +industry outcome data rather than our implementation. Different subject, same +vocabulary. The post must not re-argue the Rails architecture, and must link +that post once for readers who want the implementation. + +**Stage B - the answer to Paul's question, verified.** + +Cemri et al., *Why Do Multi-Agent LLM Systems Fail?* (arXiv 2503.13657; Zaharia, +Gonzalez, Stoica among the authors; NeurIPS 2025 D&B track). Abstract, verbatim: +"**Despite enthusiasm for Multi-Agent LLM Systems (MAS), their performance gains +on popular benchmarks are often minimal.**" 7 frameworks, 14 failure modes in 3 +categories (system design, inter-agent misalignment, task verification), Cohen's +kappa 0.88. + +Figure 1 of the PDF gives the success rates - **this is the number Paul asked +for**: + +| Framework (benchmark) | Success | +|---|---:| +| AG2 (OlympiadBench) | 59.0% | +| MetaGPT (ProgramDev) | 40.0% | +| Magentic-One (GAIA) | 38.0% | +| ChatDev (ProgramDev) | 33.3% | +| HyperAgent (SWE-Bench Lite) | 25.3% | +| AppWorld (Test-C) | 13.3% | + +⚠️ The figure's own caption must travel with these: "Performances are measured on +different benchmarks, therefore they are not directly comparable." Quoting the +spread as a league table would be the error. + +⚠️ **TRACE COUNT IS AMBIGUOUS - do not state one.** The abstract page fetch says +"1,600+ annotated traces ... with rigorous analysis of 150 traces"; the paper +body says "200 MAS execution traces". Until one is confirmed in the PDF, write +around it. + +**The counterweight, verified verbatim at Anthropic:** their multi-agent research +system "outperformed single-agent Claude Opus 4 by 90.2% on our internal +research eval." So multi-agent is not a myth. The bill: "Agents typically use +about 4x more tokens than chat interactions, and multi-agent systems use about +15x more tokens than chats," and "token usage by itself explains 80% of the +variance." + +**And the sentence that decides the post**, also Anthropic: "most coding tasks +involve fewer truly parallelizable tasks than research, and LLM agents are not +yet great at coordinating and delegating to other agents in real time." + +**The arc, which is the actual story.** Cognition published *Don't Build +Multi-Agents* (Walden Yan, June 2025) and then *Multi-Agents: What's Actually +Working* (2026-04-22), which concedes the narrower class that does work: +"multiple agents contribute intelligence to a task while writes stay +single-threaded." Anthropic's read-only-subagent pattern is the same shape in +different words. The loudest sceptic and the loudest proponent converged. + +**First-hand corroboration we can actually show:** this repo's own CLAUDE.md +runs exactly that shape - author != verifier, reviewers contribute findings, one +author writes. Sanitised: the protocol and what it caught, not prompts or model +IDs. + +**Thesis.** The honest answer to "how many got success" is that measured success +runs 13-59% and the successful shape is narrow and now agreed by both camps: +many agents may THINK, one agent WRITES. Anything with parallel writers is the +part that keeps failing. + +**Stage C - outline** + +1. Paul's question, answered in the first 100 words with the range. No throat-clearing. +2. The measured spread, with the not-comparable caveat stated as part of the number. +3. Why the enthusiasm was not stupid - the Anthropic 90.2%, and its 15x bill. +4. The convergence: Cognition's own reversal, quoted from both posts, to the + single-threaded-writes rule. +5. What that means for a team adopting this in 2026 - concrete shapes, and one + sentence of ours as corroboration, linking the Aug 20 post for implementation. +6. Close on what Paul actually thinks about the success rate. + +**Owed:** Stage C gate and Stage A re-derivation not run - no agent spawning. +Stated, and `blog-write` invoked anyway per the skill. + + +### 13m. WRITE (scheduled 2026-09-16): delegate the goal, and the origin story is contested + +Paul's row, verbatim: *"you should not ask agents task, but delegate the goal +(basically this is how should you work with teams overall)."* His parenthesis is +the important half - he is describing management, using agents as the current +example, which makes this a delegation post rather than a seventh AI post. + +**Stage A0.** 08-26, 09-02 and 09-09 are scheduled. Next slot at >=3 days clear +is **Wednesday 2026-09-16**. The AI-run flag raised in §13l stands but applies +less here: this row's spine is management doctrine and its audience is anyone +running a team. + +**Dedup.** §13l (09-09) is TOPOLOGY - which agent may write. This is BRIEFING - +what you put in the instruction. A team can have §13l's shape exactly right and +still fail by handing out tasks. Related, not overlapping. One link, no +re-argument. + +**Stage B - the doctrine, and the reason this post is not the usual one.** + +The thesis exists in doctrine, stated cleanly. ADP 6-0 defines **mission +orders** as "directives that emphasize results rather than specifying exactly +how to achieve them," and **disciplined initiative** as subordinates acting +within the commander's intent "when orders no longer fit the situation." +Commander's intent, per ADRP 5-0 as quoted in *Military Review*, "succinctly +describes what constitutes success for the operation ... the operation's +purpose, key tasks, and the conditions that define the end state." + +⚠️ **THE REBUTTAL IS THE POST'S DIFFERENTIATOR - do not write the LinkedIn +version of this topic.** Every management article on commander's intent tells a +tidy Prussian origin story. The historians disagree, in print: + +- Herrera, *History, Mission Command, and the Auftragstaktik Infatuation* + (Military Review): ADP 6-0's claim of German roots is "unfounded"; "it is long + past time to shed the infatuation with the German military experience and + fatuous lineage of mission command." +- Australian Army Journal on Sigg's research: evidence of Auftragstaktik was + "consistently polarised"; German commanders showed "a remarkable insistence + upon autonomy and freedom of action, even when this departed from, or even + contradicted, the intent of the orders issued." Conclusion: "we may discard + the idea that there was ever a golden age of German Auftragstaktik." + +So the honest shape is: the PRINCIPLE is sound and codified, the ORIGIN STORY +usually told alongside it is contested, and the people who studied it closest +found a constant tension rather than a recipe. That is a better post and it is +one nobody else in this niche is writing. + +⚠️ All doctrine quotes above reached this row through search excerpts. **Fetch +ADP 6-0 and the Military Review pieces at their primaries before writing**, per +the rule that has now corrected a figure in three consecutive rows. + +**First-hand, and it is unusually direct.** Our own instruction layer already +encodes the thesis and can be quoted from the repo: + +- `~/.claude/CLAUDE.md`: "Brief others with evidence, never with your + conclusions - a panel handed your inference will return it wearing + independent-sounding confidence." +- Same file: "Agent and skill definitions carry description and judgment, not + hardcoded scripts." +- `blog-write` skill: "Brief each with goal and artifact, never with your + conclusions. Require each to name something it would cut." + +That last clause is the operational trick worth giving away: a reviewer told +what you concluded confirms it. A reviewer given the artifact and required to +name a cut produces findings. Sanitise to shapes and rules - no prompts, no +model IDs. + +**Thesis.** Delegating a task buys you compliance and caps the result at your +own understanding of the problem. Delegating a goal - purpose, what success +looks like, the constraints that must hold - buys you judgment, and is the only +form of delegation that survives the moment the plan stops fitting. It works the +same way whether the subordinate is a person or an agent, which is why the +doctrine predates the technology by a century. + +**Stage C - outline** + +1. Open on the failure, not the principle: a perfectly-executed task list that + produced the wrong outcome. One paragraph, concrete. +2. What the doctrine actually says, quoted - mission orders, commander's intent, + disciplined initiative. Short, because the words are already good. +3. The part everyone leaves out: the origin story is contested, and the honest + version is more useful than the myth. +4. What changes in the instruction itself - a before/after table of the same + delegation written as task and as goal. +5. Where it fails: goals need constraints, or you get creative compliance. + Name the failure mode rather than pretending there is not one. +6. Close on the reviewer trick and why the same rule applies to people. + +Internal link: §13l's post (topology) ONCE. CTA: rescue context call. + +**Owed:** Stage C gate and Stage A re-derivation not run - no agent spawning. +Stated, and `blog-write` invoked anyway. + + +### 13n. BANKED (not scheduled): "release day is the worst day" - Paul's client story + +Paul, 2026-08-23, verbatim so nobody paraphrases the sharp bits away: + +> for one of our clients we saw how each release were like the most stressful +> day, and when I said this is no acceptable they thought that I'm from other +> planet, for them bugs is expected and chaos and stress on release is not +> optional. And my idea with multiple releases per day is like romantic fantasy. +> And based on so many books how to structure the discovery and delivery we +> still have what we have: team built based on the emotional qualities over +> logical. + +**Why this is banked rather than queued.** It is the strongest raw material in +the bank and it needs a decision only Paul can make before it can be written. + +**Why it matters more than the four posts scheduled around it.** The ICP cold +read of 2026-08-23 ended on exactly this gap: "after four posts I know a great +deal about how AI writes and reviews code, and there is still no story about a +company like mine in any of them. No founder, no invoice, no dollar figure, no +'here is what this cost someone.' Three of four are built on arXiv papers." This +row is the answer to that. It is a named human reaction, not a benchmark. + +**BLOCKED ON PAUL - the only thing standing between this and a slot:** +how far the client can be described. Claims-canon bans invented client work +outright, and the anecdote's whole power is that it happened. Options, cheapest +first: (a) unnamed but real - "a client", industry and team size only; (b) named +with permission; (c) composite - **not acceptable**, that is the fabricated case +study the 2026-08-22 purge removed from fourteen posts. + +**The angle, and it is not the obvious one.** The obvious post is "you can +release daily, here is how", which is a solved genre and Arkency owns it. Paul's +last sentence is the actual thesis and it is much less comfortable: teams are +assembled on emotional qualities rather than logical ones, so the delivery books +do not take. The interesting question is not why release day is stressful. It is +why an industry with two decades of published practice keeps reproducing the +same organisation, and what a founder can see from outside it. + +**Reactions worth keeping verbatim if the client permission allows:** "they +thought I'm from another planet" and "romantic fantasy" are the post. Both are +somebody else's words about a normal engineering practice, which is the whole +argument in two phrases. + +**Stack/stream:** Founder-ICP-E, arrival-purposed. Would break the AI run - this +is the seventh consecutive AI post's antidote, and §13l already flagged that +drift as needing a decision. + +**Do NOT schedule until the client question is answered.** A date on this row +would create pressure to write it with a composite client, which is the failure +mode the ratchet exists to catch. + + ## Changelog | Date | Change | @@ -1044,6 +1579,12 @@ version-named posts rather than waiting to notice. |---|---| | 2026-08-22 | §13h added: 39 of 93 substantial non-dev.to posts cite nothing external - unverifiable by construction, now ratcheted at 38. First pass on the top one (`laravel-11-migration-guide`, 20,226 impr) found no fabrication but a STALE PREMISE: context7 returned Laravel's support table showing 11.x unpatched since 2026-03-12, so we were sending readers to migrate onto a dead release. Notice added; 10→12 rewrite scheduled. Generalisation: a version-named post has an expiry its author never wrote down. | | 2026-08-22 | §13g added: N10 (Paul's "structure a team with AI harnesses") resolves to UPGRADE `claude-code-xp-team-workflow`, not a new post - the 2026-05-04 post already owns the roles/cadences/arbitration content. Upgrade carries what it predates: three exits vs two, HOLD as terminal success, author≠verifier as a different agent TYPE, and four gate-caught errors. Arrival-purposed; verdict rests on a file read, not a measurement chain, and was NOT routed through the Stage A reviewer gate. Approved by Paul. | +| 2026-08-23 | §13n BANKED (no date): Paul's client story - release day as the most stressful day, "they thought I'm from other planet", daily releases as "romantic fantasy", and the real thesis: teams built on emotional qualities over logical ones, which is why the delivery books do not take. This is the first-party human story the 2026-08-23 ICP cold read said all four scheduled posts lacked. BLOCKED on how far the client may be described - composite is not an option. Not dated on purpose: a date would create pressure to invent the client. | +| 2026-08-23 | §13m added: Paul's row ("delegate the goal, not the task - how you should work with teams overall"). WRITE scheduled 2026-09-16. Management post using agents as the example, so it widens the mix rather than extending the AI run. Doctrine spine: ADP 6-0 mission orders "emphasize results rather than specifying exactly how", commander's intent, disciplined initiative. DIFFERENTIATOR IS THE REBUTTAL: Herrera in Military Review calls ADP 6-0's German lineage "unfounded" and Sigg's research found no "golden age of Auftragstaktik" - so the principle is sound and the usual origin story is contested. All doctrine quotes UNVERIFIED at primary, fetch before writing. First-hand: our own CLAUDE.md already says brief with evidence never conclusions. Dedup vs §13l holds: topology vs briefing. | +| 2026-08-23 | §13l added: Paul's row ("agent teams are real, but how many got success?"). WRITE scheduled 2026-09-09. It has a measured answer: Cemri et al. Figure 1, success 13.3-59.0% across six MAS frameworks, with the "not directly comparable" caveat mandatory. Counterweight verified at Anthropic: multi-agent beat single-agent Opus 4 by 90.2% but costs ~15x tokens, and they state coding has "fewer truly parallelizable tasks". Story is the convergence - Cognition's own reversal to "writes stay single-threaded". Dedup vs multi-agent-llm-rails-rubyllm holds (product feature vs engineering workflow). FLAGGED: sixth consecutive AI post, positioning drift, Paul's call. | +| 2026-08-23 | §13k added: Paul's row ("AI should generate code, but verify - NOPE!"). WRITE scheduled 2026-09-02 - theme recency, not spacing, is the constraint: 4 of the last 6 days' posts touch AI verification. Survives dedup by attacking the WORKFLOW ADVICE rather than the tooling. METR BANNED (5 uses in a sibling). Harness 2026 verified at primary: 81% report more review time since adopting AI tools, 28% report >30% more. GitClear/DORA/the "63% debugging" stat flagged UNVERIFIED. | +| 2026-08-22 | §13j added: first Stage A0 run. Calendar (11 posts in 3 days, ICP-E quiet since 08-08) picked stream and date BEFORE topic. WRITE scheduled 2026-08-26, arrival-purposed: "your shop says it was reviewed - by what?" Two primaries verified verbatim (GitHub CCR >1-in-5; ISSRE 2025 >500k samples: AI code simpler but more high-risk security vulns, human code more maintainability debt). Outline gate + Stage A re-derivation OWED. | +| 2026-08-22 | §13i added: 13h re-ordered - it ranked repair work by impressions, which on this property is mostly synthetic. crewai moves 1st → last (named 67 vs page 6,039, zero clicks, combinatorial stem). Corrected order by clicks: solid-cache → propshaft → crewai. Uncited membership confirmed at 37 via `MarketingCopyTest#uncited_posts` (a grep disagreed; the instrument won). HN AI-judgment cluster DO-NOT-WRITE on dedup with two posts published the same day. | | 2026-08-22 | §13f added: N7 (Paul's "how AI helps developers") DO-NOT-WRITE on 61-slug saturation + 180-query zero-click + wrong reader. N8 `vertical ai agents` FLAG-not-write: first row to clear the demand floor (0.5-3.5 clicks/day) but decaying 2x, SERP held by Salesforce/IBM/Google, asset is a 3-min dev.to import. N9 SCHEDULED: first-party benchmark of AI code-search MCP tools. Trap A generalised to filters. | | 2026-08-22 | §13e added: §13d's replacement direction retracted (no neighbourhood — one query, already won, ceiling +0.13 clicks/day). Data-migration family examined and FLAGGED not scheduled (synthetic fingerprint unexcluded). Demand floor introduced: whole blog = 0.69 non-brand clicks/day. Queue verdict HOLD. | | 2026-08-22 | §13d added: N2 falsified on a live re-pull (52x = 1.4x clicks × 37.2x impressions; foil is 1.4% named). N3's addressable volume corrected 6,895 → ~550 (synthetic query family). Replacement topic: Rails dependency/setup. Two additions to §13a: position is an artifact when CTR is, and query-vs-query is the legitimate comparison. | diff --git a/docs/workflows/blog-pipeline.md b/docs/workflows/blog-pipeline.md index 2999f37c8..009eecc13 100644 --- a/docs/workflows/blog-pipeline.md +++ b/docs/workflows/blog-pipeline.md @@ -210,8 +210,23 @@ Which instrument, by claim type: the PRIMARY source. Press coverage of a study is not the study; go to the publisher's own page and take the caveats along with the number. - **A body of sources you need to interrogate** → NotebookLM (`notebook_create` - → `source_add` → `notebook_query`; `research_start` → `research_status` → - **`research_import`** to find them first). + → `source_add` → `notebook_query`). +- **A WIDE sweep of a topic you do not yet know the shape of** → NotebookLM deep + research: `research_start(mode="deep")` → `research_status` → + **`research_import(cited_only=True)`** → `notebook_query`. ~5 min, ~40 sources, + server-side. **FIRE IT FIRST and let it run while you do the web searches** - + in sequence it costs five minutes, in parallel it costs nothing. Without the + import step the sources never enter the notebook and `notebook_query` answers + from an empty notebook. Gate on `server_info`: `stale` = ask for `nlm login`, + `unverified` = the CHECK failed, not the credentials, so try the call anyway. + **The report is a lead, not a citation** - open the primary and quote there. +- **A diagram or mind map from that same notebook** → `studio_create(artifact_type= + "infographic" | "mind_map" | "data_table" | "slide_deck")`, poll `studio_status`, + then `download_artifact`. Use `mind_map` before outlining to see how sources + actually cluster, and `data_table` to list every number the sources state so the + claim pass has a checklist. A generated infographic is a STRUCTURE draft only: + it does not use the house palette, and its numbers are generated, so they carry + the same verification burden as prose while looking like measurements. - **What practitioners actually argue** → the HN Algolia API, and read the thread, not the headline.