New post: what senior devs catch that AI misses - #592
Merged
Conversation
Stage A found a real cluster rather than a single hot story: eight HN front-page posts in three weeks arguing the same question, with comment counts (1,335 / 937 / 869 / 590 / 573) that mean argument rather than consensus. Sean Goedecke's "LLMs reward expertise" and Senko Rašić's "'Code was never the hard part' is an insult to all programmers" are the two poles. Dedup passed with a real gap. The three closest posts (ai-code-ownership-accountability, vibe-coding-crisis-ai-code-debt, vibe-coding-disposable-by-design) are all about the CODE's properties - ownership, maintainability, disposability. This cluster is about the PERSON's value. claude-code-xp-team-workflow covers the four-eyes rule but for practitioners running agents, not for a founder deciding who to pay. Job: ARRIVAL-purposed. The search floor does not apply and quoting it would be the category error §13e warns about. First-hand evidence, and it is unusually good because it is ours and it is embarrassing: earlier today an agent removed an unsourced performance number from an old Rails post and replaced it with a confidently wrong mechanism - Propshaft "stops being a build step that scales with your asset count". It does scale; every asset is still walked, fingerprinted and copied. A fabricated number was traded for a fabricated mechanism, inside a change whose whole purpose was removing unreliable claims. That is Goedecke's "steering" running backwards, and it is the argument for expertise made from the losing side. Gates: - Cadence quotas measured per H2 with a script, not eyeballed. FAILED first pass in three sections - one paragraph over the 3-sentence cap, two sections with no single-sentence paragraph, one with no sentence over 22 words. All corrected, all six sections now pass. - The same check caught a ZERO-TOLERANCE banned pattern I had written: "Not a better model. Not a longer prompt. A different one." That is rule-of- three negative parallelism. Rewritten. - Banned words: zero. Em dashes: zero. Mannerism nouns: 3 in body prose against a cap of 5 (a naive grep says 6 - two are frontmatter keywords and one is inside a URL slug). - Opener variety: 2 of 43 paragraphs start "The"/"We" = 4.7%, against caps of 20% and 15%. - Hook shape is a timestamped real incident, NOT the banned fictional-founder formula. - Every external claim FETCHED, not recalled: both HN posts pulled and quoted verbatim, and the Propshaft behaviour verified against its own README rather than against my belief about it - which is precisely the check the post is about. - `bin/hugo-build` green, page renders 200, marketing_copy_test 4/10/0. 1,120 words. Reference cadence: patio11 (incident post). NEEDS PAUL - publishing outward. The post says plainly that our own agent introduced a wrong claim into our own blog while cleaning it up. I judge that it builds credibility rather than costing it, and it is the evidence the whole piece rests on. But it is a disclosure about us, which is on the stop list, so the PR is open rather than merged. Genericising it is a one-paragraph edit if you would rather not say it. NO INDEPENDENT VERIFIER RAN - agent spawning is unavailable this session, so the critic panel and cold-eyes gate were run inline against their written criteria by the author. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Two candidate markers were built and MEASURED and REJECTED before writing any of
this, because the honest finding is that correctness is not greppable:
- anonymous-company pattern: 10 hits, mostly legitimate ("we measured a mid-size
Rails API app (60 controllers, 45 models)")
- absolute-claim pattern: 185 hits, almost entirely ordinary prose ("remote work
is no longer a perk", "eliminates Rails boot time")
Either would cry wolf until people stopped reading it. A ratchet counts invented
SHAPES; a wrong mechanism is a well-formed sentence with the right vocabulary
pointing roughly the right way. Shipping a gate that claimed to catch it would
itself have been a wrong claim about how something works, which is the exact
defect being fixed.
What IS preventable is the link in the chain where the claim never got checked.
**A mechanism is a claim too, and the more dangerous kind.** A number announces
itself as something to verify; an explanation of how a thing works reads as
reasoning and gets waved through. That asymmetry is why wrong mechanisms outlive
wrong numbers.
**A replacement is a new claim.** Deleting an unsourced claim and writing
something in its place is authorship, not cleanup, and carries the same sourcing
burden. De-fabrication has this failure mode all of its own: the hole gets
filled with less scrutiny than the claim that was in it.
The worked example is mine, same day, same sentence, both directions. An agent
removed an unsourced Propshaft timing figure and asserted precompilation "stops
being a build step that scales with your asset count" - false. Hours later the
same claim went into a blog post and came out RIGHT, because that framing
triggered "go read the README". Same fact, same author, same day: unsourced when
it felt like fixing, sourced when it felt like writing.
Landed as:
- `claims-canon.md` - three rules, plus an explicit statement that no ratchet
here claims to catch correctness, with the measurements that settled it.
- `fabrication-ratchet.md` - a "what this gate does NOT catch" section, so a
green run is never read as "the post is accurate".
- `blog-pipeline.md` STEP 4f (BLOCKING) - list every sentence about external
behaviour and name the URL you opened in THIS session; soften or cut whatever
has none.
- `blog-write` and `blog-next` skills - the same rule where the work happens.
Applied it immediately to the post already in review, and it fired: "Propshaft
*is* faster" was asserted from belief and never checked. It is correct - the
README says "a dramatically simpler and faster asset pipeline compared to
previous options, like Sprockets" - but I had not looked. Both Propshaft claims
in that post are now sourced in its Sources section, including the distinction
the original error missed: faster per asset, still O(assets) overall.
Gates: `bin/hugo-build` green. Cadence quotas still pass in all six sections.
`okf_validate --strict` conformant, zero errors.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg
STEP 4f said "a URL you opened", which is too vague to act on. It now routes by claim type: context7 for how a library behaves (it reads the project's own docs and returns real class and method names, so it settles mechanism questions a README summary leaves open), WebSearch then WebFetch the PRIMARY source for a study or statistic, NotebookLM for a body of sources, HN Algolia for what practitioners actually argue. **Press coverage of a study is not the study.** Go to the publisher's page and take the caveats along with the number. **Search for the rebuttal, not the confirmation.** Searching for support returns support - that is what it is for. Query the criticism and the failed replications of the thing you are about to assert. Ran the whole loop against the post already in review rather than describing it: - context7 on `/rails/propshaft` independently confirmed the corrected claim and went past the README, naming `Propshaft::Processor` as handling "copying files... generating digested filenames" and showing `load_path.assets.each`. Two independent sources, same conclusion. - STEP 4f fired on my own draft. "Propshaft *is* faster" was asserted from belief and never checked. It is true - the README says "a dramatically simpler and faster asset pipeline compared to previous options, like Sprockets" - but I had not looked, which is the whole point. - The rebuttal search returned something better than a confirmation. METR's randomised trial: 16 experienced open-source developers, 246 real issues, in repositories they had worked in for years, ran 19% SLOWER with AI. They had predicted 24% faster. Afterwards, having lived through the slowdown, they still estimated 20% faster. That ~40-point gap between measurement and belief, among experts on their own code, is the post's thesis measured rather than anecdoted - so it went in, with METR's own limits stated alongside it (early-2025 Claude 3.5/3.7 via Cursor Pro, mature codebases, and their explicit refusal to generalise). Omitting those would have been the same selective-citation defect the post argues against. Fetched from metr.org, not from the eight press write-ups that agreed with each other. Post: 1,120 -> 1,352 words. All seven sections still pass the cadence quotas, mannerism nouns still 3 of 5, banned words and em dashes still zero. Gates: `bin/hugo-build` green, page renders 200, marketing_copy_test 4/10/0, `okf_validate --strict` conformant with zero errors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg
…uide **The gate.** Correctness is not greppable - established by measuring and rejecting two candidate markers (10 and 185 mostly-legitimate hits). But CHECKABILITY is mechanical, and uncheckable is where false hides. 39 of 93 substantial non-dev.to posts (>=400 words) cite nothing external. They cannot be verified by a reader, and - the part that actually bites - could not have been verified by whoever wrote them. Among them: a 7,794-word APM comparison linking to no APM tool's docs. `test_substantial_blog_posts_do_not_regress_on_uncitedness` ratchets that at 38 (39 minus the fix below). Internal links do not count: the question is whether a claim can be checked against something we do not control. 400 words is the floor because below it a post is a note, and demanding citations of a note is the noise this file's header warns about. Proven exact - dropped to 37, failed with "Expected 38 to be <= 37", restored. **The find, and it was not a fabrication.** `laravel-11-migration-guide` is the top uncited page at 20,226 impressions. context7 on `/laravel/docs` returned Laravel's own support table: 11.x active support ended 2025-09-03, security support ended 2026-03-12. We have been sending readers to migrate ONTO a release that has been unpatched for five months. Laravel 12 has the same PHP 8.2 minimum and is supported into 2027. Nothing in that post was invented. It went stale - which no shape-based gate can see, and which is worse than a fabricated case study, because stale advice is actively harmful rather than merely unsupported. The PHP 8.2 claim itself checked out. Fixed now: a dated notice at the top citing the support table and naming 12 as the real target, keeping the structural content since 12 inherits 11's skeleton. **Scheduled, not done** (§13h): the 10 -> 12 rewrite, and the verification pass over the remaining uncited guides, ranked by impressions with the per-post method. Also recorded the generalisation worth more than the instance - a version-named post carries an expiry date its author never wrote down, and the support table is a two-minute context7 query. **Fractional-CTO posts: answered with data, staying parked.** Paul asked whether fixing them profits. The three carrying the 9 fabrication hits do not appear in the 500-row impressions pull at all (<=1 impression each). The fractional posts that DO earn traffic are different files with no fabrications. Fixing them buys nothing. Gates: marketing_copy_test 5 runs / 13 assertions / 0 failures. `bin/hugo-build` green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg
Paul asked whether I could clear my own blocker with research instead of waiting.
Doing that surfaced a better defect than the one I was researching.
The Laravel notice I wrote 20 minutes ago was itself wrong. It said: do not land
on 11, go to 12, same PHP 8.2 floor. I had queried context7 for Laravel 11 and 12
and stopped there. laravel.com's actual support table has four rows:
10 | PHP 8.1-8.3 | bug fixes to 2024-08-06 | security to 2025-02-04 (EOL)
11 | PHP 8.2-8.4 | bug fixes to 2025-09-03 | security to 2026-03-12 (EOL)
12 | PHP 8.2-8.5 | bug fixes to 2026-08-13 | security to 2027-02-24
13 | PHP 8.3-8.5 | bug fixes to Q3 2027 | security to 2028-03-17
Laravel 12 left bug-fix support NINE DAYS AGO. Laravel 13 has been current since
March and needs PHP 8.3, so "8.2 is enough" would have misled anyone taking the
sensible path. The check ran, felt complete, and stopped two rows short - after
I had written the rule about verifying claims, in the same session.
That is a harder failure than skipping the check, because it produces the same
feeling of having been careful. It went into the post as its own section
("Partial checking feels exactly like checking"), and it is the stronger of the
two examples, because it happened AFTER the rule existed.
Notice rewritten with the full table, PHP 8.3 called out, Laravel 13 named as the
target, and a pointer to the official upgrade guide for the remaining legs. The
structural content stays valid: 11 rewrote the skeleton and 12 and 13 inherit it,
so this is still the largest step from 10. Rewritten freely per Paul's "without
caring who have read".
Also tightened the post's opening so no skimmer can conclude we SHIPPED the wrong
Propshaft claim. Verified in git: the bad sentence exists in exactly two commits
on one feature branch, introduced and removed before merge. `git log origin/master
-S` returns empty. It never reached a reader.
Fixed one banned word I introduced myself ("journey") in the new notice.
Gates: marketing_copy_test 5 runs / 13 assertions / 0 failures. bin/hugo-build
green, both pages render 200. All EIGHT sections of the post still pass the
cadence quotas. 1,352 -> 1,531 words.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg
Paul extended the docs-only no-wait rule to blog content (2026-08-22). Written into every instruction surface that carries the merge rule: CLAUDE.md, AGENTS.md, the blog-write skill, blog-pipeline.md STEP 7, and .okf/build/ci-gates.md. The two cases are NOT symmetric, and writing the rule without that would have dropped a real gate. Docs PRs merge without waiting because NO CI fires on them - the path filters exclude them. Blog content is different: `content/**` is a trigger path for link-check.yml, and internal links are precisely what a new post adds. "No CI wait" for content therefore means the link crawl MOVED, not that it vanished. So the rule carries an obligation instead of a blanket exemption: run `bin/rake test:links` locally in place of the wait. It is the same task the CI job runs, over a production build. Skipping both is allowed only when stated out loud, because `push: branches: [master]` still crawls after merge - which converts a pre-merge catch into a fix-forward defect rather than eliminating it. The misreading to guard against is "content-only has no link gate". Named explicitly in ci-gates.md so the next reader cannot arrive at it. Ran it on this PR rather than describing it: 1,768 pages scanned, 31,888 unique links, 114,239 OK, ZERO errors. Gates: marketing_copy_test 5 runs / 13 assertions / 0 failures. bin/hugo-build green. bin/rake test:links clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg
pftg
added a commit
that referenced
this pull request
Aug 22, 2026
Stage A found the second cluster in the same HN window - six front-page stories arguing that written material can no longer be trusted at face value, with comment counts that mean argument rather than agreement (989 / 860 / 728 / 690 / 578). Distinct from the expertise cluster the previous post used. Dedup passed. The nearest slug is a dev.to import with a generic content framework; nothing in the archive covers auditing an inherited content archive. Deliberately NOT the same post as what-senior-developers-catch-that-ai-misses: that one is one sentence caught pre-merge and argues about judgment. This one is a runnable method for a founder who owns an archive somebody else produced. Job: ARRIVAL-purposed. **Scope call I made rather than asking.** The post teaches the METHOD and does not enumerate our archive's fabrications. #592 disclosed an error we caught BEFORE merge; confessing months-live fabricated case studies is a materially bigger and irreversible disclosure that should be Paul's explicit yes, not my inference from his last one. The method post is also the more useful artifact for the ICP. One self-critical figure is included - roughly two in five of our substantial posts cited nothing - because it is a rigor gap rather than a fabrication, and it makes the method concrete instead of abstract. STEP 4f caught a real defect in my own draft, which is the second time today. The line citing Pew came from a SEARCH SUMMARY I never fetched, and it was wrong twice over: it said "a random draw of 10,000 pages" (the real sample is ~490,000 Common Crawl texts) and attributed the ~10% figure to all pages when Pew scopes it to .com domains specifically. That same summary had already misattributed one study's numbers to another study's URL earlier today, so it was exactly the source not to trust. Fetched Pew primary instead - published 2026-08-20, two days ago - and the corrected figure is better material: one in ten .com pages carry AI-authorship signals, double the .org rate and ten times .edu/.gov. Commercial sites are where this concentrates, which is the reader's site. Every external claim traced to a fetch: the Hanover Institute, Piro Inc., the $900,000 DOJ filing and the GPTZero result from Responsible Statecraft; the crossover date, sample and 4.2% false-positive rate from Graphite; the sample, tool, date range, per-domain split and detector caveat from Pew. Both studies' error bars are quoted in the body rather than buried, because the post argues for exactly that habit. Gates: cadence quotas measured per H2 with a script - FAILED first pass in three sections (four paragraphs over the 3-sentence cap, two sections with no single-sentence paragraph, one with no sentence over 22 words), all corrected, all six now pass. Banned words zero, em dashes zero, mannerism nouns 4 of 5, regression sweep clean. bin/hugo-build green. marketing_copy_test 5/13/0. bin/rake test:links run LOCALLY per the rule added today: 1,768 pages, 31,888 unique links, zero errors. 1,291 words. NO INDEPENDENT VERIFIER RAN - agent spawning unavailable this session, so the critic panel and cold-eyes gate were executed inline by the author. Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full
/blog-next→/blog-writerun. 1,120 words.Stage A: a cluster, not a hot story
Eight HN front-page posts in three weeks arguing one question, with comment counts that mean argument rather than consensus — 1,335 / 937 / 869 / 590 / 573. The two poles: Sean Goedecke's "LLMs reward expertise" and Senko Rašić's "'Code was never the hard part' is an insult to all programmers".
Dedup passed with a real gap. The three closest posts —
ai-code-ownership-accountability,vibe-coding-crisis-ai-code-debt,vibe-coding-disposable-by-design— are all about the code's properties. This cluster is about the person's value.claude-code-xp-team-workflowcovers four-eyes but for practitioners running agents, not for a founder deciding who to pay.Job: arrival-purposed. The search floor doesn't apply; quoting it would be the category error §13e warns about.
The first-hand evidence is ours, and it's embarrassing
Earlier today an agent removed an unsourced performance number from an old Rails post and replaced it with a confidently wrong mechanism: Propshaft "stops being a build step that scales with your asset count." It does scale — every asset is still walked, fingerprinted and copied. A fabricated number traded for a fabricated mechanism, inside a change whose whole purpose was removing unreliable claims.
That's Goedecke's "steering" running backwards — the argument for expertise made from the losing side.
Gates
bin/hugo-buildgreen · page renders 200 ·marketing_copy_test4 runs / 10 assertions / 0 failures.Reference cadence: patio11 (incident post).
Needs your call before merge — publishing outward
The post says plainly that our own agent introduced a wrong claim into our own blog while cleaning it up. I judge that it builds credibility rather than costing it, and it's the evidence the whole piece rests on. But it's a disclosure about us, which is on the stop list — so this PR is open rather than merged.
Genericising it is a one-paragraph edit if you'd rather not say it.
Also not covered
No independent verifier ran. Agent spawning is unavailable this session, so the critic panel and cold-eyes gate were run inline against their written criteria by the author.
Review link
http://localhost:27431/blog/what-senior-developers-catch-that-ai-misses/🤖 Generated with Claude Code
https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg