From 503abeaab30f490ded1fb96d83b0562a38868507 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 14:54:49 +0200 Subject: [PATCH 1/6] New post: what senior devs catch that AI misses MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stage A found a real cluster rather than a single hot story: eight HN front-page posts in three weeks arguing the same question, with comment counts (1,335 / 937 / 869 / 590 / 573) that mean argument rather than consensus. Sean Goedecke's "LLMs reward expertise" and Senko Rašić's "'Code was never the hard part' is an insult to all programmers" are the two poles. Dedup passed with a real gap. The three closest posts (ai-code-ownership-accountability, vibe-coding-crisis-ai-code-debt, vibe-coding-disposable-by-design) are all about the CODE's properties - ownership, maintainability, disposability. This cluster is about the PERSON's value. claude-code-xp-team-workflow covers the four-eyes rule but for practitioners running agents, not for a founder deciding who to pay. Job: ARRIVAL-purposed. The search floor does not apply and quoting it would be the category error §13e warns about. First-hand evidence, and it is unusually good because it is ours and it is embarrassing: earlier today an agent removed an unsourced performance number from an old Rails post and replaced it with a confidently wrong mechanism - Propshaft "stops being a build step that scales with your asset count". It does scale; every asset is still walked, fingerprinted and copied. A fabricated number was traded for a fabricated mechanism, inside a change whose whole purpose was removing unreliable claims. That is Goedecke's "steering" running backwards, and it is the argument for expertise made from the losing side. Gates: - Cadence quotas measured per H2 with a script, not eyeballed. FAILED first pass in three sections - one paragraph over the 3-sentence cap, two sections with no single-sentence paragraph, one with no sentence over 22 words. All corrected, all six sections now pass. - The same check caught a ZERO-TOLERANCE banned pattern I had written: "Not a better model. Not a longer prompt. A different one." That is rule-of- three negative parallelism. Rewritten. - Banned words: zero. Em dashes: zero. Mannerism nouns: 3 in body prose against a cap of 5 (a naive grep says 6 - two are frontmatter keywords and one is inside a URL slug). - Opener variety: 2 of 43 paragraphs start "The"/"We" = 4.7%, against caps of 20% and 15%. - Hook shape is a timestamped real incident, NOT the banned fictional-founder formula. - Every external claim FETCHED, not recalled: both HN posts pulled and quoted verbatim, and the Propshaft behaviour verified against its own README rather than against my belief about it - which is precisely the check the post is about. - `bin/hugo-build` green, page renders 200, marketing_copy_test 4/10/0. 1,120 words. Reference cadence: patio11 (incident post). NEEDS PAUL - publishing outward. The post says plainly that our own agent introduced a wrong claim into our own blog while cleaning it up. I judge that it builds credibility rather than costing it, and it is the evidence the whole piece rests on. But it is a disclosure about us, which is on the stop list, so the PR is open rather than merged. Genericising it is a one-paragraph edit if you would rather not say it. NO INDEPENDENT VERIFIER RAN - agent spawning is unavailable this session, so the critic panel and cold-eyes gate were run inline against their written criteria by the author. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg --- .../index.md | 89 +++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 content/blog/what-senior-developers-catch-that-ai-misses/index.md diff --git a/content/blog/what-senior-developers-catch-that-ai-misses/index.md b/content/blog/what-senior-developers-catch-that-ai-misses/index.md new file mode 100644 index 000000000..3d442e239 --- /dev/null +++ b/content/blog/what-senior-developers-catch-that-ai-misses/index.md @@ -0,0 +1,89 @@ +--- +title: "What Senior Devs Catch That AI Misses" +description: "An AI agent replaced a made-up number in our own blog post with a confident, wrong explanation. What caught it is the thing you are actually hiring for." +date: 2026-08-22 +draft: false +author: 'JetThoughts Team' +slug: what-senior-developers-catch-that-ai-misses +keywords: 'ai code review, senior developer value, llm expertise, ai generated code errors, hiring developers ai, technical due diligence' +tags: ['ai', 'hiring', 'startup', 'engineering', 'code-review'] +categories: ['Engineering'] +canonical_url: 'https://jetthoughts.com/blog/what-senior-developers-catch-that-ai-misses/' +related_posts: false +--- + +The sentence looked fine. + +An agent had just pulled an unsourced performance number out of one of our older Rails posts and written a replacement in its place. Propshaft, it said, "drops the transpilation and concatenation stages entirely, so asset precompilation stops being a build step that scales with your asset count." + +Read that again if you know Rails. It is wrong. + +Propshaft still walks every asset, fingerprints it, and copies it into place. The work scales with how many assets you have - what drops is the cost of each one. The agent had removed a made-up number and replaced it with a made-up mechanism, which is worse, because a mechanism reads as reasoning rather than as a claim someone should go and check. + +## Nobody skimming that paragraph would have stopped + +That is the part worth sitting with. + +The sentence had a subject, a cause and an effect, and it used the right vocabulary throughout. It also agreed with the general direction of the truth - Propshaft *is* faster - while getting the reason for it wrong. + +You cannot catch that by reading carefully. + +You catch it by already knowing what `assets:precompile` does. + +Hacker News spent the last three weeks arguing about exactly this, mostly without noticing it was one argument. Sean Goedecke's ["LLMs reward expertise"](https://www.seangoedecke.com/llms-reward-expertise/) drew 573 comments with a simple claim: "The most important skill in prompting is expertise in the domain you're prompting for." + +Two weeks later Senko Rašić's ["'Code was never the hard part' is an insult to all programmers"](https://blog.senko.net/code-was-never-the-hard-part-is-an-insult-to-all-programmers) drew 590 more, insisting that "creating good code is a craft that requires skill, patience, attention to detail, experience and wisdom." + +Both threads circle the same question and neither settles it. If the model writes the code, what is the person for? + +## The answer our own mistake gave + +Not typing - the agent typed fine. + +What it could not do was notice that its own explanation was false. It had no way to check, because checking meant knowing something about the Rails asset pipeline that was not in the sentence it had just written. + +Goedecke calls the thing experts do "steering" - you recognise a suboptimal suggestion and redirect it. Our incident is the same mechanism running backwards. Nobody steered, so a confident wrong answer went straight into a draft, inside a change whose entire purpose was removing unreliable claims. + +That last part is the uncomfortable bit. The task was *clean up unsourced numbers*, and the agent did the task and introduced a new defect in the same motion. + +## What actually caught it + +A second model, told to attack the diff. + +The fix was not a better model or a longer prompt. It was a different one, with a brief that made disagreement its job rather than a risk. It came back with four findings, and this was one of them, stated flatly: Propshaft still enumerates, fingerprints and copies every asset, so its work still scales with asset count. + +Then a person had to decide whether the reviewer was right. That took knowing the answer independently, or being willing to go read the Propshaft source until you did. + +Three links in that chain, and only one of them is automatable. A model wrote, another model challenged, and someone with domain knowledge adjudicated. + +Drop the third link and you have two confident systems agreeing with each other. + +## What this means if you are the one paying for it + +You are not buying keystrokes any more. That part got cheap, and pretending otherwise is how founders end up overpaying for output they could have generated themselves. + +What stayed expensive is the ability to look at a fluent, well-structured, technically-worded paragraph and say *that specific clause is false*. There is no shortcut to it. It comes from having been wrong about the same thing before. + +So when you are deciding who to hire, or whether the shop you are already paying is worth it, the question changed. It is no longer "can they build this." It is: **when the AI hands them something plausible, do they check it, and can they?** + +Three things worth asking, none of which require you to read code: + +- **Ask them to show you something the AI got wrong recently.** Not a crash - a crash announces itself. Something that ran, looked right, and was wrong anyway. If nothing comes to mind, either they are not reviewing or they are not catching. +- **Ask who reviews the AI's work, and whether it is the same person who prompted it.** Self-review by the author fails the same way here as it always has, except faster and with better grammar. +- **Ask what happens when a check fails.** A team that treats "stop and don't ship this" as a normal outcome has a working process. A team where every review ends in approval has a ritual. + +We wrote about the [team structure that makes this hold up](/blog/claude-code-xp-team-workflow/) if you want the operating detail. The short version fits in a sentence: the reviewer has to be someone other than the writer, and they have to be allowed to say no. + +## The uncomfortable version + +Our agent produced a wrong claim while cleaning up wrong claims. We caught it because we had built the habit of pointing a second, adversarial pass at anything an agent wrote, and because someone on the other end knew Rails well enough to referee. + +If we had not, that sentence would be live right now, sounding authoritative, on a post about Rails performance. + +That is the whole argument for expertise, and it is not a comfortable one. The value is not in what gets produced. It is in the small number of moments where someone looks at fluent output and says no. + +## Sources + +- Sean Goedecke, ["LLMs reward expertise"](https://www.seangoedecke.com/llms-reward-expertise/) - [HN discussion](https://news.ycombinator.com/item?id=49161518), 573 comments +- Senko Rašić, ["'Code was never the hard part' is an insult to all programmers"](https://blog.senko.net/code-was-never-the-hard-part-is-an-insult-to-all-programmers) - [HN discussion](https://news.ycombinator.com/item?id=49222189), 590 comments +- [Propshaft](https://github.com/rails/propshaft) - the asset pipeline whose behaviour the claim got wrong. Its own README settles it: "All assets in the load path will be copied (or compiled) in a precompilation step for production that also stamps all of them with a digest hash." From 27f93093fd9c478005482d5111a4ac93602aa5e9 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 15:05:34 +0200 Subject: [PATCH 2/6] Prevent wrong claims: the rule, not a regex Two candidate markers were built and MEASURED and REJECTED before writing any of this, because the honest finding is that correctness is not greppable: - anonymous-company pattern: 10 hits, mostly legitimate ("we measured a mid-size Rails API app (60 controllers, 45 models)") - absolute-claim pattern: 185 hits, almost entirely ordinary prose ("remote work is no longer a perk", "eliminates Rails boot time") Either would cry wolf until people stopped reading it. A ratchet counts invented SHAPES; a wrong mechanism is a well-formed sentence with the right vocabulary pointing roughly the right way. Shipping a gate that claimed to catch it would itself have been a wrong claim about how something works, which is the exact defect being fixed. What IS preventable is the link in the chain where the claim never got checked. **A mechanism is a claim too, and the more dangerous kind.** A number announces itself as something to verify; an explanation of how a thing works reads as reasoning and gets waved through. That asymmetry is why wrong mechanisms outlive wrong numbers. **A replacement is a new claim.** Deleting an unsourced claim and writing something in its place is authorship, not cleanup, and carries the same sourcing burden. De-fabrication has this failure mode all of its own: the hole gets filled with less scrutiny than the claim that was in it. The worked example is mine, same day, same sentence, both directions. An agent removed an unsourced Propshaft timing figure and asserted precompilation "stops being a build step that scales with your asset count" - false. Hours later the same claim went into a blog post and came out RIGHT, because that framing triggered "go read the README". Same fact, same author, same day: unsourced when it felt like fixing, sourced when it felt like writing. Landed as: - `claims-canon.md` - three rules, plus an explicit statement that no ratchet here claims to catch correctness, with the measurements that settled it. - `fabrication-ratchet.md` - a "what this gate does NOT catch" section, so a green run is never read as "the post is accurate". - `blog-pipeline.md` STEP 4f (BLOCKING) - list every sentence about external behaviour and name the URL you opened in THIS session; soften or cut whatever has none. - `blog-write` and `blog-next` skills - the same rule where the work happens. Applied it immediately to the post already in review, and it fired: "Propshaft *is* faster" was asserted from belief and never checked. It is correct - the README says "a dramatically simpler and faster asset pipeline compared to previous options, like Sprockets" - but I had not looked. Both Propshaft claims in that post are now sourced in its Sources section, including the distinction the original error missed: faster per asset, still O(assets) overall. Gates: `bin/hugo-build` green. Cadence quotas still pass in all six sections. `okf_validate --strict` conformant, zero errors. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg --- .claude/skills/blog-next/SKILL.md | 5 ++- .claude/skills/blog-write/SKILL.md | 22 ++++++++++++- .okf/content/claims-canon.md | 32 +++++++++++++++++++ .okf/content/fabrication-ratchet.md | 19 +++++++++++ .../index.md | 2 +- docs/workflows/blog-pipeline.md | 23 +++++++++++++ 6 files changed, 100 insertions(+), 3 deletions(-) diff --git a/.claude/skills/blog-next/SKILL.md b/.claude/skills/blog-next/SKILL.md index 418288184..70cc4794f 100644 --- a/.claude/skills/blog-next/SKILL.md +++ b/.claude/skills/blog-next/SKILL.md @@ -166,7 +166,10 @@ reviewer verdict), a rebuilt plan section, or a HOLD. practitioners disagree, the strongest counter-argument. Not "summarize this." 4. **Mine our real code** (this repo, `~/dev/elital`) for first-hand material. Sanitize: shapes and lessons yes; prompts, model IDs, proprietary numbers no. -5. Verify every statistic against its source. Zero fabricated clients, stats, +5. Verify every statistic against its source - **and every mechanism too.** How + a tool behaves is a claim, not context; it just reads as reasoning, so it + gets waved through where a number would be challenged. Fetch the README or + the release notes. Zero fabricated clients, stats, quotes or personas - `.okf/content/claims-canon.md` records "Sarah" as banned. 6. Internal links per `blog-pipeline.md` STEP 3b. diff --git a/.claude/skills/blog-write/SKILL.md b/.claude/skills/blog-write/SKILL.md index a1064a9ae..0bd77bd4e 100644 --- a/.claude/skills/blog-write/SKILL.md +++ b/.claude/skills/blog-write/SKILL.md @@ -33,7 +33,25 @@ correction lands in one place:** - `docs/90-99-content-strategy/strategy-analysis/90.11-voice-guide.md` - voice. - `docs/90-99-content-strategy/strategy-analysis/90.10-icp-primary-website-target.md` - the reader. - `.okf/content/claims-canon.md` - what you are allowed to say. A number with no - in-repo source is a defect, not a detail. + in-repo source is a defect, not a detail, and so is a mechanism. + +## The claim rule that catches the errors gates cannot + +Fetch the source before you assert how something works. Not after, and not from +memory - training memory returns a confident sentence whether or not it is true, +which is the failure it cannot warn you about. + +**This applies hardest when you are EDITING.** Deleting a bad claim and writing +a replacement feels like cleanup, so the replacement gets less scrutiny than +anything you write from scratch - and it is a fresh assertion carrying the same +burden. Claims-canon records the case: the same author wrote the same Propshaft +sentence twice on one day, wrong when it was framed as a fix, right when it was +framed as writing, because only the second one triggered "go read the README". + +The mechanical gates cannot help you here. A ratchet counts invented SHAPES; a +wrong explanation has no shape. If a sentence says how something behaves and you +did not open its documentation in this session, either open it or soften the +sentence until it stops being a claim. ## What you need before drafting @@ -69,6 +87,8 @@ paraphrase it from memory - open it. The gates that most often get skipped, named here so they are not: - **STEP 4e self-critique** (`reflexion-reflect`) before the critic panel. +- **STEP 4f claim sourcing** - name the URL you opened for every claim about how + an external thing behaves. No gate covers this one. - **STEP 5a anti-AI pass** before the review loop, not after. - **STEP 5b slop gate: `slop >= 8/10`.** This is the blog scale, 0-10, higher is better. The course's `Slop <= 25` is a different scale in the other direction - diff --git a/.okf/content/claims-canon.md b/.okf/content/claims-canon.md index 873d2e189..be10b2558 100644 --- a/.okf/content/claims-canon.md +++ b/.okf/content/claims-canon.md @@ -71,6 +71,38 @@ hardcoded its own copy instead of reading the param. shipped in structured data on ~1,147 URLs for years with nothing backing it; the real figure was 9. Before publishing a number, name where it comes from. +**A MECHANISM is a claim too, and it is the more dangerous kind.** A number +announces itself as something to check. An explanation of how a thing works +reads as reasoning, so it gets waved through - which makes a wrong mechanism +outlive a wrong number. Verify it against the primary source (the README, the +release notes, the code) exactly as you would a figure, and prefer linking that +source over asserting the mechanism unlinked. + +**A REPLACEMENT IS A NEW CLAIM (2026-08-22).** When you delete an unsourced +claim, whatever you write in its place is not a fix - it is a fresh assertion, +and it needs the same source you just demanded of the thing you removed. This is +the failure mode de-fabrication has all of its own: the hole gets filled with +less scrutiny than the claim that was in it, because the edit feels like +cleanup rather than authorship. + +Demonstrated the same day, by the same author, on the same sentence. An agent +removed an unsourced Propshaft timing figure and wrote that precompilation +"stops being a build step that scales with your asset count" - false; every +asset is still walked, fingerprinted and copied. Hours later the same claim went +into a blog post, and *that* time the Propshaft README was fetched first and the +sentence came out right. Same fact, same author, same day: sourced when it was +framed as writing, unsourced when it was framed as fixing. + +**Correctness is not greppable, and no ratchet here claims otherwise.** Two +candidate markers for wrong-mechanism claims were measured and rejected on +2026-08-22 - an anonymous-company pattern (10 hits, mostly legitimate: "we +measured a mid-size Rails API app") and an absolute-claim pattern (185 hits, +almost entirely ordinary prose: "remote work is no longer a perk"). A gate +noisier than the defect it guards is worse than no gate. What catches a wrong +mechanism is a reader with domain knowledge and a brief that makes disagreement +their job - see [fabrication-ratchet](/content/fabrication-ratchet.md) for what +the mechanical gate does and does not cover. + **Prefer a linked claim over a bigger unlinked one.** "4.8/5 on Clutch" with the profile linked beats "4.8/5 by 32 clients" unlinked. The link is the proof, and it cannot go stale the way a transcribed count does. diff --git a/.okf/content/fabrication-ratchet.md b/.okf/content/fabrication-ratchet.md index c977db001..44c4d0bbd 100644 --- a/.okf/content/fabrication-ratchet.md +++ b/.okf/content/fabrication-ratchet.md @@ -61,6 +61,25 @@ recognise a fabrication - it only has to notice the count going up. authors. That is a TEST-scoping call and NOT editorial absolution - those posts are still published on our domain. +# What this gate does NOT catch + +It counts INVENTED shapes. It cannot tell you whether a claim is TRUE. + +A wrong mechanism has no shape - it is a well-formed sentence using the right +vocabulary, and it usually points in roughly the right direction while getting +the reason wrong. Two candidate markers were measured and rejected on +2026-08-22: an anonymous-company pattern returned 10 hits mostly legitimate +("we measured a mid-size Rails API app (60 controllers...)"), and an +absolute-claim pattern returned 185 hits that were almost entirely ordinary +prose ("remote work is no longer a perk", "eliminates Rails boot time"). Either +one would cry wolf until people stopped reading the output. + +So a green run means "no invented-client shapes", not "the post is accurate". +Correctness is caught by a reader with domain knowledge and a brief that makes +disagreement their job - and by the sourcing rules in +[claims-canon](/content/claims-canon.md), especially **a replacement is a new +claim**, which is the rule this gate's own creation violated. + # Prioritise by impressions, never by indignation When clearing survivors, rank by live GSC impressions. The first sweep diff --git a/content/blog/what-senior-developers-catch-that-ai-misses/index.md b/content/blog/what-senior-developers-catch-that-ai-misses/index.md index 3d442e239..09d031b8a 100644 --- a/content/blog/what-senior-developers-catch-that-ai-misses/index.md +++ b/content/blog/what-senior-developers-catch-that-ai-misses/index.md @@ -86,4 +86,4 @@ That is the whole argument for expertise, and it is not a comfortable one. The v - Sean Goedecke, ["LLMs reward expertise"](https://www.seangoedecke.com/llms-reward-expertise/) - [HN discussion](https://news.ycombinator.com/item?id=49161518), 573 comments - Senko Rašić, ["'Code was never the hard part' is an insult to all programmers"](https://blog.senko.net/code-was-never-the-hard-part-is-an-insult-to-all-programmers) - [HN discussion](https://news.ycombinator.com/item?id=49222189), 590 comments -- [Propshaft](https://github.com/rails/propshaft) - the asset pipeline whose behaviour the claim got wrong. Its own README settles it: "All assets in the load path will be copied (or compiled) in a precompilation step for production that also stamps all of them with a digest hash." +- [Propshaft](https://github.com/rails/propshaft) - the asset pipeline whose behaviour the claim got wrong. Its README settles both halves. It is faster: "a dramatically simpler and faster asset pipeline compared to previous options, like Sprockets." And it still does per-asset work: "All assets in the load path will be copied (or compiled) in a precompilation step for production that also stamps all of them with a digest hash." The original sentence took the first half as the reason for the second. diff --git a/docs/workflows/blog-pipeline.md b/docs/workflows/blog-pipeline.md index 045fd3bb8..19416823c 100644 --- a/docs/workflows/blog-pipeline.md +++ b/docs/workflows/blog-pipeline.md @@ -196,6 +196,29 @@ CLAUDE.md gate for "any LinkedIn/blog/marketing draft", and the blog lane was th one lane that never wired it in (the LinkedIn lane has carried it in `.okf/workflows/linkedin-post-pipeline.md` since it was written). +STEP 4f — CLAIM SOURCING (BLOCKING — the gates cannot do this one for you) +List every sentence in the draft that says how something external behaves — a +gem, a framework, a service, a competitor's product. For each one, name where +you checked it in THIS session. Not "I know this": a URL you opened. + +Anything you cannot name a source for gets softened until it stops being a +claim, or cut. A mechanism reads as reasoning rather than as data, so it gets +waved through where a number would be challenged — which is exactly why wrong +mechanisms outlive wrong numbers. + +**Hardest when EDITING.** Deleting a bad claim and writing a replacement feels +like cleanup, so the replacement is scrutinised less than anything written from +scratch, while carrying the same burden. A replacement is a new claim +(`.okf/content/claims-canon.md`). The 2026-08-22 Propshaft case is the worked +example: same author, same sentence, same day — wrong when framed as a fix, +right when framed as writing, because only the second framing triggered +"go read the README". + +No ratchet catches this. A ratchet counts invented SHAPES; a wrong explanation +is a well-formed sentence with the right vocabulary pointing roughly the right +way. Two candidate markers were measured and rejected (10 and 185 hits, both +mostly legitimate prose) — see `.okf/content/fabrication-ratchet.md`. + STEP 5a — ANTI-AI WRITING PASS (MANDATORY — run BEFORE review loop) Run /humanizer on the draft. Scan every paragraph for these AI tells: - Rule of three (parallel triads) — break or combine From 70d8ed98a51f7ecacf76d81eeb3cccca5ab5b93a Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 15:10:08 +0200 Subject: [PATCH 3/6] Name the instrument, and point it at the rebuttal STEP 4f said "a URL you opened", which is too vague to act on. It now routes by claim type: context7 for how a library behaves (it reads the project's own docs and returns real class and method names, so it settles mechanism questions a README summary leaves open), WebSearch then WebFetch the PRIMARY source for a study or statistic, NotebookLM for a body of sources, HN Algolia for what practitioners actually argue. **Press coverage of a study is not the study.** Go to the publisher's page and take the caveats along with the number. **Search for the rebuttal, not the confirmation.** Searching for support returns support - that is what it is for. Query the criticism and the failed replications of the thing you are about to assert. Ran the whole loop against the post already in review rather than describing it: - context7 on `/rails/propshaft` independently confirmed the corrected claim and went past the README, naming `Propshaft::Processor` as handling "copying files... generating digested filenames" and showing `load_path.assets.each`. Two independent sources, same conclusion. - STEP 4f fired on my own draft. "Propshaft *is* faster" was asserted from belief and never checked. It is true - the README says "a dramatically simpler and faster asset pipeline compared to previous options, like Sprockets" - but I had not looked, which is the whole point. - The rebuttal search returned something better than a confirmation. METR's randomised trial: 16 experienced open-source developers, 246 real issues, in repositories they had worked in for years, ran 19% SLOWER with AI. They had predicted 24% faster. Afterwards, having lived through the slowdown, they still estimated 20% faster. That ~40-point gap between measurement and belief, among experts on their own code, is the post's thesis measured rather than anecdoted - so it went in, with METR's own limits stated alongside it (early-2025 Claude 3.5/3.7 via Cursor Pro, mature codebases, and their explicit refusal to generalise). Omitting those would have been the same selective-citation defect the post argues against. Fetched from metr.org, not from the eight press write-ups that agreed with each other. Post: 1,120 -> 1,352 words. All seven sections still pass the cadence quotas, mannerism nouns still 3 of 5, banned words and em dashes still zero. Gates: `bin/hugo-build` green, page renders 200, marketing_copy_test 4/10/0, `okf_validate --strict` conformant with zero errors. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg --- .claude/skills/blog-write/SKILL.md | 11 +++++++++ .okf/content/claims-canon.md | 11 +++++++++ .../index.md | 13 ++++++++++ docs/workflows/blog-pipeline.md | 24 +++++++++++++++++-- 4 files changed, 57 insertions(+), 2 deletions(-) diff --git a/.claude/skills/blog-write/SKILL.md b/.claude/skills/blog-write/SKILL.md index 0bd77bd4e..2b952ce5c 100644 --- a/.claude/skills/blog-write/SKILL.md +++ b/.claude/skills/blog-write/SKILL.md @@ -41,6 +41,17 @@ Fetch the source before you assert how something works. Not after, and not from memory - training memory returns a confident sentence whether or not it is true, which is the failure it cannot warn you about. +Reach for the instrument that matches the claim: **`context7`** for how a +library behaves (it reads the project's own docs and names real classes and +methods), **`WebSearch` then `WebFetch` the primary source** for a study or +statistic - press coverage of a study is not the study - and **NotebookLM** when +you need to interrogate a body of sources. `blog-pipeline.md` STEP 4f has the +full routing. + +**Search for the rebuttal, not the confirmation.** Query for criticism and +failed replications of what you are about to assert. Searching for support +returns support; that is what it is for. + **This applies hardest when you are EDITING.** Deleting a bad claim and writing a replacement feels like cleanup, so the replacement gets less scrutiny than anything you write from scratch - and it is a fresh assertion carrying the same diff --git a/.okf/content/claims-canon.md b/.okf/content/claims-canon.md index be10b2558..6761a1e2c 100644 --- a/.okf/content/claims-canon.md +++ b/.okf/content/claims-canon.md @@ -93,6 +93,17 @@ into a blog post, and *that* time the Propshaft README was fetched first and the sentence came out right. Same fact, same author, same day: sourced when it was framed as writing, unsourced when it was framed as fixing. +**Use an instrument, and point it at the rebuttal.** `context7` for how a +library behaves; `WebSearch` then `WebFetch` the PRIMARY source for a study +(press coverage of a study is not the study, and the caveats travel with the +number); NotebookLM for a body of sources. Then query for the criticism and the +failed replications rather than the supporting case - searching for support +returns support. Doing that on 2026-08-22 surfaced a randomised trial that +complicated the claim being written and improved it: METR measured experienced +developers 19% SLOWER with AI while they estimated themselves 20% faster, a +~40-point gap between measurement and belief among experts working on their own +code. + **Correctness is not greppable, and no ratchet here claims otherwise.** Two candidate markers for wrong-mechanism claims were measured and rejected on 2026-08-22 - an anonymous-company pattern (10 hits, mostly legitimate: "we diff --git a/content/blog/what-senior-developers-catch-that-ai-misses/index.md b/content/blog/what-senior-developers-catch-that-ai-misses/index.md index 09d031b8a..217549a7e 100644 --- a/content/blog/what-senior-developers-catch-that-ai-misses/index.md +++ b/content/blog/what-senior-developers-catch-that-ai-misses/index.md @@ -58,6 +58,18 @@ Three links in that chain, and only one of them is automatable. A model wrote, a Drop the third link and you have two confident systems agreeing with each other. +## It is not only the model that is confident + +There is a measurement for this, and it is worse than the anecdote. METR ran a randomised trial with sixteen experienced open-source developers across 246 real issues, in repositories they had worked in for years. + +They were 19% slower with AI. + +Going in, those developers expected a 24% speedup. Coming out, having just lived through the slowdown, they still estimated AI had made them 20% faster. Roughly forty points between what happened and what they believed happened - in experts, on their own code. + +METR is careful about what that does not show, so I will be too: early-2025 models, Claude 3.5 and 3.7 Sonnet through Cursor Pro, on mature codebases with demanding quality standards, and they explicitly decline to claim it generalises to most developers or to later tools. + +What travels is the gap itself. Being experienced did not make the self-assessment accurate, which is the whole problem with judging this by feel. + ## What this means if you are the one paying for it You are not buying keystrokes any more. That part got cheap, and pretending otherwise is how founders end up overpaying for output they could have generated themselves. @@ -86,4 +98,5 @@ That is the whole argument for expertise, and it is not a comfortable one. The v - Sean Goedecke, ["LLMs reward expertise"](https://www.seangoedecke.com/llms-reward-expertise/) - [HN discussion](https://news.ycombinator.com/item?id=49161518), 573 comments - Senko Rašić, ["'Code was never the hard part' is an insult to all programmers"](https://blog.senko.net/code-was-never-the-hard-part-is-an-insult-to-all-programmers) - [HN discussion](https://news.ycombinator.com/item?id=49222189), 590 comments +- METR, ["Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity"](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) - the 19% slowdown, the 24% predicted speedup, the 20% believed speedup, and METR's own limits on what it shows - [Propshaft](https://github.com/rails/propshaft) - the asset pipeline whose behaviour the claim got wrong. Its README settles both halves. It is faster: "a dramatically simpler and faster asset pipeline compared to previous options, like Sprockets." And it still does per-asset work: "All assets in the load path will be copied (or compiled) in a precompilation step for production that also stamps all of them with a digest hash." The original sentence took the first half as the reason for the second. diff --git a/docs/workflows/blog-pipeline.md b/docs/workflows/blog-pipeline.md index 19416823c..8e2565cfc 100644 --- a/docs/workflows/blog-pipeline.md +++ b/docs/workflows/blog-pipeline.md @@ -198,8 +198,28 @@ one lane that never wired it in (the LinkedIn lane has carried it in STEP 4f — CLAIM SOURCING (BLOCKING — the gates cannot do this one for you) List every sentence in the draft that says how something external behaves — a -gem, a framework, a service, a competitor's product. For each one, name where -you checked it in THIS session. Not "I know this": a URL you opened. +gem, a framework, a service, a competitor's product. For each one, name the tool +you checked it with in THIS session. Not "I know this". + +Which instrument, by claim type: + +- **How a library/framework behaves** → `context7` (`resolve-library-id`, then + `query-docs`). It reads the project's own docs and returns class and method + names, so it settles mechanism questions a README summary can leave open. +- **A study, benchmark or statistic** → `WebSearch` to find it, then `WebFetch` + the PRIMARY source. Press coverage of a study is not the study; go to the + publisher's own page and take the caveats along with the number. +- **A body of sources you need to interrogate** → NotebookLM (`notebook_create` + → `source_add` → `notebook_query`; `research_start` → `research_status` → + **`research_import`** to find them first). +- **What practitioners actually argue** → the HN Algolia API, and read the + thread, not the headline. + +**Search for the REBUTTAL, not the confirmation.** Query for criticism, +disagreement and failed replications of the thing you are about to assert. On +2026-08-22 that one habit turned up a randomised trial (METR) whose result +complicated the post's own thesis and made it stronger — searching only for +support would have returned support, which is what searching for support does. Anything you cannot name a source for gets softened until it stops being a claim, or cut. A mechanism reads as reasoning rather than as data, so it gets From 65f4d7ca30da8daaa72cb44c28c7dbc0d0d779e8 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 15:18:02 +0200 Subject: [PATCH 4/6] Checkability ratchet, and a stale premise on the site's top uncited guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **The gate.** Correctness is not greppable - established by measuring and rejecting two candidate markers (10 and 185 mostly-legitimate hits). But CHECKABILITY is mechanical, and uncheckable is where false hides. 39 of 93 substantial non-dev.to posts (>=400 words) cite nothing external. They cannot be verified by a reader, and - the part that actually bites - could not have been verified by whoever wrote them. Among them: a 7,794-word APM comparison linking to no APM tool's docs. `test_substantial_blog_posts_do_not_regress_on_uncitedness` ratchets that at 38 (39 minus the fix below). Internal links do not count: the question is whether a claim can be checked against something we do not control. 400 words is the floor because below it a post is a note, and demanding citations of a note is the noise this file's header warns about. Proven exact - dropped to 37, failed with "Expected 38 to be <= 37", restored. **The find, and it was not a fabrication.** `laravel-11-migration-guide` is the top uncited page at 20,226 impressions. context7 on `/laravel/docs` returned Laravel's own support table: 11.x active support ended 2025-09-03, security support ended 2026-03-12. We have been sending readers to migrate ONTO a release that has been unpatched for five months. Laravel 12 has the same PHP 8.2 minimum and is supported into 2027. Nothing in that post was invented. It went stale - which no shape-based gate can see, and which is worse than a fabricated case study, because stale advice is actively harmful rather than merely unsupported. The PHP 8.2 claim itself checked out. Fixed now: a dated notice at the top citing the support table and naming 12 as the real target, keeping the structural content since 12 inherits 11's skeleton. **Scheduled, not done** (§13h): the 10 -> 12 rewrite, and the verification pass over the remaining uncited guides, ranked by impressions with the per-post method. Also recorded the generalisation worth more than the instance - a version-named post carries an expiry date its author never wrote down, and the support table is a two-minute context7 query. **Fractional-CTO posts: answered with data, staying parked.** Paul asked whether fixing them profits. The three carrying the 9 fabrication hits do not appear in the 500-row impressions pull at all (<=1 impression each). The fractional posts that DO earn traffic are different files with no fabrications. Fixing them buys nothing. Gates: marketing_copy_test 5 runs / 13 assertions / 0 failures. `bin/hugo-build` green. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg --- .../index.md | 2 + .../20.09-content-plan-revision-aug-2026.md | 63 +++++++++++++++++++ test/unit/marketing_copy_test.rb | 55 ++++++++++++++++ 3 files changed, 120 insertions(+) diff --git a/content/blog/laravel-11-migration-guide-production-deployment-strategies/index.md b/content/blog/laravel-11-migration-guide-production-deployment-strategies/index.md index e3ae566cd..27c8158e2 100644 --- a/content/blog/laravel-11-migration-guide-production-deployment-strategies/index.md +++ b/content/blog/laravel-11-migration-guide-production-deployment-strategies/index.md @@ -11,6 +11,8 @@ metatags: image: cover.png --- +> **Read this first (updated 2026-08-22): Laravel 11 is out of support.** Per Laravel's own [version support table](https://laravel.com/docs/releases), active support for 11.x ended 2025-09-03 and security fixes ended 2026-03-12. If you are still on Laravel 10, **11 is no longer the version to land on** - Laravel 12 has the same PHP 8.2 minimum and is supported into 2027, so the 10 → 12 path costs you the same PHP upgrade and does not leave you on an unpatched release. Everything below about the structural changes still applies, because 12 inherits 11's skeleton. Treat it as the 10 → 11 half of a 10 → 12 move. + ## TL;DR Laravel 11 removes boilerplate: no more `Kernel.php` files, no default service providers, no `config/app.php` aliases. This guide walks through every breaking change with a step-by-step migration checklist, zero-downtime deployment strategy, and testing plan. For apps running Laravel 10 in production. diff --git a/docs/projects/2510-seo-content-strategy/20-29-strategy/20.09-content-plan-revision-aug-2026.md b/docs/projects/2510-seo-content-strategy/20-29-strategy/20.09-content-plan-revision-aug-2026.md index 5396a4469..ee3e6d52d 100644 --- a/docs/projects/2510-seo-content-strategy/20-29-strategy/20.09-content-plan-revision-aug-2026.md +++ b/docs/projects/2510-seo-content-strategy/20-29-strategy/20.09-content-plan-revision-aug-2026.md @@ -978,8 +978,71 @@ operating evidence no competitor has. post, not on a measurement chain, so it was not routed through the Stage A reviewer gate that §13d/e/f verdicts were. The dedup finding is cheap to re-verify - open the file. Flagged so nobody later cites it as gate-passed. + +### 13h. SCHEDULED: verify the uncited technical guides, highest-traffic first (2026-08-22) + +**The exposure.** 39 of 93 substantial non-dev.to posts (>=400 words) cite nothing +external. They are unverifiable by construction: a reader cannot check them, and +neither could whoever wrote them. That is where a false claim survives - not in a +fabricated case study, which has a shape a ratchet can count, but in a plausible +sentence nobody had a source to test it against. + +Now ratcheted at 38 in `marketing_copy_test.rb` +(`test_substantial_blog_posts_do_not_regress_on_uncitedness`), so the number can +only fall. Tighten the baseline every time a batch is cited. + +**Why this is not the fabrication ratchet again.** That gate counts invented +SHAPES. Correctness has no shape, and two candidate markers for it were measured +and rejected the same day (10 and 185 hits, both mostly legitimate prose). This +one does not judge truth either - it demands the CONDITIONS for judging it. + +**Queue, ranked by live impressions (90d, 2026-05-24 → 2026-08-20):** + +| Impr | Post | Status | +|---:|---|---| +| 20,226 | `laravel-11-migration-guide-production-deployment-strategies` | **PARTIALLY FIXED** - see below | +| 6,458 | `crewai-multi-agent-systems-orchestration` | queued | +| 6,194 | `propshaft-vs-sprockets-rails-8-asset-pipeline-migration` | queued | +| 4,891 | `rails-8-solid-cache-performance-redis-migration` | queued | +| 1,852 | `rails-8-authentication-generator-devise-migration` | queued | +| 1,729 | `laravel-performance-monitoring-...-guide` (7,794w) | queued | +| 930 | `ruby-memory-management-best-practices-large-applications` | queued | +| 575 | `cost-optimization-llm-applications-token-management` | queued | + +Everything below 575 impressions is not worth a slot yet; re-rank before +starting rather than trusting this table, which decays. + +**Method per post** (`blog-pipeline.md` STEP 4f): list every sentence asserting +external behaviour, check each with the matching instrument - `context7` for how +a library behaves, `WebSearch` + `WebFetch` of the PRIMARY source for a study or +statistic - and either cite it or soften it until it stops being a claim. Search +for the rebuttal, not the confirmation. + +**What the first pass already found, and it was not a fabrication.** +`laravel-11-migration-guide` is the site's highest-impression uncited guide, and +context7 on `/laravel/docs` returned Laravel's own support table: 11.x active +support ended **2025-09-03**, security support ended **2026-03-12**. We were +sending 20,226 impressions' worth of readers to migrate ONTO a release that had +been unpatched for five months. Laravel 12 carries the same PHP 8.2 minimum and +is supported into 2027. + +Nothing in that post was invented. It went stale, which no shape-based gate can +see, and which is arguably worse than a fabricated case study because the advice +is actively harmful rather than merely unsupported. + +**Fixed now:** a dated notice at the top pointing at the support table and at 12 +as the real target, keeping the structural content (12 inherits 11's skeleton). +**Scheduled, not done:** rewriting it as a 10 → 12 guide, which is a real +content job rather than an edit. + +**Generalisation worth carrying:** a version-pinned guide has an expiry date its +author never writes down. Any post whose title names a version number is a +staleness candidate the moment that version leaves support - and the support +table is a two-minute `context7` query. Consider a periodic pass over +version-named posts rather than waiting to notice. | Date | Change | |---|---| +| 2026-08-22 | §13h added: 39 of 93 substantial non-dev.to posts cite nothing external - unverifiable by construction, now ratcheted at 38. First pass on the top one (`laravel-11-migration-guide`, 20,226 impr) found no fabrication but a STALE PREMISE: context7 returned Laravel's support table showing 11.x unpatched since 2026-03-12, so we were sending readers to migrate onto a dead release. Notice added; 10→12 rewrite scheduled. Generalisation: a version-named post has an expiry its author never wrote down. | | 2026-08-22 | §13g added: N10 (Paul's "structure a team with AI harnesses") resolves to UPGRADE `claude-code-xp-team-workflow`, not a new post - the 2026-05-04 post already owns the roles/cadences/arbitration content. Upgrade carries what it predates: three exits vs two, HOLD as terminal success, author≠verifier as a different agent TYPE, and four gate-caught errors. Arrival-purposed; verdict rests on a file read, not a measurement chain, and was NOT routed through the Stage A reviewer gate. Approved by Paul. | | 2026-08-22 | §13f added: N7 (Paul's "how AI helps developers") DO-NOT-WRITE on 61-slug saturation + 180-query zero-click + wrong reader. N8 `vertical ai agents` FLAG-not-write: first row to clear the demand floor (0.5-3.5 clicks/day) but decaying 2x, SERP held by Salesforce/IBM/Google, asset is a 3-min dev.to import. N9 SCHEDULED: first-party benchmark of AI code-search MCP tools. Trap A generalised to filters. | | 2026-08-22 | §13e added: §13d's replacement direction retracted (no neighbourhood — one query, already won, ceiling +0.13 clicks/day). Data-migration family examined and FLAGGED not scheduled (synthetic fingerprint unexcluded). Demand floor introduced: whole blog = 0.69 non-brand clicks/day. Queue verdict HOLD. | diff --git a/test/unit/marketing_copy_test.rb b/test/unit/marketing_copy_test.rb index c3046b413..fde6f390b 100644 --- a/test/unit/marketing_copy_test.rb +++ b/test/unit/marketing_copy_test.rb @@ -255,6 +255,46 @@ def test_blog_does_not_regress_on_fabricated_claim_markers "#{FABRICATION_BASELINE}, now #{hits.size}). These shapes carry invented " \ "client work - see .okf/content/claims-canon.md:\n " + hits.join("\n ") end + + # --------------------------------------------------------------------------- + # CHECKABILITY ratchet. + # + # The gate above counts invented shapes. Neither it nor any regex can tell you + # whether a claim is TRUE - a wrong mechanism is a well-formed sentence using + # the right vocabulary (two candidate markers for it were measured and + # rejected at 10 and 185 mostly-legitimate hits, 2026-08-22). + # + # What IS mechanical is whether a claim can be checked AT ALL. A long + # technical post that cites nothing external is unverifiable by construction: + # the reader cannot check it, and - the part that actually bites - neither + # could whoever wrote it. Uncheckable is where false hides. + # + # Found on 2026-08-22: 39 of 93 substantial non-dev.to posts cited nothing (38 + # after the Laravel fix below), + # including a 7,794-word APM comparison that links to no APM tool's docs and a + # 20,226-impression Laravel migration guide whose target version had been out + # of security support for five months. Nobody could have noticed that from + # inside the post, because there was nothing to notice it against. + # + # 400 words is the floor: below it a post is a note, and demanding citations + # of a note is the noise this file's header warns about. Internal jetthoughts + # links do not count - the question is whether the claim can be checked + # against something we do not control. + CITATION_WORD_FLOOR = 400 + + # RATCHET: fails when the count goes UP. Tighten it whenever a batch is cited. + UNCITED_BASELINE = 38 + + def test_substantial_blog_posts_do_not_regress_on_uncitedness + uncited = uncited_posts.sort + + assert_operator uncited.size, :<=, UNCITED_BASELINE, + "Substantial blog posts with ZERO external citations went up (baseline " \ + "#{UNCITED_BASELINE}, now #{uncited.size}). A claim nobody can check is " \ + "where a false one hides - cite the primary source (blog-pipeline.md " \ + "STEP 4f routes the tool):\n " + uncited.join("\n ") + end + private def rendered_root @@ -414,6 +454,21 @@ def phrase_hits(relative, body) end end + def uncited_posts + posts = blog_source_files + + assert posts.any?, "No blog source found - this gate would pass by finding nothing." + + posts.filter_map do |path| + body = File.read(path, encoding: "bom|utf-8").split(/^---\s*$/m)[2].to_s + next if body.split.size < CITATION_WORD_FLOOR + + external = body.scan(%r{\]\((https?://[^)]+)\)}).flatten + .reject { |url| url.include?("jetthoughts.com") } + path.sub("#{REPO_ROOT}/", "") if external.empty? + end + end + # Frontmatter lives in the first few lines; bound the read so 689 posts stay cheap. def blog_source_files Dir.glob(File.join(REPO_ROOT, "content/blog/**/*.md")).reject do |path| From 0b5ee2f7c6d218814dd62099bf19a3c3337d9c20 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 15:23:47 +0200 Subject: [PATCH 5/6] Partial checking feels like checking: my own correction was stale too Paul asked whether I could clear my own blocker with research instead of waiting. Doing that surfaced a better defect than the one I was researching. The Laravel notice I wrote 20 minutes ago was itself wrong. It said: do not land on 11, go to 12, same PHP 8.2 floor. I had queried context7 for Laravel 11 and 12 and stopped there. laravel.com's actual support table has four rows: 10 | PHP 8.1-8.3 | bug fixes to 2024-08-06 | security to 2025-02-04 (EOL) 11 | PHP 8.2-8.4 | bug fixes to 2025-09-03 | security to 2026-03-12 (EOL) 12 | PHP 8.2-8.5 | bug fixes to 2026-08-13 | security to 2027-02-24 13 | PHP 8.3-8.5 | bug fixes to Q3 2027 | security to 2028-03-17 Laravel 12 left bug-fix support NINE DAYS AGO. Laravel 13 has been current since March and needs PHP 8.3, so "8.2 is enough" would have misled anyone taking the sensible path. The check ran, felt complete, and stopped two rows short - after I had written the rule about verifying claims, in the same session. That is a harder failure than skipping the check, because it produces the same feeling of having been careful. It went into the post as its own section ("Partial checking feels exactly like checking"), and it is the stronger of the two examples, because it happened AFTER the rule existed. Notice rewritten with the full table, PHP 8.3 called out, Laravel 13 named as the target, and a pointer to the official upgrade guide for the remaining legs. The structural content stays valid: 11 rewrote the skeleton and 12 and 13 inherit it, so this is still the largest step from 10. Rewritten freely per Paul's "without caring who have read". Also tightened the post's opening so no skimmer can conclude we SHIPPED the wrong Propshaft claim. Verified in git: the bad sentence exists in exactly two commits on one feature branch, introduced and removed before merge. `git log origin/master -S` returns empty. It never reached a reader. Fixed one banned word I introduced myself ("journey") in the new notice. Gates: marketing_copy_test 5 runs / 13 assertions / 0 failures. bin/hugo-build green, both pages render 200. All EIGHT sections of the post still pass the cadence quotas. 1,352 -> 1,531 words. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg --- .../index.md | 8 +++++++- .../index.md | 14 +++++++++++++- 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/content/blog/laravel-11-migration-guide-production-deployment-strategies/index.md b/content/blog/laravel-11-migration-guide-production-deployment-strategies/index.md index 27c8158e2..6b0de2e3a 100644 --- a/content/blog/laravel-11-migration-guide-production-deployment-strategies/index.md +++ b/content/blog/laravel-11-migration-guide-production-deployment-strategies/index.md @@ -11,7 +11,13 @@ metatags: image: cover.png --- -> **Read this first (updated 2026-08-22): Laravel 11 is out of support.** Per Laravel's own [version support table](https://laravel.com/docs/releases), active support for 11.x ended 2025-09-03 and security fixes ended 2026-03-12. If you are still on Laravel 10, **11 is no longer the version to land on** - Laravel 12 has the same PHP 8.2 minimum and is supported into 2027, so the 10 → 12 path costs you the same PHP upgrade and does not leave you on an unpatched release. Everything below about the structural changes still applies, because 12 inherits 11's skeleton. Treat it as the 10 → 11 half of a 10 → 12 move. +> **Read this first (updated 2026-08-22). Do not land on Laravel 11 - target 13.** +> +> Both ends of this guide are now out of support. Per Laravel's own [support policy table](https://laravel.com/docs/13.x/releases): Laravel 10 security fixes ended **2025-02-04**, and Laravel 11 security fixes ended **2026-03-12**. Laravel 12 stopped receiving bug fixes on **2026-08-13** and is on security-only until February 2027. +> +> **Laravel 13 is the current release** (released 2026-03-17, bug fixes into Q3 2027, security to 2028). It needs **PHP 8.3 or higher** - a step past the 8.2 this guide assumes, so budget for that. Laravel's own release notes call 13 "a relatively minor upgrade in terms of effort" and say "most Laravel applications may upgrade to Laravel 13 without changing much application code." +> +> **The structural work below is still the work.** Laravel 11 rewrote the application skeleton - no `Kernel.php`, no default service providers, no `config/app.php` aliases - and 12 and 13 inherit that skeleton. Coming from Laravel 10, this is the hard leg, and 11 → 12 → 13 on top of it is comparatively light. Read this as the first and largest step, then follow the [official upgrade guide](https://laravel.com/docs/13.x/upgrade) for the remainder. Use PHP 8.3+ throughout rather than the 8.2 targeted below. ## TL;DR diff --git a/content/blog/what-senior-developers-catch-that-ai-misses/index.md b/content/blog/what-senior-developers-catch-that-ai-misses/index.md index 217549a7e..01e717a81 100644 --- a/content/blog/what-senior-developers-catch-that-ai-misses/index.md +++ b/content/blog/what-senior-developers-catch-that-ai-misses/index.md @@ -12,7 +12,7 @@ canonical_url: 'https://jetthoughts.com/blog/what-senior-developers-catch-that-a related_posts: false --- -The sentence looked fine. +The sentence looked fine, and it sat in an open pull request waiting to be merged. An agent had just pulled an unsourced performance number out of one of our older Rails posts and written a replacement in its place. Propshaft, it said, "drops the transpilation and concatenation stages entirely, so asset precompilation stops being a build step that scales with your asset count." @@ -70,6 +70,18 @@ METR is careful about what that does not show, so I will be too: early-2025 mode What travels is the gap itself. Being experienced did not make the self-assessment accurate, which is the whole problem with judging this by feel. +## Partial checking feels exactly like checking + +Here is the same failure again, from later the same day, after the rule above had been written down. + +We found an old Laravel migration guide with no citations at all and went to verify it. The framework's support table said Laravel 11 had been out of security support since March. So a correction went on the page: do not land on 11, go to 12, same PHP floor. + +Two of those three facts were right. + +Laravel 12 had stopped getting bug fixes nine days earlier, and Laravel 13 - current since March, and the sensible target - requires PHP 8.3 rather than the 8.2 the correction promised. The table had four rows and two of them got read. + +Nobody skipped the check. The check got done, felt done, and stopped one row short of the answer. That is a harder failure to design against than not checking at all, because it produces the same feeling of having been careful. + ## What this means if you are the one paying for it You are not buying keystrokes any more. That part got cheap, and pretending otherwise is how founders end up overpaying for output they could have generated themselves. From ee42b79d1a1bf8e0b02eb6885ae2bf7a31d86c80 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 15:37:43 +0200 Subject: [PATCH 6/6] Content PRs stop waiting for CI - and inherit the one gate that matters Paul extended the docs-only no-wait rule to blog content (2026-08-22). Written into every instruction surface that carries the merge rule: CLAUDE.md, AGENTS.md, the blog-write skill, blog-pipeline.md STEP 7, and .okf/build/ci-gates.md. The two cases are NOT symmetric, and writing the rule without that would have dropped a real gate. Docs PRs merge without waiting because NO CI fires on them - the path filters exclude them. Blog content is different: `content/**` is a trigger path for link-check.yml, and internal links are precisely what a new post adds. "No CI wait" for content therefore means the link crawl MOVED, not that it vanished. So the rule carries an obligation instead of a blanket exemption: run `bin/rake test:links` locally in place of the wait. It is the same task the CI job runs, over a production build. Skipping both is allowed only when stated out loud, because `push: branches: [master]` still crawls after merge - which converts a pre-merge catch into a fix-forward defect rather than eliminating it. The misreading to guard against is "content-only has no link gate". Named explicitly in ci-gates.md so the next reader cannot arrive at it. Ran it on this PR rather than describing it: 1,768 pages scanned, 31,888 unique links, 114,239 OK, ZERO errors. Gates: marketing_copy_test 5 runs / 13 assertions / 0 failures. bin/hugo-build green. bin/rake test:links clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg --- .claude/skills/blog-write/SKILL.md | 10 ++++++++++ .okf/build/ci-gates.md | 17 +++++++++++++++++ AGENTS.md | 2 +- CLAUDE.md | 2 +- docs/workflows/blog-pipeline.md | 6 ++++++ 5 files changed, 35 insertions(+), 2 deletions(-) diff --git a/.claude/skills/blog-write/SKILL.md b/.claude/skills/blog-write/SKILL.md index 2b952ce5c..8f02db95e 100644 --- a/.claude/skills/blog-write/SKILL.md +++ b/.claude/skills/blog-write/SKILL.md @@ -129,6 +129,16 @@ criteria scored, and the scores written into the commit message. Feature branch, commit, `gh pr create` with the evidence. Never push to master. +**Do NOT wait for CI on a content-only PR** (Paul 2026-08-22). Prose and +frontmatter cannot move the app build, so local gates decide and you merge as +soon as they are green. + +One thing does not come free: `content/**` triggers the link checker, and +internal links are exactly what a post adds. Run **`bin/rake test:links`** +locally in place of the wait. If you skip it, say so in the handback - the +master push-run still crawls, so a broken link becomes a fix-forward defect +rather than a caught one. + **End the handback with the local review link** - `http://localhost:/blog//`. One dev server per session, never 1313: diff --git a/.okf/build/ci-gates.md b/.okf/build/ci-gates.md index 747095ad3..fe6236fb6 100644 --- a/.okf/build/ci-gates.md +++ b/.okf/build/ci-gates.md @@ -198,6 +198,23 @@ A CSS/test/docs-only PR does not trigger it. Non-PR events carry no `paths` filt Caveat: with native `paths`, a filtered-out PR reports NO check (not a passing one). Fine unless `link_check` becomes a *required* status check - then add a skip-reporting companion job. +## Content PRs do not wait for CI - but they DO owe the link check (2026-08-22) + +Docs/instruction/tooling PRs merge on local gates because no CI fires on them. +Blog CONTENT was folded into the same no-wait rule by Paul on 2026-08-22, and +the two cases are NOT symmetric: `content/**` is a trigger path above, so a +content PR does fire this workflow, and internal links are precisely what a new +post adds. + +The rule therefore carries an obligation rather than a blanket exemption: run +**`bin/rake test:links`** locally in place of the wait - it is the same task the +job runs, over a production build. Skipping both is allowed only when stated out +loud, because the `push: branches: [master]` trigger still crawls after merge, +which converts a pre-merge catch into a fix-forward defect. + +Reading "content-only merges on local gates" as "content-only has no link gate" +is the misreading to guard against; the gate moved from CI to your terminal. + # Visual regression is NOT a CI gate today - but the historical blocker is gone A CI screenshot job (`quick_test` + `bin/qtest`) was built and removed in PR #386. At the time the divergence was unfixable: baselines were captured on the then-Alpine/musl docker image while CI runs Ubuntu (glibc), and text rendered differently enough that measured divergence ran **3-28%** (mobile code blocks 0.28, plain content pages up to 0.21) - far above any tolerance that still catches a real regression. diff --git a/AGENTS.md b/AGENTS.md index 95b735393..ebcd92902 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,7 +5,7 @@ Session start: always read `@docs/workflows/flow-router.md` (rules live in `CLAU **Async-first communication (SOP for ALL agents):** written, discoverable artifacts are the default for every decision, finding, status change, and handoff — sync interaction is the exception and its outcome gets written back same-day. Full rules + canonical surfaces table: `.agents/skills/async-first-communication/SKILL.md`. A task is not done until its state is readable asynchronously. **Cold-start surface: `STATUS.md` at repo root** (goals + WIP + blockers, links only) — a session that changed what's in flight updates it in the same commit/PR, or the session is not done. -**Four eyes on every stage, author != verifier (SOP for ALL agents):** the point is agents collaborating to find the TRUTH — a single agent cannot find its own blind spot, because it checks the thing it MEANT to build. Every change is **written by one sub-agent and verified by a different one**; the author never produces the evidence for its own claim. The review lands before the artifact leaves the workshop: a plan is peer-reviewed BEFORE the user is asked, a diff BEFORE commit, a finding is reproduced before it is reported, a measurement re-derived before it is quoted. Brief reviewers with the goal and the artifact, never your conclusions; ask for measurements, not verdicts. Routing: internal sub-agents for every per-stage review, `/codex:review` ONCE at the final verify before merge for USER-FACING changes (it is slow — never in the inner loop); docs-only and instruction-layer changes ship on internal review + local gates — no CI fires on them (path filters) and none is waited for (Paul 2026-08-22); findings apply as a follow-up. No reviewer tooling in this runtime? Fall back agent → external → peer session → human, and mark the change UNREVIEWED if none is reachable. Full protocol: the `jt-delivery:contract` skill (plugin `jt-delivery@jetthoughts`); repo bindings: `docs/workflows/autonomous-delivery-prompt.md`. +**Four eyes on every stage, author != verifier (SOP for ALL agents):** the point is agents collaborating to find the TRUTH — a single agent cannot find its own blind spot, because it checks the thing it MEANT to build. Every change is **written by one sub-agent and verified by a different one**; the author never produces the evidence for its own claim. The review lands before the artifact leaves the workshop: a plan is peer-reviewed BEFORE the user is asked, a diff BEFORE commit, a finding is reproduced before it is reported, a measurement re-derived before it is quoted. Brief reviewers with the goal and the artifact, never your conclusions; ask for measurements, not verdicts. Routing: internal sub-agents for every per-stage review, `/codex:review` ONCE at the final verify before merge for USER-FACING changes (it is slow — never in the inner loop); docs-only and instruction-layer changes ship on internal review + local gates — no CI fires on them (path filters) and none is waited for (Paul 2026-08-22); findings apply as a follow-up. **Blog CONTENT merges the same way** (Paul 2026-08-22) — but unlike docs, `content/**` DOES fire the link checker, so run `bin/rake test:links` locally in place of the wait, or state that you did not. No reviewer tooling in this runtime? Fall back agent → external → peer session → human, and mark the change UNREVIEWED if none is reachable. Full protocol: the `jt-delivery:contract` skill (plugin `jt-delivery@jetthoughts`); repo bindings: `docs/workflows/autonomous-delivery-prompt.md`. **Session memory (ruflo-first, SOP for ALL agents):** search ruflo memory (`mcp__ruflo__memory_search`, MCP path) before non-obvious decisions; store decisions/corrections via `mcp__ruflo__memory_store` as they land. Details + prefix conventions: `CLAUDE.md` §Memory. diff --git a/CLAUDE.md b/CLAUDE.md index 7c53bf87f..e0fd293cd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -97,7 +97,7 @@ Operational knowledge lives in `.okf/` (`build/`, `content/`, `design/`, `workfl - **Session-random dev-server port (Paul 2026-08-17)**: `PORT=$((20000 + RANDOM % 20000)) bin/dev`, once per session; never assume 1313; never kill a server you didn't start. - **Reflection trigger**: Paul reports "code is bad" / "over-engineered" → HALT, 5-Why, fix config, then proceed. - **Self-critique on voice-sensitive content (BLOCKING)**: any LinkedIn/blog/marketing draft runs `reflexion-reflect` BEFORE first handback; after pushback on the same draft, escalate to `reflexion-critique`. Solo iteration misses pattern-level tells (5 corrections on a 0/10-scored post, 2026-05-11). -- **Content-only changes SKIP the visual suites (Paul 2026-07-31)**: markdown prose/frontmatter only (no themes/layouts/CSS/body HTML) → `bin/hugo-build` + rendered scroll gate; NOT qtest/test/dtest. The moment the diff touches a template/stylesheet/body HTML, the visual gate applies in full — check the actual diff, not the intent. Docs/instruction-only PRs also skip the app-build CI (publish.yml paths-ignore, 2026-08-21). **Docs/instruction/tooling-only PRs merge on local gates — do NOT wait for CI** (Paul 2026-08-22, on #575): this includes non-build scripts (e.g. `bin/agent-bootstrap`) that cannot change the built site; CI polling before merge is only for diffs that can alter the app build. +- **Content-only changes SKIP the visual suites (Paul 2026-07-31)**: markdown prose/frontmatter only (no themes/layouts/CSS/body HTML) → `bin/hugo-build` + rendered scroll gate; NOT qtest/test/dtest. The moment the diff touches a template/stylesheet/body HTML, the visual gate applies in full — check the actual diff, not the intent. Docs/instruction-only PRs also skip the app-build CI (publish.yml paths-ignore, 2026-08-21). **Docs/instruction/tooling-only PRs merge on local gates — do NOT wait for CI** (Paul 2026-08-22, on #575): this includes non-build scripts (e.g. `bin/agent-bootstrap`) that cannot change the built site; CI polling before merge is only for diffs that can alter the app build. **Extended to BLOG CONTENT (Paul 2026-08-22)**: a prose/frontmatter-only diff under `content/` merges on local gates too — no CI wait. One caveat that is not optional: `content/**` DOES trigger the link checker (`.github/workflows/link-check.yml` paths), and internal links are exactly what a post adds, so run **`bin/rake test:links`** locally before merging instead. If you skip it, say so — the master push-run still crawls, and a broken internal link is then a fix-forward defect rather than a caught one. - **Visual regression gate (BLOCKING for HTML/CSS/template changes)**: `bin/qtest --changed` before EVERY commit; the full `bin/test` + `bin/dtest` pair ONCE at PR prep — a PR must never open without the dtest leg (CI is Linux). Failures are commit blockers: fix, or confirm intentional AND update both `macos/` + `linux/` baselines in the same commit. Baselines compare against git HEAD, not the working tree — identical difference_level across runs means a stale committed baseline (`md5 -q` the file to confirm), not flaky rendering. Full procedure incl. Linux re-record: `.okf/build/test-gates.md`. (Evidence: the 2026-04-30 35-failure incident — 15 commits with no visual gate; the suite caught a real `.fl-button` regression.) - **Feature-branch + PR for ALL changes (BLOCKING — Paul 2026-08-17)**: never push to master, any change type. Branch → related commits → gates green → `gh pr create` (body: summary + per-commit description + visual evidence). **One sprint = ONE PR** (Paul 2026-04-30), **capped at ~500 changed lines of CODE** (Paul 2026-08-21; docs, `.okf/` logs, binaries/baselines don't count — docs-only may batch bigger); an oversized code sprint splits into sequential PRs, merge N before opening N+1. `gh pr merge --auto` does NOT queue here — it merges immediately; poll `gh pr checks` and merge without it. **Master moved → REBASE, never merge it in** (Paul 2026-08-20): tag a backup ref, `git rebase origin/master` (`--autostash` if unstaged changes block; never a bare `git stash`), `git push --force-with-lease`. Branch cut from a since-squash-merged branch replays its own merged commits — don't fight it: `git reset --hard origin/master` + cherry-pick your commits; always cut from `origin/master`. `.okf/log.md` conflicts on essentially every parallel PR — resolution is near-always keep-both. diff --git a/docs/workflows/blog-pipeline.md b/docs/workflows/blog-pipeline.md index 8e2565cfc..94710f5af 100644 --- a/docs/workflows/blog-pipeline.md +++ b/docs/workflows/blog-pipeline.md @@ -476,6 +476,12 @@ STEP 7 — VALIDATE - og:image resolves - Cover image renders correctly - All internal links work +- `bin/rake test:links` — run it LOCALLY on a content PR (Paul 2026-08-22). + Content-only PRs merge on local gates and do not wait for CI, but `content/**` + is one of the link checker's trigger paths and internal links are exactly what + a post adds. This command is what you are skipping the wait for. Skipping it + too is allowed only if you say so — the master push-run still crawls, and the + broken link is then found after merge instead of before. STEP 8 — UPDATE CONTENT PLAN Edit docs/projects/2510-seo-content-strategy/20-29-strategy/20.09-content-plan-revision-aug-2026.md: