diff --git a/content/blog/how-to-audit-content-you-didnt-write/index.md b/content/blog/how-to-audit-content-you-didnt-write/index.md index 0007a1908..fb076a71b 100644 --- a/content/blog/how-to-audit-content-you-didnt-write/index.md +++ b/content/blog/how-to-audit-content-you-didnt-write/index.md @@ -24,7 +24,7 @@ It does not exist. The interesting part is who the reports were written for. Piro's founder said it on LinkedIn: "When someone asks ChatGPT, Gemini, or Perplexity about your category, an answer comes back in one confident paragraph... we spent months reverse-engineering it." -Not readers. The machine that answers readers. +Those reports were never aimed at readers. Their audience was the machine that answers readers, and the reports were shaped to be the thing it repeats. ## Your blog runs on the same economics @@ -46,7 +46,7 @@ You would think there is a number. There are several and they disagree. Ten percent, a third, or half. The numbers are not contradicting each other so much as answering different questions, on different samples, with different detectors. -Both teams publish their error bars, which is the part worth copying. Graphite reports a 4.2% false-positive rate against GPT-4o content and no measurement at all of AI-assisted-then-human-edited writing. Pew says plainly that "AI detection models aren't perfect" on individual documents and only hold up in aggregate. +Both teams publish their error bars, and that habit is worth stealing regardless of what you make of their numbers. Graphite reports a 4.2% false-positive rate against GPT-4o content and no measurement at all of AI-assisted-then-human-edited writing. Pew says plainly that "AI detection models aren't perfect" on individual documents and only hold up in aggregate. One more number from Pew is the one that should interest you: **around one in ten `.com` pages carry those signals - roughly double the `.org` rate and ten times what `.edu` and `.gov` show.** Commercial sites are where this concentrates, and yours is a commercial site. @@ -54,7 +54,7 @@ So the honest position is that the web's average is unknown and the detectors th ## Four checks, and the commands that run them -We ran these on our own archive, in this order, ranked by what it costs you to skip each one. +Run them in this order. It is ranked by what each one costs you to skip, not by how quick it is. The commands assume a Hugo or Jekyll-shaped repo where posts are markdown files. Adapt the paths; the shapes they look for are the same everywhere. If you do not have repo access, these are exactly the four things to ask whoever does. @@ -62,9 +62,9 @@ The commands assume a Hugo or Jekyll-shaped repo where posts are markdown files. Export your top pages from Search Console, then work down that list and nothing else. -We got this wrong first. The worst-sounding claim we found sat on a page flagged `featured` in the site config, which felt urgent, and it turned out to have four impressions in ninety days while the page that actually mattered had thousands. +The instinct is to start with whatever reads worst. Resist it - the page that makes you wince and the page that gets read are rarely the same one, and a `featured` flag in your site config tells you what someone decided once, not what anyone has done since. -`featured` is a flag someone set once. Impressions are what readers did. +Impressions are what readers actually did. ```bash # Export "Pages" from Search Console as CSV, then rank what you actually have: @@ -93,11 +93,11 @@ Run the first one and read every hit. Real client work names the client or does **3. Ask whether a claim can be checked at all.** -This one surprised us. +Count how many of your long posts link to nothing outside your own domain. -Roughly two in five of our own substantial posts cited nothing external whatsoever - no link to a framework's documentation, a study, a release note, anything at all. Those posts are not necessarily wrong. +A post with no external citation is not necessarily wrong. It is unverifiable, which means nobody could have checked it - including whoever wrote it, at the moment they wrote it. -They are unverifiable, which means nobody could have checked them, including the person who wrote them. Uncheckable is where wrong survives. +Uncheckable is where wrong survives, and the number usually comes back higher than anyone guesses. Count yours: @@ -116,9 +116,9 @@ A post making technical claims with zero citations is not a red flag about that Any post with a version number in the title has a shelf life its author never wrote down. -We found a migration guide sending real traffic to a framework release whose security support had ended five months earlier. Nothing in it was invented. +A migration guide that recommends Laravel 11 today is sending readers onto a release whose security support ended in March 2026. Nothing in that guide has to be invented for it to do damage. -It was true when written and became harmful without changing a word. +It was true when written, and became harmful without a word of it changing. ```bash # every post whose title names a version - each one has an expiry date diff --git a/content/blog/what-senior-developers-catch-that-ai-misses/index.md b/content/blog/what-senior-developers-catch-that-ai-misses/index.md index dfaca6de3..65f31241f 100644 --- a/content/blog/what-senior-developers-catch-that-ai-misses/index.md +++ b/content/blog/what-senior-developers-catch-that-ai-misses/index.md @@ -1,6 +1,6 @@ --- title: "What Senior Devs Catch That AI Misses" -description: "An AI agent replaced a made-up number in our own blog post with a confident, wrong explanation. What caught it is the thing you are actually hiring for." +description: "A change swapped a made-up number for a confident, wrong explanation of how Propshaft works. Spotting the difference is the thing you are actually hiring for." date: 2026-08-22 draft: false author: 'JetThoughts Team' @@ -16,7 +16,7 @@ canonical_url: 'https://jetthoughts.com/blog/what-senior-developers-catch-that-a related_posts: false --- -Here is a diff that sat in an open pull request, waiting to be merged. +Here is a diff we stopped in review. It is small, it is plausible, and it is wrong in a way you cannot see without knowing Rails. ```diff - Propshaft is dramatically faster than Sprockets: precompilation drops from @@ -25,9 +25,9 @@ Here is a diff that sat in an open pull request, waiting to be merged. + precompilation stops being a build step that scales with your asset count. ``` -An agent had pulled an unsourced timing figure out of one of our older Rails posts and written that replacement in its place. The removal was correct. Nobody had ever measured those 45-60 seconds. +The deletion is correct. That timing figure had no measurement behind it and deserved to go. -Read the addition again if you know Rails. It is wrong. +Read the addition again. It is wrong. Propshaft still walks every asset, fingerprints it, and copies it into place. Its own README says so: @@ -35,11 +35,11 @@ Propshaft still walks every asset, fingerprints it, and copies it into place. It The work scales with how many assets you have. What drops is the cost of each one, because transpiling and bundling are gone - which is a real and useful thing to say, and not what the sentence said. -The agent had removed a made-up number and replaced it with a made-up mechanism. That is worse, because a mechanism reads as reasoning rather than as a claim someone should go and check. +So a made-up number was swapped for a made-up mechanism. That is the worse trade, because a mechanism reads as reasoning rather than as a claim someone should go and check - and reasoning gets waved through where a number gets questioned. ## Nobody skimming that paragraph would have stopped -That is the part worth sitting with. +Sit with that for a second. The sentence had a subject, a cause and an effect, and it used the right vocabulary throughout. It also agreed with the general direction of the truth - Propshaft *is* faster - while getting the reason for it wrong. @@ -53,15 +53,17 @@ Two weeks later Senko Rašić's ["'Code was never the hard part' is an insult to Both threads circle the same question and neither settles it. If the model writes the code, what is the person for? -## The answer our own mistake gave +## What the diff answers -Not typing - the agent typed fine. +It was not the typing. + +The prose came out clean, grammatical and confident on the first pass, and would have survived any editor who did not happen to know how Rails compiles assets. What it could not do was notice that its own explanation was false. It had no way to check, because checking meant knowing something about the Rails asset pipeline that was not in the sentence it had just written. -Goedecke calls the thing experts do "steering" - you recognise a suboptimal suggestion and redirect it. Our incident is the same mechanism running backwards. Nobody steered, so a confident wrong answer went straight into a draft, inside a change whose entire purpose was removing unreliable claims. +Goedecke calls the thing experts do "steering" - you recognise a suboptimal suggestion and redirect it. This diff is that mechanism running backwards: without someone who knows the asset pipeline, there is nothing to steer against, and the confident answer wins by default. -That last part is the uncomfortable bit. The task was *clean up unsourced numbers*, and the agent did the task and introduced a new defect in the same motion. +Note what the change was *for*. The task was removing an unsourced number, and the same edit introduced a new defect while completing it. Cleanup is where this happens most, because a correction feels like tidying rather than authorship, and gets read that way. ## What actually caught it @@ -89,19 +91,19 @@ Going in, those developers expected a 24% speedup. Coming out, having just lived METR is careful about what that does not show, so I will be too: early-2025 models, Claude 3.5 and 3.7 Sonnet through Cursor Pro, on mature codebases with demanding quality standards, and they explicitly decline to claim it generalises to most developers or to later tools. -What travels is the gap itself. Being experienced did not make the self-assessment accurate, which is the whole problem with judging this by feel. +What travels is the gap itself. Experience did not make those developers better at estimating their own speed. It made them more certain about an estimate that was forty points off. ## Partial checking feels exactly like checking Here is the same failure again, from later the same day, after the rule above had been written down. -We found an old Laravel migration guide with no citations at all and went to verify it. The framework's support table said Laravel 11 had been out of security support since March. So a correction went on the page: do not land on 11, go to 12, same PHP floor. +Take a Laravel migration guide that recommends upgrading to 11. Checking it against Laravel's support table shows 11 left security support in March, so the obvious correction is: do not land on 11, go to 12, same PHP floor. -Two of those three facts were right. +Two of those three facts are right. -Laravel 12 had stopped getting bug fixes nine days earlier, and Laravel 13 - current since March, and the sensible target - requires PHP 8.3 rather than the 8.2 the correction promised. The table had four rows and two of them got read. +Laravel 12 stopped getting bug fixes on 13 August 2026. Laravel 13 has been current since March and requires PHP 8.3, not the 8.2 that correction promises. The table has four rows, and stopping after two produces a fix that is still wrong. -Nobody skipped the check. The check got done, felt done, and stopped one row short of the answer. That is a harder failure to design against than not checking at all, because it produces the same feeling of having been careful. +Nobody skipped the check there. The check ran, felt complete, and stopped one row short. That is harder to design against than not checking at all, because it produces the identical feeling of having been careful. ## What this means if you are the one paying for it @@ -121,11 +123,11 @@ We wrote about the [team structure that makes this hold up](/blog/claude-code-xp ## The uncomfortable version -Our agent produced a wrong claim while cleaning up wrong claims. We caught it because we had built the habit of pointing a second, adversarial pass at anything an agent wrote, and because someone on the other end knew Rails well enough to referee. +Every defect in this post was caught the same way: a second pass whose brief was to disagree, followed by someone who knew the subject well enough to referee the disagreement. -If we had not, that sentence would be live right now, sounding authoritative, on a post about Rails performance. +Neither half works alone. The reviewer that only agrees is decoration, and the reviewer that objects to something nobody can adjudicate is noise. -That is the whole argument for expertise, and it is not a comfortable one. The value is not in what gets produced. It is in the small number of moments where someone looks at fluent output and says no. +Expertise earns its money in a handful of moments per week, and none of them look like productivity. Someone reads a paragraph that scans perfectly and says no, and cannot always explain why until they go and check. ## Sources diff --git a/content/blog/when-did-a-test-last-fail-on-purpose/index.md b/content/blog/when-did-a-test-last-fail-on-purpose/index.md index d2fbe6206..dca95b350 100644 --- a/content/blog/when-did-a-test-last-fail-on-purpose/index.md +++ b/content/blog/when-did-a-test-last-fail-on-purpose/index.md @@ -36,7 +36,7 @@ Production emits absolute URLs. Your `/blog/foo/` becomes `https://jetthoughts.c So the crawler saw a page full of absolute links, classified all of them as "not my problem", and reported success on what remained - which on our homepage was a single skip-link anchor. -The fix is `--remap`, pointing the public origin back at the built tree: +Adding `--remap` points the public origin back at the built tree: ```ruby # Rakefile @@ -60,7 +60,7 @@ From 15,642 links checked to 114,239. ## Run this on your own repo before you keep reading -The diagnostic is the same whatever you use. +Whatever checker you run, the diagnostic is the same. **Compare what your checker says it inspected against how many links your built site actually contains.** @@ -90,9 +90,9 @@ The link checker was found by accident, which was the uncomfortable part. So we Three of eight were caught. -The predictions mattered more than the score. Writing "the banned-phrase ratchet will catch this" before planting it turns a vague sense of coverage into a falsifiable claim - and two of those claims were wrong in a specific way. +Those written-down predictions mattered more than the score. Writing "the banned-phrase ratchet will catch this" before planting it turns a vague sense of coverage into a falsifiable claim - and two of those claims were wrong in a specific way. -The ratchet was carrying slack. It was set to fail above 14 hits when the tree actually had 11, so a planted phrase landed in the gap and the suite stayed green. A ratchet with three spare slots does not guard the last three defects. +One ratchet was carrying slack. It was set to fail above 14 hits when the tree actually had 11, so a planted phrase landed in the gap and the suite stayed green. A ratchet with three spare slots does not guard the last three defects. ```ruby # The fix is boring: set the baseline to the MEASURED count, @@ -118,7 +118,7 @@ Run from a git worktree, it lost its reference images and wrote fresh captures o Someone finally tested the tester: injected `body { background: red !important }`, confirmed the rule reached the built CSS bundle, confirmed the page referenced that fingerprinted file, then measured the captured PNG against the baseline. -Candidate `[255,0,0]`. Baseline `[255,255,255]`. Difference level **0.68**. +Candidate `[255,0,0]` against a baseline of `[255,255,255]`, for a difference level of **0.68**. The run reported `0 failures`. @@ -141,7 +141,7 @@ If your CI output cannot distinguish "inspected everything and found nothing" fr ## What we do now, and what it costs -A new test is not finished until someone has broken the thing it guards and watched it fail. Not a flaky failure - a deliberate one. +A new test is not finished until someone has broken the thing it guards and watched it fail. Flaky failures do not count; this has to be deliberate, and someone has to be watching when it goes red. That adds maybe two minutes to writing a test.