Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions content/blog/how-to-audit-content-you-didnt-write/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ It does not exist.

The interesting part is who the reports were written for. Piro's founder said it on LinkedIn: "When someone asks ChatGPT, Gemini, or Perplexity about your category, an answer comes back in one confident paragraph... we spent months reverse-engineering it."

Not readers. The machine that answers readers.
Those reports were never aimed at readers. Their audience was the machine that answers readers, and the reports were shaped to be the thing it repeats.

## Your blog runs on the same economics

Expand All @@ -46,25 +46,25 @@ You would think there is a number. There are several and they disagree.

Ten percent, a third, or half. The numbers are not contradicting each other so much as answering different questions, on different samples, with different detectors.

Both teams publish their error bars, which is the part worth copying. Graphite reports a 4.2% false-positive rate against GPT-4o content and no measurement at all of AI-assisted-then-human-edited writing. Pew says plainly that "AI detection models aren't perfect" on individual documents and only hold up in aggregate.
Both teams publish their error bars, and that habit is worth stealing regardless of what you make of their numbers. Graphite reports a 4.2% false-positive rate against GPT-4o content and no measurement at all of AI-assisted-then-human-edited writing. Pew says plainly that "AI detection models aren't perfect" on individual documents and only hold up in aggregate.

One more number from Pew is the one that should interest you: **around one in ten `.com` pages carry those signals - roughly double the `.org` rate and ten times what `.edu` and `.gov` show.** Commercial sites are where this concentrates, and yours is a commercial site.

So the honest position is that the web's average is unknown and the detectors that estimate it are themselves approximate. Which is fine, because the average was never the thing you needed. **You need to know about your property, and your property is countable.**

## Four checks, and the commands that run them

We ran these on our own archive, in this order, ranked by what it costs you to skip each one.
Run them in this order. It is ranked by what each one costs you to skip, not by how quick it is.

The commands assume a Hugo or Jekyll-shaped repo where posts are markdown files. Adapt the paths; the shapes they look for are the same everywhere. If you do not have repo access, these are exactly the four things to ask whoever does.

**1. Rank by who reads it, not by how bad it looks.**

Export your top pages from Search Console, then work down that list and nothing else.

We got this wrong first. The worst-sounding claim we found sat on a page flagged `featured` in the site config, which felt urgent, and it turned out to have four impressions in ninety days while the page that actually mattered had thousands.
The instinct is to start with whatever reads worst. Resist it - the page that makes you wince and the page that gets read are rarely the same one, and a `featured` flag in your site config tells you what someone decided once, not what anyone has done since.

`featured` is a flag someone set once. Impressions are what readers did.
Impressions are what readers actually did.

```bash
# Export "Pages" from Search Console as CSV, then rank what you actually have:
Expand Down Expand Up @@ -93,11 +93,11 @@ Run the first one and read every hit. Real client work names the client or does

**3. Ask whether a claim can be checked at all.**

This one surprised us.
Count how many of your long posts link to nothing outside your own domain.

Roughly two in five of our own substantial posts cited nothing external whatsoever - no link to a framework's documentation, a study, a release note, anything at all. Those posts are not necessarily wrong.
A post with no external citation is not necessarily wrong. It is unverifiable, which means nobody could have checked it - including whoever wrote it, at the moment they wrote it.

They are unverifiable, which means nobody could have checked them, including the person who wrote them. Uncheckable is where wrong survives.
Uncheckable is where wrong survives, and the number usually comes back higher than anyone guesses.

Count yours:

Expand All @@ -116,9 +116,9 @@ A post making technical claims with zero citations is not a red flag about that

Any post with a version number in the title has a shelf life its author never wrote down.

We found a migration guide sending real traffic to a framework release whose security support had ended five months earlier. Nothing in it was invented.
A migration guide that recommends Laravel 11 today is sending readers onto a release whose security support ended in March 2026. Nothing in that guide has to be invented for it to do damage.

It was true when written and became harmful without changing a word.
It was true when written, and became harmful without a word of it changing.

```bash
# every post whose title names a version - each one has an expiry date
Expand Down
38 changes: 20 additions & 18 deletions content/blog/what-senior-developers-catch-that-ai-misses/index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "What Senior Devs Catch That AI Misses"
description: "An AI agent replaced a made-up number in our own blog post with a confident, wrong explanation. What caught it is the thing you are actually hiring for."
description: "A change swapped a made-up number for a confident, wrong explanation of how Propshaft works. Spotting the difference is the thing you are actually hiring for."
date: 2026-08-22
draft: false
author: 'JetThoughts Team'
Expand All @@ -16,7 +16,7 @@ canonical_url: 'https://jetthoughts.com/blog/what-senior-developers-catch-that-a
related_posts: false
---

Here is a diff that sat in an open pull request, waiting to be merged.
Here is a diff we stopped in review. It is small, it is plausible, and it is wrong in a way you cannot see without knowing Rails.

```diff
- Propshaft is dramatically faster than Sprockets: precompilation drops from
Expand All @@ -25,21 +25,21 @@ Here is a diff that sat in an open pull request, waiting to be merged.
+ precompilation stops being a build step that scales with your asset count.
```

An agent had pulled an unsourced timing figure out of one of our older Rails posts and written that replacement in its place. The removal was correct. Nobody had ever measured those 45-60 seconds.
The deletion is correct. That timing figure had no measurement behind it and deserved to go.

Read the addition again if you know Rails. It is wrong.
Read the addition again. It is wrong.

Propshaft still walks every asset, fingerprints it, and copies it into place. Its own README says so:

> All assets in the load path will be copied (or compiled) in a precompilation step for production that also stamps all of them with a digest hash

The work scales with how many assets you have. What drops is the cost of each one, because transpiling and bundling are gone - which is a real and useful thing to say, and not what the sentence said.

The agent had removed a made-up number and replaced it with a made-up mechanism. That is worse, because a mechanism reads as reasoning rather than as a claim someone should go and check.
So a made-up number was swapped for a made-up mechanism. That is the worse trade, because a mechanism reads as reasoning rather than as a claim someone should go and check - and reasoning gets waved through where a number gets questioned.

## Nobody skimming that paragraph would have stopped

That is the part worth sitting with.
Sit with that for a second.

The sentence had a subject, a cause and an effect, and it used the right vocabulary throughout. It also agreed with the general direction of the truth - Propshaft *is* faster - while getting the reason for it wrong.

Expand All @@ -53,15 +53,17 @@ Two weeks later Senko Rašić's ["'Code was never the hard part' is an insult to

Both threads circle the same question and neither settles it. If the model writes the code, what is the person for?

## The answer our own mistake gave
## What the diff answers

Not typing - the agent typed fine.
It was not the typing.

The prose came out clean, grammatical and confident on the first pass, and would have survived any editor who did not happen to know how Rails compiles assets.

What it could not do was notice that its own explanation was false. It had no way to check, because checking meant knowing something about the Rails asset pipeline that was not in the sentence it had just written.

Goedecke calls the thing experts do "steering" - you recognise a suboptimal suggestion and redirect it. Our incident is the same mechanism running backwards. Nobody steered, so a confident wrong answer went straight into a draft, inside a change whose entire purpose was removing unreliable claims.
Goedecke calls the thing experts do "steering" - you recognise a suboptimal suggestion and redirect it. This diff is that mechanism running backwards: without someone who knows the asset pipeline, there is nothing to steer against, and the confident answer wins by default.

That last part is the uncomfortable bit. The task was *clean up unsourced numbers*, and the agent did the task and introduced a new defect in the same motion.
Note what the change was *for*. The task was removing an unsourced number, and the same edit introduced a new defect while completing it. Cleanup is where this happens most, because a correction feels like tidying rather than authorship, and gets read that way.

## What actually caught it

Expand Down Expand Up @@ -89,19 +91,19 @@ Going in, those developers expected a 24% speedup. Coming out, having just lived

METR is careful about what that does not show, so I will be too: early-2025 models, Claude 3.5 and 3.7 Sonnet through Cursor Pro, on mature codebases with demanding quality standards, and they explicitly decline to claim it generalises to most developers or to later tools.

What travels is the gap itself. Being experienced did not make the self-assessment accurate, which is the whole problem with judging this by feel.
What travels is the gap itself. Experience did not make those developers better at estimating their own speed. It made them more certain about an estimate that was forty points off.

## Partial checking feels exactly like checking

Here is the same failure again, from later the same day, after the rule above had been written down.

We found an old Laravel migration guide with no citations at all and went to verify it. The framework's support table said Laravel 11 had been out of security support since March. So a correction went on the page: do not land on 11, go to 12, same PHP floor.
Take a Laravel migration guide that recommends upgrading to 11. Checking it against Laravel's support table shows 11 left security support in March, so the obvious correction is: do not land on 11, go to 12, same PHP floor.

Two of those three facts were right.
Two of those three facts are right.

Laravel 12 had stopped getting bug fixes nine days earlier, and Laravel 13 - current since March, and the sensible target - requires PHP 8.3 rather than the 8.2 the correction promised. The table had four rows and two of them got read.
Laravel 12 stopped getting bug fixes on 13 August 2026. Laravel 13 has been current since March and requires PHP 8.3, not the 8.2 that correction promises. The table has four rows, and stopping after two produces a fix that is still wrong.

Nobody skipped the check. The check got done, felt done, and stopped one row short of the answer. That is a harder failure to design against than not checking at all, because it produces the same feeling of having been careful.
Nobody skipped the check there. The check ran, felt complete, and stopped one row short. That is harder to design against than not checking at all, because it produces the identical feeling of having been careful.

## What this means if you are the one paying for it

Expand All @@ -121,11 +123,11 @@ We wrote about the [team structure that makes this hold up](/blog/claude-code-xp

## The uncomfortable version

Our agent produced a wrong claim while cleaning up wrong claims. We caught it because we had built the habit of pointing a second, adversarial pass at anything an agent wrote, and because someone on the other end knew Rails well enough to referee.
Every defect in this post was caught the same way: a second pass whose brief was to disagree, followed by someone who knew the subject well enough to referee the disagreement.

If we had not, that sentence would be live right now, sounding authoritative, on a post about Rails performance.
Neither half works alone. The reviewer that only agrees is decoration, and the reviewer that objects to something nobody can adjudicate is noise.

That is the whole argument for expertise, and it is not a comfortable one. The value is not in what gets produced. It is in the small number of moments where someone looks at fluent output and says no.
Expertise earns its money in a handful of moments per week, and none of them look like productivity. Someone reads a paragraph that scans perfectly and says no, and cannot always explain why until they go and check.

## Sources

Expand Down
12 changes: 6 additions & 6 deletions content/blog/when-did-a-test-last-fail-on-purpose/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Production emits absolute URLs. Your `/blog/foo/` becomes `https://jetthoughts.c

So the crawler saw a page full of absolute links, classified all of them as "not my problem", and reported success on what remained - which on our homepage was a single skip-link anchor.

The fix is `--remap`, pointing the public origin back at the built tree:
Adding `--remap` points the public origin back at the built tree:

```ruby
# Rakefile
Expand All @@ -60,7 +60,7 @@ From 15,642 links checked to 114,239.

## Run this on your own repo before you keep reading

The diagnostic is the same whatever you use.
Whatever checker you run, the diagnostic is the same.

**Compare what your checker says it inspected against how many links your built site actually contains.**

Expand Down Expand Up @@ -90,9 +90,9 @@ The link checker was found by accident, which was the uncomfortable part. So we

Three of eight were caught.

The predictions mattered more than the score. Writing "the banned-phrase ratchet will catch this" before planting it turns a vague sense of coverage into a falsifiable claim - and two of those claims were wrong in a specific way.
Those written-down predictions mattered more than the score. Writing "the banned-phrase ratchet will catch this" before planting it turns a vague sense of coverage into a falsifiable claim - and two of those claims were wrong in a specific way.

The ratchet was carrying slack. It was set to fail above 14 hits when the tree actually had 11, so a planted phrase landed in the gap and the suite stayed green. A ratchet with three spare slots does not guard the last three defects.
One ratchet was carrying slack. It was set to fail above 14 hits when the tree actually had 11, so a planted phrase landed in the gap and the suite stayed green. A ratchet with three spare slots does not guard the last three defects.

```ruby
# The fix is boring: set the baseline to the MEASURED count,
Expand All @@ -118,7 +118,7 @@ Run from a git worktree, it lost its reference images and wrote fresh captures o

Someone finally tested the tester: injected `body { background: red !important }`, confirmed the rule reached the built CSS bundle, confirmed the page referenced that fingerprinted file, then measured the captured PNG against the baseline.

Candidate `[255,0,0]`. Baseline `[255,255,255]`. Difference level **0.68**.
Candidate `[255,0,0]` against a baseline of `[255,255,255]`, for a difference level of **0.68**.

The run reported `0 failures`.

Expand All @@ -141,7 +141,7 @@ If your CI output cannot distinguish "inspected everything and found nothing" fr

## What we do now, and what it costs

A new test is not finished until someone has broken the thing it guards and watched it fail. Not a flaky failure - a deliberate one.
A new test is not finished until someone has broken the thing it guards and watched it fail. Flaky failures do not count; this has to be deliberate, and someone has to be watching when it goes red.

That adds maybe two minutes to writing a test.

Expand Down
Loading