fix(dev-server): return 404 status code for non-existent pages

[cnaples79]

Summary

Fixes #11095 - Dev server now returns proper 404 status code for non-existent pages instead of 200.

Problem

The dev server's historyApiFallback configuration was serving the SPA for all routes with HTTP 200 status, even for non-existent pages. This made debugging difficult and caused issues when browsers attempted to load non-existent CSS/JS files (receiving HTML content instead).

Solution

Added middleware (docusaurus-404-status) that:

• Intercepts HTML responses from the dev server
• Detects when the 404 page is being served by checking for common indicators
• Sets the correct 404 status code and message

Detection Strategy

The middleware identifies 404 pages using multiple patterns:

1. Default English text: "Page Not Found"
2. Translated versions: "404" + hero__title class (from NotFound component structure)
3. Case-insensitive fallbacks: "not found" or "page not found"

This approach supports:

• ✅ Default English NotFound component
• ✅ All 40+ official Docusaurus translations
• ✅ Custom 404 pages following common patterns

Performance Considerations

• Only intercepts GET requests that accept HTML
• Skips all static assets (js, css, images, fonts)
• Minimal memory overhead (buffers only HTML responses)
• Early exit for non-relevant requests

Limitations

Custom 404 pages that completely remove all "404" and "not found" text may not be detected. Users customizing their NotFound component should ensure it includes recognizable 404 indicators.

Test Plan

Tested manually:

• Access non-existent route (e.g., /foo/bar/baz) returns 404
• Existing routes continue to return 200
• Static assets return appropriate status codes
• Translated sites return 404 for non-existent pages

Related

• Issue: 404 page in dev server returns 200 status code instead of 404  #11095
• Related discussion: Similar issue where dev server returns incorrect status codes

---

[slorber]

This code is too fragile, complex, unreliable, untested to solve a minor issue so I'm not really willing to merge this, at least not in its current state.

Is this AI-generated?

[cnaples79]

@slorber No, it is not.

Do you have any specific feedback on a better implementation so that I can work on an update?

[slorber]

Sorry, with Hacktoberfest and your PR formatting, it's not easy to distinguish PRs that have been created by AI from those created by a real human.

---

I unfortunately don't know how to fix it, but the fix should be reliable and not something that only works for English, avoid hardcoding a list of asset file extensions, and avoid monkey-patching the Response object.

You say the problem comes from the usage of historyApiFallback:

    historyApiFallback: {
      rewrites: [{from: /\/*/, to: baseUrl}],
    },

Does it serve 404 without that configuration?

If that's the case, maybe we should look there and try to provide a better configuration.

• Maybe not using any rewrite if baseUrl: '/'?
• Maybe only rewriting if the initial URL doesn't start with the baseUrl?

[cnaples79]

@slorber no worries.

I have an idea instead of using the html inspecting middlewre, I may try a smaller helper that specifically checks incoming request paths. I'll add unit tests as well. Will update this soon!