Repository navigation
Incorrect handling of UTF-8 encoding during preprocessing #44
Description
Example:
cppreference-doc-20250209\reference\en.cppreference.com\w\cpp\header\bit.html:4 which is the page title
- the raw dump has it correctly:
<title>Standard library header <bit> (C++20) - cppreference.com</title>
it's just UTF-8 encoded
- the html in the zip (also in the .tar.xz) has it twice encoded:
html-book-20250209.zip\reference\en\cpp\header\bit.html
<title>Standard library header <bit>Â (C++20) - cppreference.com</title>
adding \u00C2 kruft
I'd be happy to look into it.
Activity
[-]script mangles &nbps; (a.k.a. \u00A0) and I assume any char > `\0x7F` that's beyond the ASCII table[/-][+]script mangles &nbps; (a.k.a. \u00A0) and I assume any char > \0x7F that's beyond the ASCII table[/+][-]script mangles &nbps; (a.k.a. \u00A0) and I assume any char > \0x7F that's beyond the ASCII table[/-][+]Incorrect handling of UTF-8 encoding during preprocessing[/+]PeterFeicht commentedon Apr 25, 2025
Yep, looks like a bug. I'm assuming we need to explicitly set the source encoding somewhere during preprocessing.
refack commentedon Apr 25, 2025
cppreference-doc/commands/preprocess.py
Line 389 in be3ce3c
HTMLParser(encoding='utf-8')I'll post a PR shortly.
I got sucked into a slightly bigger change using BeautifulSoup which makes things simpler and more robust. Strangely MediaWiki renders wonky HTML mixing up end tags order e.g.
<div><ul><li></li></div></ul>. But that should be a different PR and a whole new conversation.