Skip to content

Fix CityHash128 for 1..3-byte tails: read bytes as unsigned like the C++ reference - #472

Open
vitos73 wants to merge 1 commit into
housepower:masterfrom
vitos73:fix/cityhash128-signed-tail-bytes
Open

vitos73 wants to merge 1 commit into
housepower:masterfrom
vitos73:fix/cityhash128-signed-tail-bytes

Conversation

@vitos73

@vitos73 vitos73 commented Sep 9, 2026

Copy link
Copy Markdown

Problem

ClickHouseCityHash.hashLen0to16() handles the 1..3-byte tail with signed byte arithmetic:

byte a = s[pos]; byte b = s[pos + (len >>> 1)]; byte c = s[pos + len - 1];
int y = (int) a + (((int) b) << 8);
int z = len + (((int) c) << 2);

The reference implementation that ClickHouse uses to validate compressed blocks (contrib/cityhash102) reads a, b, c as uint8 and computes y, z as uint32. Any tail byte >= 0x80 sign-extends into y/z and the 128-bit hash diverges from what the server computes.

The first 16 bytes of the hashed data become the seed, so the branch is reached only when the compressed block is 17, 18 or 19 bytes long — an LZ4 payload of 8..10 bytes, which is the forced flush of a 7..9-byte remainder of a Data packet after the 1 MB frames. When such a tail has a high-bit byte, the server rejects the INSERT:

DB::Exception: Checksum doesn't match: corrupted data. Reference: ..., Actual: ...
Size of compressed block: 18.

It happens on roughly 3 in 2^20 INSERTs larger than 1 MB, so it looks like random network or memory corruption. With a DateTime as the last inserted column it even depends on the calendar: byte 2 of the current epoch crossed 0x80 on 2026-08-15, which is when we started seeing it after years of silence. The read path (CompressedBuffedReader) does not validate checksums, so the defect is visible only on INSERT.

Fix

Mask the three bytes with & 0xFF and compute y, z as unsigned 32-bit values, matching cityhash102. No other behaviour changes; for inputs without high-bit tail bytes the output is identical to before.

Test

ClickHouseCityHashTest pins the Java port against 906 vectors produced by the C++ reference implementation from the ClickHouse repository: every length 0..300 with random, high-bit-only and ASCII-only buffers, plus the three 17..19-byte frames of a real incident. Without the fix the test fails exactly for lengths 1, 2, 3, 17, 18, 19 and never for ASCII input; with the fix all 906 vectors match. (Lengths 1..3 are unreachable through the driver because a compressed block is at least 9 bytes; 17..19 are the ones that bite.)

End-to-end reproduction against a real server (one String row whose serialized block is 1 MB + 7/8/9 bytes with 0xC3 0xA9 at the end fails with block sizes 17/18/19, the same lengths with ASCII pass) and the full investigation are written up here: https://github.com/SerpstatGlobal/ClickHouse-Native-JDBC/blob/serpstat/2.7.1/docs/STORY.md

Note: verified locally with -Djacoco.skip=true; the jacoco agent configured in the current pom.xml aborts the surefire fork on Java 21, which is unrelated to this change. The overstated maxOutputLength in CompressedBuffedWriter (#469) is a separate issue and not part of this PR.

…C++ reference

ClickHouseCityHash.hashLen0to16() widened the three tail bytes a, b, c from
signed Java bytes, while cityhash102 (the implementation ClickHouse uses to
validate compressed blocks) reads them as uint8 and computes y and z as
uint32. Any tail byte >= 0x80 therefore produced a checksum the server could
not reproduce.

The branch is reached when the hashed length is 17..19 bytes (the first 16
bytes go into the seed), i.e. for a compressed block whose LZ4 payload is
8..10 bytes: the forced flush of a 7..9-byte Data packet remainder. When such a
tail carries a high-bit byte the server rejects the whole INSERT with
"Checksum doesn't match: corrupted data. ... Size of compressed block: 18".
It is rare (about 3 in 2^20 inserts larger than 1 MB) and looks like random
corruption; with a DateTime as the last inserted column it also depends on the
calendar, because byte 2 of the current epoch crossed 0x80 on 2026-08-15.

ClickHouseCityHashTest pins the port against 906 vectors produced by the C++
reference (contrib/cityhash102 from the ClickHouse repository): every length
0..300 with random, high-bit-only and ASCII-only buffers, plus the three
17..19-byte frames of a real incident. Without the fix the test fails exactly
for lengths 1, 2, 3, 17, 18, 19 and never for ASCII input.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant