Repository navigation
Conversation
…C++ reference ClickHouseCityHash.hashLen0to16() widened the three tail bytes a, b, c from signed Java bytes, while cityhash102 (the implementation ClickHouse uses to validate compressed blocks) reads them as uint8 and computes y and z as uint32. Any tail byte >= 0x80 therefore produced a checksum the server could not reproduce. The branch is reached when the hashed length is 17..19 bytes (the first 16 bytes go into the seed), i.e. for a compressed block whose LZ4 payload is 8..10 bytes: the forced flush of a 7..9-byte Data packet remainder. When such a tail carries a high-bit byte the server rejects the whole INSERT with "Checksum doesn't match: corrupted data. ... Size of compressed block: 18". It is rare (about 3 in 2^20 inserts larger than 1 MB) and looks like random corruption; with a DateTime as the last inserted column it also depends on the calendar, because byte 2 of the current epoch crossed 0x80 on 2026-08-15. ClickHouseCityHashTest pins the port against 906 vectors produced by the C++ reference (contrib/cityhash102 from the ClickHouse repository): every length 0..300 with random, high-bit-only and ASCII-only buffers, plus the three 17..19-byte frames of a real incident. Without the fix the test fails exactly for lengths 1, 2, 3, 17, 18, 19 and never for ASCII input.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ClickHouseCityHash.hashLen0to16()handles the 1..3-byte tail with signedbytearithmetic:The reference implementation that ClickHouse uses to validate compressed blocks (
contrib/cityhash102) readsa,b,casuint8and computesy,zasuint32. Any tail byte>= 0x80sign-extends intoy/zand the 128-bit hash diverges from what the server computes.The first 16 bytes of the hashed data become the seed, so the branch is reached only when the compressed block is 17, 18 or 19 bytes long — an LZ4 payload of 8..10 bytes, which is the forced flush of a 7..9-byte remainder of a
Datapacket after the 1 MB frames. When such a tail has a high-bit byte, the server rejects the INSERT:It happens on roughly 3 in 2^20 INSERTs larger than 1 MB, so it looks like random network or memory corruption. With a
DateTimeas the last inserted column it even depends on the calendar: byte 2 of the current epoch crossed0x80on 2026-08-15, which is when we started seeing it after years of silence. The read path (CompressedBuffedReader) does not validate checksums, so the defect is visible only on INSERT.Fix
Mask the three bytes with
& 0xFFand computey,zas unsigned 32-bit values, matching cityhash102. No other behaviour changes; for inputs without high-bit tail bytes the output is identical to before.Test
ClickHouseCityHashTestpins the Java port against 906 vectors produced by the C++ reference implementation from the ClickHouse repository: every length 0..300 with random, high-bit-only and ASCII-only buffers, plus the three 17..19-byte frames of a real incident. Without the fix the test fails exactly for lengths 1, 2, 3, 17, 18, 19 and never for ASCII input; with the fix all 906 vectors match. (Lengths 1..3 are unreachable through the driver because a compressed block is at least 9 bytes; 17..19 are the ones that bite.)End-to-end reproduction against a real server (one
Stringrow whose serialized block is 1 MB + 7/8/9 bytes with0xC3 0xA9at the end fails with block sizes 17/18/19, the same lengths with ASCII pass) and the full investigation are written up here: https://github.com/SerpstatGlobal/ClickHouse-Native-JDBC/blob/serpstat/2.7.1/docs/STORY.mdNote: verified locally with
-Djacoco.skip=true; the jacoco agent configured in the currentpom.xmlaborts the surefire fork on Java 21, which is unrelated to this change. The overstatedmaxOutputLengthinCompressedBuffedWriter(#469) is a separate issue and not part of this PR.