Update Rust crate fastcdc to v5 - #2710
Open
renovate[bot] wants to merge 1 commit into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains the following updates:
3.2.1→5.0.0Release Notes
nlfiedler/fastcdc-rs (fastcdc)
v5.0.0Compare Source
Breaking Changes
v2020now requires evenmin_size,avg_size, andmax_size. The"rolling two bytes each time" scan tests candidates in byte pairs starting
at
min_size / 2,avg_size / 2, andmax_size / 2; an odd valuetruncates when halved and silently shifts those boundaries by one byte
(issue #52). For example
min_size = 65let the scan return a chunk asshort as 64 bytes, violating the documented minimum.
FastCDC,StreamCDC, andAsyncStreamCDCnowdebug_assertthat all three sizesare even, as does the underlying
cut/cut_gearscan itself. Callersusing an odd size such as the
max_size = 65535from prior examples mustswitch to an even value (e.g.
65534); the built-in examples and doctestshave been updated accordingly. Release builds are unaffected by the
assertion (consistent with the existing
MINIMUM_MIN/AVERAGE_MIN/etc.range checks), but odd sizes remain unsupported either way. (#52)
Fixed
v2020forced/tail chunks could report a stale hash. The leftover atthe very end of a source is not required to be even (unlike
min_size/avg_size/max_size, a file's length isn't under the caller's control),and the scan never folded that trailing odd byte into the hash before
forcing a cut, so the returned fingerprint silently omitted the chunk's
last byte. The trailing byte is now folded into the hash (matching what a
byte-at-a-time scan would accumulate) without being tested as its own
boundary candidate. Cut points for even-sized parameters are unchanged;
only the hash of a chunk ending in a natural odd-length leftover changes.
(#52)
Iterator::size_hintupper bound could violate the trait contract. Inronomon,v2016, andv2020, a non-empty tail shorter thanmin_sizestill yields one final chunk, but
size_hintcomputed its upper bound asremaining / min_size, which floors to0in that case — understating theactual number of items left. Now uses
remaining.div_ceil(min_size)for theupper bound and reports a lower bound of
1while data remains. Chunkboundaries are unchanged. (#50)
v2020::AsyncStreamCDCand thev2020_cutexample could select differentmasks than
FastCDC/StreamCDCfor the sameavg_size. The syncconstructors round
avg_size.log2()to the nearest bit (so e.g.12288selects the same bucket as
16384), butAsyncStreamCDCand the exampleused
usize::ilog2, which floors instead (selecting the8192bucket).Both now go through a new shared
v2020::select_masksfunction, so allthree front-ends pick identical masks for identical arguments.
Boundary change: Only affects
AsyncStreamCDCoutput and thev2020_cutexample's output, and only for non-power-of-twoavg_sizevalues;
FastCDCandStreamCDCcut points are unchanged. (#51)Performance
v2020::cut_gearinner loop: restored array-typed GEAR lookups. The 4.0.0change from
&[u64; 256]to&[u64]reintroduced apanic_bounds_checkonevery GEAR table lookup in the hot scan (4 of them per loop iteration).
cut_gearnow converts the tables to&[u64; 256]once viatry_into, whichthe compiler can prove in-bounds for a
u8-derived index. Cut points andhashes are unchanged (the existing fixture tests pin them); emitted asm drops
from 8 to 4
panic_bounds_checksites incut_gear, and an interleaved A/Bmeasured ~7–14% throughput on random/text/zeros across chunk sizes (M1 Pro and
a dedicated-CPU x86 VM). The
&[u64]/Cowpublic signature is unchanged.Added
v2020::FastCDC::rechunk— re-points an existingFastCDCat a new sourceand resets iteration, reusing the already-computed normalization masks and gear
tables. The cheap way to chunk many in-memory buffers with identical parameters:
avoids recomputing masks and (for a non-zero seed) re-allocating the gear tables
on every
FastCDC::new. The iterator already yields each chunk's offset/lengthwithout copying, so callers needing the chunk bytes can slice the source. Cut
points are identical to a freshly constructed
FastCDC.v4.0.1Compare Source
Fixed
logarithm2()helper (roundedlog2) withusize::ilog2()(flooredlog2),which silently changed cut points for any
avg_sizethat is not a power oftwo. Power-of-two sizes were unaffected. Cut points now match 3.2.1 again.
v4.0.0Compare Source
Many changes suggested by Claude Code that seem worth making despite breaking the API. The changes needed are minor, just changing
u32tousizefor the common use case.Breaking Changes
Size parameter types changed from
u32tousizeacross all three modules (v2016,v2020,ronomon):new(),with_level(),with_level_and_seed()forFastCDC,StreamCDC, andAsyncStreamCDCMINIMUM_MIN,MINIMUM_MAX,AVERAGE_MIN,AVERAGE_MAX,MAXIMUM_MIN,MAXIMUM_MAXare nowusizeinstead ofu32cut_gear()gear parameters changed from&[u64; 256]to&[u64]get_gear_with_seed()return type changed from(Box<[u64; 256]>, Box<[u64; 256]>)to(Cow<'static, [u64]>, Cow<'static, [u64]>)Error::Displayoutput format changed (e.g."chunker error: Empty"→"no more data")assert!()todebug_assert!(), meaning invalid sizes will no longer panic in release buildsMinor Changes
MASKSconstant inv2016is nowpub(was private)Normalizationenum in bothv2016andv2020now derivesEqandPartialEqNormalization::bits()inv2016is nowpub(was private), and got a doc comment inv2020Bug Fixes & Performance Improvements
size_hint()corrected in all three iterators: the lower bound was incorrectly returning the upper bound; now returns1.min(upper_bound), which is semantically correctStreamCDCandAsyncStreamCDC: replaceddrain(..).collect()+resize()withextend_from_slice()+copy_within(), avoiding unnecessary reallocationget_gear_with_seed()optimized: whenseed == 0, the static GEAR tables are borrowed directly viaCow::Borrowedinstead of heap-allocating a copymask()inronomonchanged from2u32.pow(bits) - 1to(1u32 << bits) - 1(equivalent but avoids potential debug-mode panics on overflow)Configuration
📅 Schedule: (UTC)
🚦 Automerge: Enabled.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR was generated by Mend Renovate. View the repository job log.
This change is