Skip to content

Fix perf regression in Read::read_to_end on short reads due to not checking if the cursor has initialized bytes #158083

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
asder8215 wants to merge 1 commit into
base: main
Choose a base branch
from
+17 −2
Open

Fix perf regression in Read::read_to_end on short reads due to not checking if the cursor has initialized bytes #158083

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 17 additions & 2 deletions library/std/src/io/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,8 @@ pub(crate) fn default_read_to_end<R: Read + ?Sized>(
.and_then(|s| s.checked_add(1024)?.checked_next_multiple_of(DEFAULT_BUF_SIZE))
.unwrap_or(DEFAULT_BUF_SIZE);

let mut initialized_len = 0;

const PROBE_SIZE: usize = 32;

fn small_probe_read<R: Read + ?Sized>(r: &mut R, buf: &mut Vec<u8>) -> Result<usize> {
Expand All @@ -446,6 +448,8 @@ pub(crate) fn default_read_to_end<R: Read + ?Sized>(

if read == 0 {
return Ok(0);
} else {
initialized_len = buf.capacity() - buf.len();
}
}

Expand All @@ -459,6 +463,8 @@ pub(crate) fn default_read_to_end<R: Read + ?Sized>(

if read == 0 {
return Ok(buf.len() - start_len);
} else {
initialized_len = buf.capacity() - buf.len();
}
}

Expand All @@ -467,13 +473,17 @@ pub(crate) fn default_read_to_end<R: Read + ?Sized>(
buf.try_reserve(PROBE_SIZE)?;
}

let buf_unfilled_len = buf.capacity() - buf.len();
let mut spare = buf.spare_capacity_mut();
let buf_len = cmp::min(spare.len(), max_read_size);
spare = &mut spare[..buf_len];
let mut read_buf: BorrowedBuf<'_, u8> = spare.into();

// Note that we don't track already initialized bytes here, but this is fine
// because we explicitly limit the read size
if initialized_len == buf_unfilled_len {

@a1phyr a1phyr Jun 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition is wrong: you compare the old buffer capacity and the new buffer spare capacity, but the start of the buffer has changed since then. It would be less error prone to track initialized bytes counting from the beginning of the Vec.

@Darksonn Darksonn Jun 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, please store the full capacity of the vector, including anything already written.

// SAFETY: These bytes were initialized but not filled in the previous loop
unsafe { read_buf.set_init() };
}

let mut cursor = read_buf.unfilled();
let result = loop {
match r.read_buf(cursor.reborrow()) {
Expand All @@ -484,9 +494,14 @@ pub(crate) fn default_read_to_end<R: Read + ?Sized>(
}
};

let cursor_bytes_unfilled = cursor.capacity();
let bytes_read = cursor.written();
let is_init = read_buf.is_init();

if is_init {
initialized_len = cursor_bytes_unfilled;
}

// SAFETY: BorrowedBuf's invariants mean this much memory is initialized.
unsafe {
let new_len = bytes_read + buf.len();
Expand Down
Loading