json: vajson: buffer files in memory before parsing - #431
Conversation
|
The created documentation from the pull request is available at: docu-html |
There was a problem hiding this comment.
🟡 Not ready to approve
File reads are not validated for I/O errors and the file is opened without binary mode, which can lead to incorrect behavior on read failures and inconsistent byte handling.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
This PR optimizes vaJson file parsing by buffering file contents into memory before parsing, avoiding expensive seekg/tellg backtracking operations on file-backed stream buffers (notably improving performance on QNX).
Changes:
- Replace direct parsing from a file stream with a one-time read into
std::stringand parsing viastd::istringstream. - Adjust includes to support iterator-based buffering.
File summaries
| File | Description |
|---|---|
| score/json/internal/parser/vajson/vajson_impl/reader/json_data.cpp | Buffer JSON file contents in memory before parsing to speed up Snap/Restore backtracking. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 2
- Review effort level: Low
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
6a278bf to
bd7747e
Compare
FromFile parsed directly off the filebuf. The reader's Snap/Restore backtracking does seekg/tellg, which triggers lseek/read syscalls on a filebuf and made FromFile (especially on QNX) significantly slower than in-memory FromBuffer. Read the file into a std::string once and parse it via std::istringstream, as FromBuffer already does: seeks become O(1) in-memory pointer moves. Runtime improvement (measured on x86_64 linux) is around 25%: Before: BM_RealisticConfig_FromFile/512 30748530 ns 30684328 ns 22 bytes_per_second=1.52507Mi/s After: BM_RealisticConfig_FromFile/512 22465309 ns 22427569 ns 30 bytes_per_second=2.08653Mi/s
bd7747e to
f9fa567
Compare
There was a problem hiding this comment.
🟡 Not ready to approve
The new file-read path needs bounds-checked casting between std::streamoff, std::size_t, and std::streamsize to avoid overflow/narrowing bugs on large files.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Review details
Comments suppressed due to low confidence (2)
score/json/internal/parser/vajson/vajson_impl/reader/json_data.cpp:97
FromFilenow returnskStreamFailurenot only when the file can't be opened but also when size/reading fails. The API documentation in json_data.h currently only mentions the open-failure case; please update it so callers know reads can fail too (and under what conditions).
return MakeErrorResult<JsonData>(JsonErrc::kStreamFailure, "Could not read file");
score/json/internal/parser/vajson/vajson_impl/reader/json_data.cpp:105
sizeis astd::streamoffand is passed directly tostd::istream::read, which expects astd::streamsize. For large files this can narrow/overflow and lead to incorrect reads or huge/undefined allocations when casting tostd::size_t. Add explicit bounds checks and cast tostd::streamsizebefore callingread().
file.seekg(0, std::ios::end);
const std::streamoff size{file.tellg()};
file.seekg(0, std::ios::beg);
if (file.fail() || (size < 0))
{
- Files reviewed: 1/1 changed files
- Comments generated: 0 new
- Review effort level: Low
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
FromFile parsed directly off the filebuf. The reader's Snap/Restore backtracking does seekg/tellg, which triggers lseek/read syscalls on a filebuf and made FromFile (especially on QNX) significantly slower than in-memory FromBuffer.
Read the file into a std::string once and parse it via std::istringstream, as FromBuffer already does: seeks become O(1) in-memory pointer moves.
Runtime improvement (measured on x86_64 linux) is around 25%:
Before:
BM_RealisticConfig_FromFile/512 30748530 ns 30684328 ns 22 bytes_per_second=1.52507Mi/s
After:
BM_RealisticConfig_FromFile/512 22465309 ns 22427569 ns 30 bytes_per_second=2.08653Mi/s