Skip to content

json: vajson: buffer files in memory before parsing - #431

Open
fbaeuerle wants to merge 1 commit into
eclipse-score:mainfrom
fbaeuerle:fb/json_fromfile_perf
Open

json: vajson: buffer files in memory before parsing#431
fbaeuerle wants to merge 1 commit into
eclipse-score:mainfrom
fbaeuerle:fb/json_fromfile_perf

Conversation

@fbaeuerle

@fbaeuerle fbaeuerle commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

FromFile parsed directly off the filebuf. The reader's Snap/Restore backtracking does seekg/tellg, which triggers lseek/read syscalls on a filebuf and made FromFile (especially on QNX) significantly slower than in-memory FromBuffer.

Read the file into a std::string once and parse it via std::istringstream, as FromBuffer already does: seeks become O(1) in-memory pointer moves.

Runtime improvement (measured on x86_64 linux) is around 25%:
Before:
BM_RealisticConfig_FromFile/512 30748530 ns 30684328 ns 22 bytes_per_second=1.52507Mi/s
After:
BM_RealisticConfig_FromFile/512 22465309 ns 22427569 ns 30 bytes_per_second=2.08653Mi/s

@github-project-automation github-project-automation Bot moved this to In Progress in BAS - Baselibs FT Jul 30, 2026
@fbaeuerle
fbaeuerle requested a deployment to workflow-approval July 30, 2026 13:00 — with GitHub Actions Waiting
@fbaeuerle
fbaeuerle requested a deployment to workflow-approval July 30, 2026 13:00 — with GitHub Actions Waiting
@github-actions github-actions Bot added comp-json Related to score/json component c++ C++ code labels Jul 30, 2026
@github-actions

Copy link
Copy Markdown

The created documentation from the pull request is available at: docu-html

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

File reads are not validated for I/O errors and the file is opened without binary mode, which can lead to incorrect behavior on read failures and inconsistent byte handling.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR optimizes vaJson file parsing by buffering file contents into memory before parsing, avoiding expensive seekg/tellg backtracking operations on file-backed stream buffers (notably improving performance on QNX).

Changes:

  • Replace direct parsing from a file stream with a one-time read into std::string and parsing via std::istringstream.
  • Adjust includes to support iterator-based buffering.
File summaries
File Description
score/json/internal/parser/vajson/vajson_impl/reader/json_data.cpp Buffer JSON file contents in memory before parsing to speed up Snap/Restore backtracking.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 2
  • Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Comment thread score/json/internal/parser/vajson/vajson_impl/reader/json_data.cpp
Comment thread score/json/internal/parser/vajson/vajson_impl/reader/json_data.cpp
@fbaeuerle
fbaeuerle force-pushed the fb/json_fromfile_perf branch from 6a278bf to bd7747e Compare July 30, 2026 14:16
@fbaeuerle
fbaeuerle requested a deployment to workflow-approval July 30, 2026 14:16 — with GitHub Actions Waiting
@fbaeuerle
fbaeuerle requested a deployment to workflow-approval July 30, 2026 14:16 — with GitHub Actions Waiting
FromFile parsed directly off the filebuf. The reader's Snap/Restore
backtracking does seekg/tellg, which triggers lseek/read syscalls on a
filebuf and made FromFile (especially on QNX) significantly slower than
in-memory FromBuffer.

Read the file into a std::string once and parse it via std::istringstream,
as FromBuffer already does: seeks become O(1) in-memory pointer moves.

Runtime improvement (measured on x86_64 linux) is around 25%:
Before:
BM_RealisticConfig_FromFile/512      30748530 ns     30684328 ns           22 bytes_per_second=1.52507Mi/s
After:
BM_RealisticConfig_FromFile/512      22465309 ns     22427569 ns           30 bytes_per_second=2.08653Mi/s
@fbaeuerle
fbaeuerle force-pushed the fb/json_fromfile_perf branch from bd7747e to f9fa567 Compare July 30, 2026 14:34
@fbaeuerle
fbaeuerle temporarily deployed to workflow-approval July 30, 2026 14:34 — with GitHub Actions Inactive
@fbaeuerle
fbaeuerle temporarily deployed to workflow-approval July 30, 2026 14:34 — with GitHub Actions Inactive
@4og
4og requested a review from Copilot July 30, 2026 15:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The new file-read path needs bounds-checked casting between std::streamoff, std::size_t, and std::streamsize to avoid overflow/narrowing bugs on large files.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Comments suppressed due to low confidence (2)

score/json/internal/parser/vajson/vajson_impl/reader/json_data.cpp:97

  • FromFile now returns kStreamFailure not only when the file can't be opened but also when size/reading fails. The API documentation in json_data.h currently only mentions the open-failure case; please update it so callers know reads can fail too (and under what conditions).
            return MakeErrorResult<JsonData>(JsonErrc::kStreamFailure, "Could not read file");

score/json/internal/parser/vajson/vajson_impl/reader/json_data.cpp:105

  • size is a std::streamoff and is passed directly to std::istream::read, which expects a std::streamsize. For large files this can narrow/overflow and lead to incorrect reads or huge/undefined allocations when casting to std::size_t. Add explicit bounds checks and cast to std::streamsize before calling read().
        file.seekg(0, std::ios::end);
        const std::streamoff size{file.tellg()};
        file.seekg(0, std::ios::beg);
        if (file.fail() || (size < 0))
        {
  • Files reviewed: 1/1 changed files
  • Comments generated: 0 new
  • Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c++ C++ code comp-json Related to score/json component

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

2 participants