Skip to content

Make Blob Building Incremental #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
itzmeanjan merged 49 commits into from
Jul 25, 2025
Merged

Make Blob Building Incremental #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
6d96f7c
Incrementally updatable blob-builder struct
itzmeanjan Jul 16, 2025
f03db61
Add API comments for BlobBuilder
itzmeanjan Jul 16, 2025
62e92a8
Update how chunkset-id is computed
itzmeanjan Jul 16, 2025
e4d8738
Update tests to work with new BlobBuilder API
itzmeanjan Jul 16, 2025
b7fd629
Update benchmarks to work with new BlobBuilder API
itzmeanjan Jul 16, 2025
7598b34
Update doc-test in lib.rs
itzmeanjan Jul 16, 2025
65dae42
Add new dependency for figuring out number of logical CPUs
itzmeanjan Jul 16, 2025
30af22c
Update `decds` CLI tool to use new BlobBuilder incremental API
itzmeanjan Jul 16, 2025
b3977f1
Trap failure of any bash commands in test script
itzmeanjan Jul 16, 2025
0712cd9
Blob-builder `update` method can now exploit rayon-based data-paralle…
itzmeanjan Jul 17, 2025
f7bbe24
Bump `rlnc` to latest version
itzmeanjan Jul 21, 2025
e08db87
Put independent components of Blob Builder (CLI command `break`) into…
itzmeanjan Jul 21, 2025
7a4299e
Address issue in building file-path when finalizing erasure-coded chunks
itzmeanjan Jul 21, 2025
cd779a9
Make inter-thread communication channel async
itzmeanjan Jul 21, 2025
1290b68
Update test-runner shell script to compile with platform-specific opt…
itzmeanjan Jul 21, 2025
738eb01
Change buffer sizes - it demonstrates better performance
itzmeanjan Jul 21, 2025
da70e18
Update decds CLI test script to work with 1GB random file
itzmeanjan Jul 21, 2025
0098b9b
Add `tokio` - async runtime engine as a dependency of the project
itzmeanjan Jul 23, 2025
ac63ae3
Rewrite logic for breaking arbitrary-size blob data into proof-carryi…
itzmeanjan Jul 23, 2025
412a933
Refactor code-block into its own function
itzmeanjan Jul 23, 2025
9eaccb3
Add new Makefile recipe for ease of building executable
itzmeanjan Jul 23, 2025
50cf066
Update custom Cargo profile `optimized` with aggressive optimization …
itzmeanjan Jul 23, 2025
6173e89
Show progress of chunkify process with updating text
itzmeanjan Jul 23, 2025
e26faae
Add progress text in `break` command output
itzmeanjan Jul 23, 2025
3eb6260
Each tokio task computes blob-level Merkle inclusion proof for itself
itzmeanjan Jul 23, 2025
e1a2c70
Simplify script which tests `decds` CLI
itzmeanjan Jul 23, 2025
27064f2
Remove unused optimization flag declaration from BASH script
itzmeanjan Jul 23, 2025
9c6be1c
Detailed log for kind of error, that was encountered during execution…
itzmeanjan Jul 23, 2025
c1955b4
Use `console` crate to lookup terminal size
itzmeanjan Jul 23, 2025
6deb550
Check for errors when adding chunk to repairing chunkset
itzmeanjan Jul 23, 2025
ae3972d
Make bash script for generic so that it runs on both Linux and Mac
itzmeanjan Jul 23, 2025
e3c0807
Update github actions script to run renamed BASH script
itzmeanjan Jul 23, 2025
563eb07
Do not use `-p` option with `time` command
itzmeanjan Jul 23, 2025
7eaf413
Update Blob repairing tests to not ignore return value of `add_chunk`
itzmeanjan Jul 23, 2025
34c7e7a
Use `JoinSet` to spawn a lot of worker tasks
itzmeanjan Jul 24, 2025
4118929
For communicating in between blob-reader and blob-builder tasks use M…
itzmeanjan Jul 24, 2025
13ac327
Human readable byte presentation function now displays upto two place…
itzmeanjan Jul 25, 2025
d906e0e
For communication in between blob-builder and partial-chunk writer ta…
itzmeanjan Jul 25, 2025
c6af16d
Update github actions CI script to check that `decds` works with MSRV
itzmeanjan Jul 25, 2025
4f31178
Mention `make build` recipe to avoid system-wide installation of `decds`
itzmeanjan Jul 25, 2025
e563d26
Update code-coverage report and add instrutions to uninstall `decds`
itzmeanjan Jul 25, 2025
132b9ec
Implement Error and Debug traits for custom error types
itzmeanjan Jul 25, 2025
fcfead5
Update example in lib.rs
itzmeanjan Jul 25, 2025
1a443f5
Mention script on test `decds` CLI
itzmeanjan Jul 25, 2025
cfa813d
`decds` CLI version command now shows git commit from which it was bu…
itzmeanjan Jul 25, 2025
4fd5260
Update bash script to show which version of `decds` CLI it is using
itzmeanjan Jul 25, 2025
294ebb4
Bump version to 0.2.0 - ready for release
itzmeanjan Jul 25, 2025
83a3aa4
Bump dependency `rlnc` to latest version
itzmeanjan Jul 25, 2025
59494c5
Do not use features from https://github.com/rust-lang/rust/issues/536…
itzmeanjan Jul 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions .github/workflows/test_ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,22 @@ jobs:

windows-latest, # x86_64
windows-11-arm, # arm64
]
]
rust: [
1.85.0, # MSRV of this project
stable
]
max-parallel: 4

steps:
- uses: actions/checkout@v4
- uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: ${{ matrix.rust }}

- name: Build and Test on ${{ matrix.os }}
- name: Build and Test on ${{ matrix.os }}, with Rust ${{ matrix.rust }}
run: make test

- name: Test decds CLI on Linux/Mac
- name: Test `decds` CLI on ${{ matrix.os }}, with Rust ${{ matrix.rust }}
if: runner.os == 'Linux' || runner.os == 'macOS'
run: bash scripts/test_decds_on_linux.sh
run: bash scripts/test_decds_on_unix.sh
Loading
Loading