How can the benchmarks be run/reproduced? / Literally the entire "benchmark suite" is missing.

[DianaNites]

The provided "benchmark suite" link seems to contain exclusively generic code with absolutely nothing for configuration/setup/actually running either zerofs or the competing project being benchmarked. The entire benchmark suite is missing.

[DianaNites]

Well the lack of response answers all questions I had about the project and my investigation into using it over other similar ones. Conclusion: Do not use, sadly. It sounded interesting on paper, but theres no way to validate any of its claims and the author is seemingly hostile to any attempt to do so.

[Barre]

Worth separating the two things here.

The ask itself (instructions for reproducing the benchmarks) is reasonable, and I'll improve that. Happy to take a PR if anyone wants to help.

Concluding that I'm "hostile to attempts to validate the project" because I didn't reply to a discussion within three weeks? That's strange.

[DianaNites]

Since initially opening the issue(which was silently closed and transferred to an unacknowledged discussion), I had done more research and seen your comments on social media across places like hacker news, specifically your responses to other people asking about the benchmarks and the lack of their actual existence. People have been asking for much longer than I have.

You are not slick.

[Barre]

The benchmark code is in bench/ and is what I run to produce the numbers. If you ran it and hit specific issues, or wildly different numbers, I’m happy to look at those. Beyond that I don’t have more to add here.

[DianaNites]

See thats the same gaslighting you always do. There is no benchmark code in bench/, there is a tool/framework that could be used to run benchmarks, but you do not actually include any benchmarks. The tool or framework to run benchmarks is not a benchmark itself. There is no code in bench/ that can be run as-is to output a result of any sort.

A benchmark is the specific code, specific versions, specific configuration, ideally specific runtime environment, that produces a result, because all of these significantly affect results and are thus absolutely necessary to understanding the results and especially comparing them over time and reproducing them.

You do not include any benchmarks.

I'm not going to bother arguing with you anymore, you are a bad faith actor and its really sad because ZeroFS sounds really promising on paper and I really wanted to use it, but it fundamentally cannot be trusted with such a dishonest maintainer.

[Barre]

cd bench && cargo run --release.

That’s the benchmark.

If you ran it and got different numbers, post them. If you didn’t run it, then the claim that no benchmarks exist is one you’re making without having checked, or even bothered to try to reproduce, which is a strange thing to escalate over three rounds.