Object storage performance impact on operations

[solracsf]

Hi. kudos for v1 stable release ❤️

One could imagine that, like any other software, the better the underlying systems are, the better the software performs, basic rule.

Yet, let me ask this: Amazon S3 delivers much better performance compared to another S3-compatible services 99% of the times. But the cost is not the same either...

For not-so-intensive operations, is a hdd-backed S3 service (Hetzner, OVHcloud, Scaleway...) enough for ZeroFS to perform correctly? Same applies for service latency.

In which cases should we go for a performant, yet expensive, S3 service?

Again, thanks for ZeroFS 👍🏻

[Barre]

Hi,

Thanks for the kind words.

Any S3-compatible object store that supports conditional writes as well as read-your-writes consistency should work with ZeroFS. This includes, but is not limited to, Hetzner, Cloudflare R2, Wasabi...

In which cases should we go for a performant, yet expensive, S3 service?

This shouldn't really matter that much for most use cases, as ZeroFS is pretty good at hiding latency. But your mileage may vary depending on the use case.

[solracsf]

Thanks.

JuiceFS has a compatibility benchmark (juicefs objbench) that can be run to easily detect compatibility (but they also support a lot of storage backends beside S3...) and performance; do you believe a similar CLI tool could be useful on ZeroFS ?

[Barre]

Not really, because ZeroFS already checks for conditional-writes support at startup. ZeroFS cannot reliably check for read-your-writes, as even if the object store is eventually consistent and doesn't provide read-your-writes, it could provide consistency fast enough for ZeroFS to pass the test but catastrophically fail during heavier use.

[solracsf]

If conditional writes is mandatory, shouldn't this be specified in the README?
For now it just says Works with any S3-compatible storage while not true if conditional writes not supported (many don't, like MEGA or OVHCLOUD).

[Barre]

For now it just says Works with any S3-compatible storage while not true if conditional writes not supported

Conditional writes are part of the core S3 spec. Providers that don't support them aren't fully S3-compatible, regardless of how they market themselves.