Tune IO performance

[prime000000]

This patch adds a couple of tunable, which I found useful in our buildbarn setup.

We have 10 bare metal hosts managed by k8s. Each host has two 8TiB NVME SSD linked in software mirror raid. These SSDs support write speed of about 2.5Gb/s. We deploy buildbarn with memory limit set to 64Gi (rest is used for CI) and locate 1TiB of disk space for it, sharded over 10 files.

1. I think it always make sense to call fallocate, when creating the /blocks file. ext4 filesystem on linux default to lazy allocation, and if we create 100Tb file, by default 0 blocks are allocated upfront. Instead, allocation code is running during Pwrite, or even during writeback. Causing all sort of problems with fragmentation and lock contention inside the kernel.

2. I do not think mmap is always useful for file read, when file does not fit into memory (which is our case). Pread should give more predictable performance, since OS sees full range of requested data in a single syscall. Instead of second guessing how much data it needs to prefetch from the page-fault pattern.

3. It makes sense to sync data to disk after Pwrite. NVME is fast enough, so there is no degradation in performance. But disk write is issued synchronously from buildbarn thread. Without this, dirty pages are piled up in memory, and write to disk is started at random point in time in the future, when process tries allocating new memory, hits container memory limit and starts kernel memory reclaim.

With all three options applied I was able to get full underlying NVME write speed in synthetic benchmarks with consistent timings.

Without these options, buildbarn died under our CI load, when we tries switching to buildbarn from our current bazel-remote setup.

I did not have enough time to do a clear before/after experiment in our CI setup, but all my experience of working with storages tells me that these changes are worthwhile.