Rocks DB example append flush vs immediate return requirement

[toth-istvan-zoltan]

Hi!

I'm trying to implement RaftLogStorage and I'm a bit confused about a requirement in the documentation and how the example works.

Version: 0.9.21

From the documentation of append:

Append log entries and call the callback once logs are persisted on disk.

It should returns immediately after saving the input log entries in memory, and calls the callback when the entries are persisted on disk, i.e., avoid blocking.

This method is still async because preparing the IO is usually async.

In the rocksstorage example the db WAL is flushed which I think would take a while, clashing with the documentation above. (I might be wrong, I'm new to Rust tbh).

self.db.flush_wal(true).map_err(|e| StorageError::write_logs(&e))?;

Also, the documentation states that:

All write-IO must be serialized, i.e., the internal implementation must NOT apply a latter write request before a former write request is completed. This rule applies to both vote and log IO. E.g., Saving a vote and appending a log entry must be serialized too.

I do not see anything like this in the rocksstorage example. If the flush operations are indeed synchronous, then this requirement might be met accidentally (if only one operation is called at a time), but that is not ensured by the storage itself.

Should the save_vote, append, truncate and purge be put into a queue and executed sequentially, one-by-one, in the order of the calls?

[drmingdrmer]

In the rocksstorage example the db WAL is flushed which I think would take a while, clashing with the documentation above. (I might be wrong, I'm new to Rust tbh).

the example is meant to provide a simple implementation of the RaftLogStorage and does not implement the two-phase append mentioned in the doc.

In a real world application, the append implementation does not need to flush_wal() before append() returns, flush_wal() can be delayed by putting it in a spawned task. And in this task, flush the WAL and call the callback.

I do not see anything like this in the rocksstorage example. If the flush operations are indeed synchronous, then this requirement might be met accidentally (if only one operation is called at a time), but that is not ensured by the storage itself.

Right. In this example there is no IO re-order and an application should not re-order any IO.

OpenRaft calls RaftLogStorage method in a serialized manner(no re-order) and the storage implementation just need to respect this calling order. A pitfall could be that in append() it spawn a task to hand the IO, then two spawn tasks can not guarantee the execution order and will finally result in an IO re-order. But append() implementation can put the input into a channel and the receiving end(maybe another task or thread) of the channel guarantees to process the inputs in order.

Should the save_vote, append, truncate and purge be put into a queue and executed sequentially, one-by-one, in the order of the calls?

yes.

[Jason5Lee]

@drmingdrmer Does the documentation imply that logs must be persistent, meaning an in-memory implementation shouldn't be used in production?

[drmingdrmer]

You're absolutely right. Memory-based storage should never be used for persistent data in production environments, as data will be lost on restart or failure.