TwoCommit
There can occur situations, when during connection commit blobbers will have different states. Less than enough to recover data blobbers can receive write marker and commit connection S1, the rest of bobbers can stay with previous uncommitted state S0 (broken). If a client will decide to repair such allocation, there won't be enough bobbers to restore committed state and there is no way to rollback, since blobbers have already merged changes.
It can happen, when blobbers will crash during connection commit, or client will send commit only partially to blobbers.
To provide a way to get back to previous correct state (S0), even if it is committed on part of the network, we introduce double commit approach. We add new state of the filesystem "pre-committed" S1 (in yellow)
We should assume that client-blobber interactions (data upload, connection_commit) are decoupled from blobber-blockchain-validator interactions (write_marker submit, challenge complete). It is achieved mainly by using chains of writemarkers during completing challenges, which in order allows blobbers not to store all the versions of allocations, but use the latest one and do not store any earlier versions.
- user uploads 2 files, f1 and f2
- user commits them with writemarker W1
- files are moved to precommit directory with version 1
- user uploads f3 and deletes f2
- user commits with writemarker W2
- f1 and f2 are moved to filestore with version 1
- f3 and delete of f2 (REF table record) are moved to the precommit dir with version 2
- user uploads f4
- user commits with W3
- f2 is deleted in filestore, f3 is added changing fs version to 2
- f4 is moved to precommit changing the version to 3
- user uploads 2 files, f1 and f2
- user commits them with writemarker W1
- files are moved to precommit directory with version 1
- user uploads f3 and deletes f2
- user commits with writemarker W2
- f1 and f2 are moved to filestore with version 1
- f3 and delete of f2 (REF table record) are moved to the precommit dir with version 2
- user uploads f4
- user rollbacks with empty writemarker W2e
- precommit dir is discarded as well as temp dir and allocation gets back to version 1
- f3 is aploaded and committed with writemarker W3
At the moment changes are stored to getAllocTempDir (tmp) directory. We will add new directory getAllocPreCommitDir (precommit). After changes are pre-committed they are moved to this directory from getAllocTempDir. It will allow us to store S1 (pic) in the different isolated sandbox.
It is the main part of double-commit protocol, we pre-commit on CommitWrite, but instead of connectionObj.ApplyChanges we move changes to getAllocPreCommitDir dir.
When client commits write blobber does several actions:
- commit previously pre-committed data with
connectionObj.ApplyChangesand moving them fromgetAllocPreCommitDirto filesystem as before on commit - pre-commit current data that is speculative, move it to
getAllocPreCommitDirfromgetAllocTempDir
##Rollback
We add Rollback, to be able to rollback pre-committed state S1(pic) to committed S0(pic) state. We do not delete pre-committed changes though, since these changes could be already committed to blockchain and challenged, instead we store them locally for some time, but remove from getAllocPreCommitDir to some other temp dir. These changes won't be used anyhow in the future, only to complete challenges.
/v1/file and /v1/dir uses pre-committed state to manipulate it, so instead of merged state before, we will use files from connectionObj.ApplyChanges section to build state upon
It is suggested to send empty writemarker after client session is ended to commit all the changes and empty pre-commit and speculative sections.
Challenge protocol do not change, everything works the same, the only difference is that rolled back data can be challenged and processed the same way as regular challenge. It is added for generality, to change as less as possible.
