Conversation
Mirko-von-Leipzig
left a comment
Mirko-von-Leipzig
left a comment
There was a problem hiding this comment.
I'm about halfway through, I think so far my main question/suggestion is moving the proof inputs out of the db.
sergerad
force-pushed
the
sergerad-replicas
branch
from
4907595 to
637af22
Compare
Mirko-von-Leipzig
left a comment
Mirko-von-Leipzig
left a comment
There was a problem hiding this comment.
I left some minor (potentially followup) issues.
I think we can head towards an embedded store, which will also change this code somewhat, so we can have another clean up pass then perhaps.
There was a problem hiding this comment.
I suspect we could get away with a basic VecDeque for our use case since block numbers are ordered and we should only ever be inserting in sequential order?
Need to review usage site still though.
There was a problem hiding this comment.
Proof cache push can occur out of order
| self.0.send_if_modified(|current| { | ||
| if new_tip > *current { | ||
| *current = new_tip; | ||
| true | ||
| } else { | ||
| false | ||
| } | ||
| }); |
There was a problem hiding this comment.
I'm unsure when we would be calling this out-of-order?
There was a problem hiding this comment.
It should never be called out of order based on how we use it atm. I would still prefer to keep the type impl robust against out of order calls.
| // Cache the proof bytes for replica subscriptions. | ||
| proof_cache.push(block_num, ProofNotification::new(block_num, proof_bytes)); | ||
|
|
||
| // Advance the proven tip (this also notifies replica watch subscribers). |
There was a problem hiding this comment.
Won't this be much simpler if we pass the proof back to the scheduler, who can then arrange and insert the blocks in order?
There was a problem hiding this comment.
Nit: feels a bit weird to call these notifications. They're really just serialized block and block proofs, that we happen to use in the context of a cache/event system.
There was a problem hiding this comment.
Renamed to replica.rs
| } | ||
| next = next.child(); | ||
| } | ||
| // Wait for tip change. |
There was a problem hiding this comment.
fyi I'm unsure if you made this a do-while loop on purpose, but this changed call would return immediately on the first iteration, so its possible to to have this be the loop condition e.g.
while tip_rx.changed().await.is_ok() {
...
}There was a problem hiding this comment.
The thing is that we want to read the tip before waiting for tip change
3 participants
Store Replica Mode
Adds a replica mode to the store, enabling (replica) store instances to sync blocks and proofs
from an upstream store. This allows the RPC layer to be scaled horizontally by pointing multiple
RPC nodes at multiple replica stores, without adding load to the primary store or block producer.
Architecture
The store now operates in one of two mutually exclusive modes, controlled by the new
StoreModeenum:
BlockProducermode — existing behaviour. Accepts blocks from the block producer via theBlockProducergRPC service, runs the proof scheduler, and exposes theNtxBuilderservice.Replicamode — new. Syncs blocks from an upstream store via the newStoreReplicagRPCservice. Only exposes the
RpcandStoreReplicaservices — no block producer service, no NTX builder service, and no proof scheduler task.New:
StoreReplicagRPC serviceImplemented on the upstream store side. Exposes two streaming RPCs:
SubscribeBlocks— streams committed blocks to a subscriber starting from a given blocknumber.
SubscribeProofs— streams block proofs to a subscriber starting from a given block number.Operational
run-node.shupdated to demonstrate a topology with a primary storeand two replicas, with RPC nodes pointing at the replicas.