Split-brain — SQL and Redis written non-atomically on every block commit

[krishnanmano]

Labels: bug, critical, data-integrity, statesqldb
File: core/ledger/kvledger/txmgmt/statedb/statesqldb/statesqldb.go:289-361
Also affects: statesqldb/redis_client.go:52

ApplyUpdates writes world-state to SQL first, then calls
redisSchema.SetHeight. Any crash or error between these two operations leaves
SQL and Redis permanently inconsistent. There is no two-phase commit, saga
compensation, or replay-on-restart mechanism. On a payment network this is a
ledger integrity violation — Redis height can drift ahead of or behind the SQL
state indefinitely.

Impact: Silent data divergence between SQL and Redis on any node crash or
Redis error during block commit. Block-height cursor used by private-data purge
and snapshot export becomes unreliable.

Suggested Fix

Treat Redis as a derived cache that is always rebuildable from SQL:

1. Write SQL successfully.
2. On SQL success, write Redis height.
3. On Redis failure, write a pending_redis_height marker into SQL.
4. On peer startup, replay any pending Redis height markers before accepting new
   blocks.

Alternatively, implement a WAL-based approach where the Redis SetHeight is
retried with backoff and the SQL block is not considered committed until Redis
is updated, or Redis is removed from the commit critical path entirely.