[PureGo] Generalize stream durability model - #682
Open
zlata-stefanovic-db wants to merge 2 commits into
Open
Conversation
zlata-stefanovic-db
changed the base branch from
purego-descriptor-cache-lifecycle
to
main
August 10, 2026 14:38
Add protocol-neutral unit-count acknowledgments, partial replay slicing, and submission receipts while preserving atomic proto and JSON behavior. Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
zlata-stefanovic-db
force-pushed
the
purego-stream-durability-model
branch
from
August 10, 2026 15:27
355d04d to
dec7d80
Compare
zlata-stefanovic-db
marked this pull request as ready for review
August 10, 2026 15:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Generalizes the internal stream core so one implementation can serve both the existing atomic protocols (Protocol Buffers, JSON) and a future record-count protocol, with no public API change.
EncoderHooks,AckModelHooks,OpenFunc, and an exportedWireStreamlet a protocol instantiate the core over its own payload type.Atomic proto/JSON behavior is unchanged: those encoders report one unit per item, never produce partial acknowledgments, and continue to resolve wire offsets through the existing path.
Behavior fixes found while reviewing this change
Sendis still outstanding, where thatSendthen fails, is no longer converted into a non-retryable protocol error. The core cannot know how much of that send reached the server, which is not a server violation — so the unusable ack is discarded and the retryable send failure drives recovery. Without this, a transient failure killed the stream permanently instead of reconnecting, and blamed the server for a violation it never committed.Close()/Terminate()caller.Test plan
cd purego && go test -race ./...cd purego && go test -count=12 -race ./internal/stream— no flakiness in the new timing-sensitive testscd purego && go vet ./...andgofmt -lcleandeadcode ./...reports nothingdurability_model_test.goexercises the new generic seam through a protocol-neutral row-based fake: unit-resolution table cases, partial acks, replay of only the unacknowledged remainder, receipt-preserved prefix acks, and hook validation. The two correctness fixes above have regression guards that were each confirmed to fail beforehand — the recovery guard passes against the pre-change core but failed against this branch, and the teardown guard hangs to the test timeout without its fix.Notes
NEXT_CHANGELOG.mdentry: the change is confined topurego/internal/. Partial acknowledgments and submission receipts are unreachable for the shipping proto/JSON protocols, and the recovery fix restores behaviormainalready has, so there is no user-visible difference relative tomain.main. It was previously stacked onpurego-descriptor-cache-lifecycle, which was closed without merging.