Repository navigation
feat(streams): AT_TIMESTAMP shard iterator on all four backends - #468
Open
LeeroyHannigan wants to merge 1 commit into
Open
LeeroyHannigan wants to merge 1 commit into
LeeroyHannigan wants to merge 1 commit into
Conversation
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
October 23, 2026 16:30
This was referenced Oct 24, 2026
LeeroyHannigan
force-pushed
the
fix/stream-record-single-clock
branch
2 times, most recently
from
October 29, 2026 21:05
f1708df to
7aebd15
Compare
DynamoDB Streams added the AT_TIMESTAMP shard iterator type on 2026-10-15. GetShardIterator takes ShardIteratorType AT_TIMESTAMP with a Timestamp member (epoch seconds, fractional milliseconds allowed) and returns an iterator positioned at the first record created at or after that time. ExtendDB answered SerializationException for the type. Behaviour, measured on the service (us-east-1, 2026-10-23; raw requests and responses recorded in the issue): - The seek compares at millisecond precision. A Timestamp of 14:04:08.5 skips records written at 14:04:08.050 to .071 and starts at the next group, while the integer 14:04:08 includes them. - A timestamp older than the oldest retained record returns everything with no error; one past the newest record positions after it, so GetRecords returns records written from then on. - Validation order: the type enum, then Timestamp presence or absence for the type, then SequenceNumber presence or absence for the type, then the future check, then stream and shard existence. Every parameter message carries the prefix "One or more parameter values were invalid: ". - The future buffer is about one second: now + 1 s was accepted, now + 2 s refused. A Timestamp sent as a JSON string, boolean or object is a SerializationException; null, zero and negative are the "Valid timestamp must be provided" refusal. Implementation: - ShardIteratorType::AtTimestamp and GetShardIteratorInput.timestamp. - The handler refuses a missing or null type with the service's "Member must not be null" text, pre-validates the type enum (the service's member order, rendered without the offending value as the service does), refuses a Timestamp whose JSON type is not a number with the service's deserializer texts, then runs the parameter rules in streams_iterator_params.rs. The same module now owns the AT_SEQUENCE_NUMBER to AFTER conversion. - StreamEngine::first_sequence_number_at_or_after(shard_id, at_ms) on all four backends, each in its own stream_at_timestamp.rs. SQLite, PostgreSQL and MongoDB take MIN(sequence_number) over created_at >= T. Cassandra seeks the clustering key with the HLC millisecond prefix of T as the lower bound and filters created_at within that partition: created_at_ms <= hlc_ms holds for every record since the one-clock-read change, so created_at_ms >= T implies the sequence is at or after the bound and no qualifying record is skipped, and the filter drops a record written under a clamped clock whose prefix clears the bound while its created_at is before T. MongoDB and SQLite gain a (shard_id, created_at) index for the seek; on MongoDB the existing index walked 50,000 documents (84 ms) for a mid-range seek over a 100,000-record shard. - Resolution: a match positions AT that record (the existing n - 1 conversion); no match positions after the newest record. The tip is read before the seek so a write landing between the two reads is not skipped. - TRIM_HORIZON and LATEST with a SequenceNumber now refuse as the service does (#461), since the same check covers them. Tests: tests/test_streams_at_timestamp.py, eleven tests with every expected value from the measurement, including the at-or-after boundary at a record's own millisecond, passing on SQLite, PostgreSQL, MongoDB and Cassandra; nine of the original ten fail against a server built from the base commit. The existing tests/test_streams.py passes on all four. Unit tests cover the millisecond floor, the validation order and texts, the Timestamp JSON-type texts, the future buffer, the Cassandra bound, and the tip-then-seek order against a double that writes during the tip read (the test fails with the reads swapped). A Cassandra direct test inserts a clamped record and asserts the seek does not return it for a T after its created_at. Docs: a Streams section in differences-from-dynamodb.md, the usage guide example, the streams design doc, and EXTENDDB_STREAM_SINCE in samples/stream_consumer.py. Closes #460. Closes #461.
LeeroyHannigan
force-pushed
the
fix/stream-record-single-clock
branch
from
October 30, 2026 11:10
7aebd15 to
a5f4390
Compare
LeeroyHannigan
force-pushed
the
feat/streams-at-timestamp
branch
from
October 30, 2026 11:10
0bc15d3 to
4444bc6
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #460. Closes #461. Depends on #463 (this PR is based on its branch and retargets to
mainwhen it merges).DynamoDB Streams added the
AT_TIMESTAMPshard iterator type on 2026-10-15.GetShardIteratortakesShardIteratorType: AT_TIMESTAMPwith aTimestampmember (epoch seconds, fractional milliseconds allowed) and returns an iterator positioned at the first record created at or after that time. ExtendDB answeredSerializationExceptionfor the type.Behaviour, measured on the service
us-east-1, 2026-10-23, raw requests and responses recorded per probe.
Timestampof14:04:08.5skips records written at14:04:08.050to.071and starts at the next group; the integer14:04:08includes them.GetRecordsreturns records written from then on.Timestamppresence or absence for the type, thenSequenceNumberpresence or absence for the type, then the future check, then stream and shard existence. Every parameter message carries the prefixOne or more parameter values were invalid:. The enum violation names no value:Value at 'shardIteratorType' failed to satisfy constraint: Member must satisfy enum value set: [AFTER_SEQUENCE_NUMBER, LATEST, AT_TIMESTAMP, AT_SEQUENCE_NUMBER, TRIM_HORIZON]; a missing type isMember must not be null.Timestampsent as a JSON string isSerializationException: STRING_VALUE cannot be converted to Date; a boolean isTRUE_VALUE cannot be converted to Date; an object isStart of structure or map found where not expected. Null, zero and negative are theValid timestamp must be provided when the iterator type is AT_TIMESTAMPrefusal.Change
ShardIteratorType::AtTimestampandGetShardIteratorInput.timestamp.handle_get_shard_iteratorrefuses a missing or null type, pre-validates the enum in the service's member order, refuses a non-numericTimestampwith the service's deserializer texts, then runs the parameter rules in the newstreams_iterator_params.rs, which also owns theAT_SEQUENCE_NUMBERtoAFTERconversion.TRIM_HORIZONandLATESTwith aSequenceNumbernow refuse as the service does (GetShardIterator accepts SequenceNumber with TRIM_HORIZON and LATEST #461), since the same rule covers them.StreamEngine::first_sequence_number_at_or_after(shard_id, at_ms)on all four backends, each in its ownstream_at_timestamp.rs. SQLite, PostgreSQL and MongoDB takeMIN(sequence_number)overcreated_at >= T. Cassandra seeks the clustering key with the HLC millisecond prefix ofTas the lower bound and filterscreated_atwithin that partition:created_at_ms <= hlc_msholds for every record after fix(streams): one clock read per stream record on all four backends #463, socreated_at_ms >= Timplies the sequence is at or after the bound and no qualifying record is skipped, and the filter drops a record written under a clamped clock whose prefix clears the bound while itscreated_atis beforeT.(shard_id, created_at)index for the seek. On MongoDB the existing(shard_id, sequence_number)index walked 50,000 documents (84 ms) for a mid-range seek over a shard of 100,000 records; PostgreSQL has the same shape (19 ms at 100,000) and gets its index in a separate data migration (PostgreSQL: index for the AT_TIMESTAMP stream seek needs a data migration #467).Tests
tests/test_streams_at_timestamp.py: eleven wire tests with every expected value from the measurement, including the at-or-after boundary at a record's own millisecond (found by bisecting through the API), the tip-plus-one case, the full validation matrix, the enum and missing-type texts, theTimestampJSON-type texts, and the future buffer. 11 of 11 on SQLite, PostgreSQL, MongoDB and Cassandra servers built from this commit; 9 of the original 10 fail against a server built from the base commit. The existingtests/test_streams.pypasses on all four..9996ms fraction stays at123), the validation order and texts, theTimestampJSON-type texts, the future buffer, the Cassandra bound, and the tip-then-seek order against aStreamEnginedouble that writes during the tip read (the test fails with the two reads swapped).tests/direct/streams.rs: a clamped record (sequence prefix 8 s after itscreated_at) is not returned for aTafter itscreated_at, and the record whosecreated_atis at or afterTis.Gates on this commit:
cargo fmt,cargo +1.97.0 clippy --all-targets -- -D warnings,cargo +1.88.0 check --workspace --all-targets, 1,693 workspace unit tests.Docs
A Streams section in
docs/differences-from-dynamodb.md(fixed shards, and theAT_TIMESTAMPcontract including the one-second future buffer), the usage guide example, the streams design doc, andEXTENDDB_STREAM_SINCEonsamples/stream_consumer.py.Not changed here
DescribeStream.ShardFilter:CHILD_SHARDSis public and ExtendDB has noShardFiltersupport; the service's timestamp filter onDescribeStreamis not in the public API.(shard_id, created_at)index needs a data migration and anextenddb migratestep on existing deployments; it is PostgreSQL: index for the AT_TIMESTAMP stream seek needs a data migration #467 rather than part of this change.