Skip to content

feat(streams): AT_TIMESTAMP shard iterator on all four backends - #468

Open
LeeroyHannigan wants to merge 1 commit into
fix/stream-record-single-clockfrom
feat/streams-at-timestamp
Open

LeeroyHannigan wants to merge 1 commit into
fix/stream-record-single-clockfrom
feat/streams-at-timestamp

Conversation

@LeeroyHannigan

Copy link
Copy Markdown
Collaborator

Closes #460. Closes #461. Depends on #463 (this PR is based on its branch and retargets to main when it merges).

DynamoDB Streams added the AT_TIMESTAMP shard iterator type on 2026-10-15. GetShardIterator takes ShardIteratorType: AT_TIMESTAMP with a Timestamp member (epoch seconds, fractional milliseconds allowed) and returns an iterator positioned at the first record created at or after that time. ExtendDB answered SerializationException for the type.

Behaviour, measured on the service

us-east-1, 2026-10-23, raw requests and responses recorded per probe.

  • The seek compares at millisecond precision. A Timestamp of 14:04:08.5 skips records written at 14:04:08.050 to .071 and starts at the next group; the integer 14:04:08 includes them.
  • A timestamp older than the oldest retained record returns everything with no error. One past the newest record positions after it, so GetRecords returns records written from then on.
  • Validation runs in this order: the type enum, then Timestamp presence or absence for the type, then SequenceNumber presence or absence for the type, then the future check, then stream and shard existence. Every parameter message carries the prefix One or more parameter values were invalid: . The enum violation names no value: Value at 'shardIteratorType' failed to satisfy constraint: Member must satisfy enum value set: [AFTER_SEQUENCE_NUMBER, LATEST, AT_TIMESTAMP, AT_SEQUENCE_NUMBER, TRIM_HORIZON]; a missing type is Member must not be null.
  • The future buffer is about one second: now + 1 s was accepted, now + 2 s refused.
  • A Timestamp sent as a JSON string is SerializationException: STRING_VALUE cannot be converted to Date; a boolean is TRUE_VALUE cannot be converted to Date; an object is Start of structure or map found where not expected. Null, zero and negative are the Valid timestamp must be provided when the iterator type is AT_TIMESTAMP refusal.

Change

  • ShardIteratorType::AtTimestamp and GetShardIteratorInput.timestamp.
  • handle_get_shard_iterator refuses a missing or null type, pre-validates the enum in the service's member order, refuses a non-numeric Timestamp with the service's deserializer texts, then runs the parameter rules in the new streams_iterator_params.rs, which also owns the AT_SEQUENCE_NUMBER to AFTER conversion. TRIM_HORIZON and LATEST with a SequenceNumber now refuse as the service does (GetShardIterator accepts SequenceNumber with TRIM_HORIZON and LATEST #461), since the same rule covers them.
  • StreamEngine::first_sequence_number_at_or_after(shard_id, at_ms) on all four backends, each in its own stream_at_timestamp.rs. SQLite, PostgreSQL and MongoDB take MIN(sequence_number) over created_at >= T. Cassandra seeks the clustering key with the HLC millisecond prefix of T as the lower bound and filters created_at within that partition: created_at_ms <= hlc_ms holds for every record after fix(streams): one clock read per stream record on all four backends #463, so created_at_ms >= T implies the sequence is at or after the bound and no qualifying record is skipped, and the filter drops a record written under a clamped clock whose prefix clears the bound while its created_at is before T.
  • Resolution: a match positions at that record; no match positions after the newest record. The tip is read before the seek, so a write landing while the tip is read is still ahead of the resolved position; with the reads in the other order it would be skipped.
  • MongoDB and SQLite gain a (shard_id, created_at) index for the seek. On MongoDB the existing (shard_id, sequence_number) index walked 50,000 documents (84 ms) for a mid-range seek over a shard of 100,000 records; PostgreSQL has the same shape (19 ms at 100,000) and gets its index in a separate data migration (PostgreSQL: index for the AT_TIMESTAMP stream seek needs a data migration #467).

Tests

  • tests/test_streams_at_timestamp.py: eleven wire tests with every expected value from the measurement, including the at-or-after boundary at a record's own millisecond (found by bisecting through the API), the tip-plus-one case, the full validation matrix, the enum and missing-type texts, the Timestamp JSON-type texts, and the future buffer. 11 of 11 on SQLite, PostgreSQL, MongoDB and Cassandra servers built from this commit; 9 of the original 10 fail against a server built from the base commit. The existing tests/test_streams.py passes on all four.
  • Unit tests: the millisecond floor (a .9996 ms fraction stays at 123), the validation order and texts, the Timestamp JSON-type texts, the future buffer, the Cassandra bound, and the tip-then-seek order against a StreamEngine double that writes during the tip read (the test fails with the two reads swapped).
  • Cassandra tests/direct/streams.rs: a clamped record (sequence prefix 8 s after its created_at) is not returned for a T after its created_at, and the record whose created_at is at or after T is.

Gates on this commit: cargo fmt, cargo +1.97.0 clippy --all-targets -- -D warnings, cargo +1.88.0 check --workspace --all-targets, 1,693 workspace unit tests.

Docs

A Streams section in docs/differences-from-dynamodb.md (fixed shards, and the AT_TIMESTAMP contract including the one-second future buffer), the usage guide example, the streams design doc, and EXTENDDB_STREAM_SINCE on samples/stream_consumer.py.

Not changed here

  • DescribeStream.ShardFilter: CHILD_SHARDS is public and ExtendDB has no ShardFilter support; the service's timestamp filter on DescribeStream is not in the public API.
  • The PostgreSQL (shard_id, created_at) index needs a data migration and an extenddb migrate step on existing deployments; it is PostgreSQL: index for the AT_TIMESTAMP stream seek needs a data migration #467 rather than part of this change.

DynamoDB Streams added the AT_TIMESTAMP shard iterator type on 2026-10-15.
GetShardIterator takes ShardIteratorType AT_TIMESTAMP with a Timestamp
member (epoch seconds, fractional milliseconds allowed) and returns an
iterator positioned at the first record created at or after that time.
ExtendDB answered SerializationException for the type.

Behaviour, measured on the service (us-east-1, 2026-10-23; raw requests and
responses recorded in the issue):

- The seek compares at millisecond precision. A Timestamp of 14:04:08.5
  skips records written at 14:04:08.050 to .071 and starts at the next
  group, while the integer 14:04:08 includes them.
- A timestamp older than the oldest retained record returns everything
  with no error; one past the newest record positions after it, so
  GetRecords returns records written from then on.
- Validation order: the type enum, then Timestamp presence or absence for
  the type, then SequenceNumber presence or absence for the type, then the
  future check, then stream and shard existence. Every parameter message
  carries the prefix "One or more parameter values were invalid: ".
- The future buffer is about one second: now + 1 s was accepted, now + 2 s
  refused. A Timestamp sent as a JSON string, boolean or object is a
  SerializationException; null, zero and negative are the "Valid timestamp
  must be provided" refusal.

Implementation:

- ShardIteratorType::AtTimestamp and GetShardIteratorInput.timestamp.
- The handler refuses a missing or null type with the service's "Member
  must not be null" text, pre-validates the type enum (the service's member
  order, rendered without the offending value as the service does), refuses
  a Timestamp whose JSON type is not a number with the service's
  deserializer texts, then runs the parameter rules in
  streams_iterator_params.rs. The same module now owns the
  AT_SEQUENCE_NUMBER to AFTER conversion.
- StreamEngine::first_sequence_number_at_or_after(shard_id, at_ms) on all
  four backends, each in its own stream_at_timestamp.rs. SQLite,
  PostgreSQL and MongoDB take MIN(sequence_number) over created_at >= T.
  Cassandra seeks the clustering key with the HLC millisecond prefix of T
  as the lower bound and filters created_at within that partition:
  created_at_ms <= hlc_ms holds for every record since the one-clock-read
  change, so created_at_ms >= T implies the sequence is at or after the
  bound and no qualifying record is skipped, and the filter drops a record
  written under a clamped clock whose prefix clears the bound while its
  created_at is before T. MongoDB and SQLite gain a (shard_id, created_at)
  index for the seek; on MongoDB the existing index walked 50,000
  documents (84 ms) for a mid-range seek over a 100,000-record shard.
- Resolution: a match positions AT that record (the existing n - 1
  conversion); no match positions after the newest record. The tip is
  read before the seek so a write landing between the two reads is not
  skipped.
- TRIM_HORIZON and LATEST with a SequenceNumber now refuse as the service
  does (#461), since the same check covers them.

Tests: tests/test_streams_at_timestamp.py, eleven tests with every
expected value from the measurement, including the at-or-after boundary
at a record's own millisecond, passing on SQLite, PostgreSQL, MongoDB and
Cassandra; nine of the original ten fail against a server built from the
base commit. The existing tests/test_streams.py passes on all four. Unit
tests cover the millisecond floor, the validation order and texts, the
Timestamp JSON-type texts, the future buffer, the Cassandra bound, and
the tip-then-seek order against a double that writes during the tip read
(the test fails with the reads swapped). A Cassandra direct test inserts a
clamped record and asserts the seek does not return it for a T after its
created_at.

Docs: a Streams section in differences-from-dynamodb.md, the usage guide
example, the streams design doc, and EXTENDDB_STREAM_SINCE in
samples/stream_consumer.py.

Closes #460. Closes #461.
@LeeroyHannigan
LeeroyHannigan force-pushed the fix/stream-record-single-clock branch from 7aebd15 to a5f4390 Compare October 30, 2026 11:10
@LeeroyHannigan
LeeroyHannigan force-pushed the feat/streams-at-timestamp branch from 0bc15d3 to 4444bc6 Compare October 30, 2026 11:10

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant