Skip to content

Conversation

@jwils
Copy link
Collaborator

@jwils jwils commented Jan 15, 2026

This commit implements the core S3 data export functionality for the
elasticgraph-warehouse_lambda gem, including the main WarehouseLambda
class and WarehouseDumper implementation.

Components added:

  • WarehouseLambda: Main class that orchestrates indexer, S3 client, and
    warehouse dumper with from_parsed_yaml factory method
  • WarehouseDumper: Implements DatastoreIndexingRouter interface to dump
    indexing operations to S3 as gzipped JSONL files
  • Configures indexer with WarehouseDumper as the datastore_router to
    redirect all indexing operations to S3 instead of the datastore

Key features:

  • Groups operations by GraphQL type and writes separate JSONL files per type
  • Generates deterministic S3 keys with versioning, date partitioning, and UUIDs:
    s3://bucket/dumped-data///v//.jsonl.gz
  • Structured logging for observability (batch counts, file sizes, S3 keys)
  • Extracts data from operation payloads including id, __eg_version, and record fields
  • Filters operations to only include primary index updates
  • Skips operations with no datastore payload (e.g., delete operations)
  • Returns BulkResult with success status for all operations
  • Leverages aws-sdk-s3 with SHA256 checksum validation and conditional writes
  • S3 client initialization with optional region override from config

Testing:

  • 37 comprehensive test examples covering all components
  • Tests verify S3 operations, JSONL formatting, compression, and logging
  • Uses stubbed AWS S3 client for fast, deterministic tests
  • Full RBS type signatures for type safety
  • 100% test coverage (from_env marked :nocov: until LambdaFunction added in next commit)

@jwils jwils force-pushed the joshuaw/warehouse-lambda-1-scaffold branch from 26fc5bf to 7d17137 Compare January 15, 2026 16:01
@jwils jwils force-pushed the joshuaw/warehouse-lambda-2-core branch from 0a4c78d to bd44498 Compare January 15, 2026 16:02
@jwils jwils force-pushed the joshuaw/warehouse-lambda-1-scaffold branch 2 times, most recently from 2cfa97e to 4cf4c6e Compare January 16, 2026 17:54
@jwils jwils force-pushed the joshuaw/warehouse-lambda-2-core branch from bd44498 to 85b800b Compare January 16, 2026 17:54
@jwils jwils force-pushed the joshuaw/warehouse-lambda-1-scaffold branch 3 times, most recently from 6174878 to 735851d Compare January 17, 2026 15:05
@jwils jwils force-pushed the joshuaw/warehouse-lambda-2-core branch 2 times, most recently from 182a666 to 3ecf91f Compare January 17, 2026 15:15
@jwils jwils force-pushed the joshuaw/warehouse-lambda-1-scaffold branch from 735851d to 3d20dca Compare January 17, 2026 15:32
@jwils jwils force-pushed the joshuaw/warehouse-lambda-2-core branch 3 times, most recently from 03a355f to acbf663 Compare January 17, 2026 20:03
@jwils jwils force-pushed the joshuaw/warehouse-lambda-1-scaffold branch from 3d20dca to b2e296b Compare January 17, 2026 20:21
@jwils jwils force-pushed the joshuaw/warehouse-lambda-2-core branch from acbf663 to af3f70c Compare January 17, 2026 20:32
Base automatically changed from joshuaw/warehouse-lambda-1-scaffold to main January 17, 2026 20:44
@jwils jwils force-pushed the joshuaw/warehouse-lambda-2-core branch 5 times, most recently from 8c6089c to 1c62fe0 Compare January 17, 2026 22:13
@jwils jwils force-pushed the joshuaw/warehouse-lambda-2-core branch 7 times, most recently from 315be59 to 7cea41d Compare January 18, 2026 02:11
@jwils jwils force-pushed the joshuaw/warehouse-lambda-2-core branch from 1a6f1d7 to 1229b07 Compare January 19, 2026 03:41
Copy link
Collaborator

@myronmarston myronmarston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some more feedback. Most of it is cosmetic...with the exception of the version vs json_schema_version mixup--that one matters!

@jwils jwils force-pushed the joshuaw/warehouse-lambda-2-core branch from 1229b07 to e201d75 Compare January 19, 2026 14:31
Copy link
Collaborator

@myronmarston myronmarston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM apart from the build failure. Left a suggestion which should fix it.

@jwils jwils enabled auto-merge (squash) January 19, 2026 23:25
@jwils jwils disabled auto-merge January 19, 2026 23:26
jwils and others added 4 commits January 19, 2026 17:48
This commit implements the core S3 data export functionality for the
elasticgraph-warehouse_lambda gem, including the main WarehouseLambda
class and WarehouseDumper implementation.

Components added:
- WarehouseLambda: Main class that orchestrates indexer, S3 client, and
  warehouse dumper with from_parsed_yaml factory method
- WarehouseDumper: Implements DatastoreIndexingRouter interface to dump
  indexing operations to S3 as gzipped JSONL files
- Configures indexer with WarehouseDumper as the datastore_router to
  redirect all indexing operations to S3 instead of the datastore

Key features:
- Groups operations by GraphQL type and writes separate JSONL files per type
- Generates deterministic S3 keys with versioning, date partitioning, and UUIDs:
  s3://bucket/dumped-data/<prefix>/<Type>/v<version>/<date>/<uuid>.jsonl.gz
- Structured logging for observability (batch counts, file sizes, S3 keys)
- Extracts data from operation payloads including id, __eg_version, and record fields
- Filters operations to only include primary index updates
- Skips operations with no datastore payload (e.g., delete operations)
- Returns BulkResult with success status for all operations
- Leverages aws-sdk-s3 with SHA256 checksum validation and conditional writes
- S3 client initialization with optional region override from config

Testing:
- 37 comprehensive test examples covering all components
- Tests verify S3 operations, JSONL formatting, compression, and logging
- Uses stubbed AWS S3 client for fast, deterministic tests
- Full RBS type signatures for type safety
- 100% test coverage (from_env marked :nocov: until LambdaFunction added in next commit)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-authored-by: Myron Marston <myron.marston@gmail.com>
@jwils jwils force-pushed the joshuaw/warehouse-lambda-2-core branch from 33a6d6d to b716416 Compare January 20, 2026 01:09
…da.rbs

Co-authored-by: Myron Marston <myron.marston@gmail.com>
@jwils jwils merged commit 8b172c8 into main Jan 20, 2026
22 checks passed
@jwils jwils deleted the joshuaw/warehouse-lambda-2-core branch January 20, 2026 02:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants