-
Notifications
You must be signed in to change notification settings - Fork 19
Add WarehouseLambda core with S3 WarehouseDumper #974
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
26fc5bf to
7d17137
Compare
0a4c78d to
bd44498
Compare
2cfa97e to
4cf4c6e
Compare
bd44498 to
85b800b
Compare
6174878 to
735851d
Compare
182a666 to
3ecf91f
Compare
735851d to
3d20dca
Compare
03a355f to
acbf663
Compare
3d20dca to
b2e296b
Compare
acbf663 to
af3f70c
Compare
8c6089c to
1c62fe0
Compare
315be59 to
7cea41d
Compare
elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda/warehouse_dumper.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda/warehouse_dumper.rb
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda/warehouse_dumper.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda/warehouse_dumper.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda/warehouse_dumper.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/spec/unit/elastic_graph/warehouse_lambda/warehouse_dumper_spec.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/spec/unit/elastic_graph/warehouse_lambda/warehouse_dumper_spec.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/spec/unit/elastic_graph/warehouse_lambda/warehouse_dumper_spec.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/spec/unit/elastic_graph/warehouse_lambda/warehouse_dumper_spec.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/spec/unit/elastic_graph/warehouse_lambda_spec.rb
Outdated
Show resolved
Hide resolved
1a6f1d7 to
1229b07
Compare
myronmarston
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left some more feedback. Most of it is cosmetic...with the exception of the version vs json_schema_version mixup--that one matters!
elasticgraph-warehouse_lambda/spec/unit/elastic_graph/warehouse_lambda_spec.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda/warehouse_dumper.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda/warehouse_dumper.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda/warehouse_dumper.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda/warehouse_dumper.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda.rb
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/sig/elastic_graph/warehouse_lambda.rbs
Outdated
Show resolved
Hide resolved
elasticgraph-warehouse_lambda/spec/unit/elastic_graph/warehouse_lambda/warehouse_dumper_spec.rb
Outdated
Show resolved
Hide resolved
1229b07 to
e201d75
Compare
myronmarston
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM apart from the build failure. Left a suggestion which should fix it.
elasticgraph-warehouse_lambda/sig/elastic_graph/warehouse_lambda.rbs
Outdated
Show resolved
Hide resolved
This commit implements the core S3 data export functionality for the elasticgraph-warehouse_lambda gem, including the main WarehouseLambda class and WarehouseDumper implementation. Components added: - WarehouseLambda: Main class that orchestrates indexer, S3 client, and warehouse dumper with from_parsed_yaml factory method - WarehouseDumper: Implements DatastoreIndexingRouter interface to dump indexing operations to S3 as gzipped JSONL files - Configures indexer with WarehouseDumper as the datastore_router to redirect all indexing operations to S3 instead of the datastore Key features: - Groups operations by GraphQL type and writes separate JSONL files per type - Generates deterministic S3 keys with versioning, date partitioning, and UUIDs: s3://bucket/dumped-data/<prefix>/<Type>/v<version>/<date>/<uuid>.jsonl.gz - Structured logging for observability (batch counts, file sizes, S3 keys) - Extracts data from operation payloads including id, __eg_version, and record fields - Filters operations to only include primary index updates - Skips operations with no datastore payload (e.g., delete operations) - Returns BulkResult with success status for all operations - Leverages aws-sdk-s3 with SHA256 checksum validation and conditional writes - S3 client initialization with optional region override from config Testing: - 37 comprehensive test examples covering all components - Tests verify S3 operations, JSONL formatting, compression, and logging - Uses stubbed AWS S3 client for fast, deterministic tests - Full RBS type signatures for type safety - 100% test coverage (from_env marked :nocov: until LambdaFunction added in next commit) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Co-authored-by: Myron Marston <myron.marston@gmail.com>
33a6d6d to
b716416
Compare
…da.rbs Co-authored-by: Myron Marston <myron.marston@gmail.com>
This commit implements the core S3 data export functionality for the
elasticgraph-warehouse_lambda gem, including the main WarehouseLambda
class and WarehouseDumper implementation.
Components added:
warehouse dumper with from_parsed_yaml factory method
indexing operations to S3 as gzipped JSONL files
redirect all indexing operations to S3 instead of the datastore
Key features:
s3://bucket/dumped-data///v//.jsonl.gz
Testing: