feat: Dataflow template for importing Aerospike backups into Bigtable#189
Conversation
Co-authored-by: Kajetan Boroszko <kajetan@unoperate.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a suite of tools for migrating data from Aerospike to Cloud Bigtable, including an adapter for data transformation, a JNI-based backup reader, a Dataflow template, and a Kafka Connect SMT. The review identified several critical areas for improvement: the ReadRecordResult class needs equals() and hashCode() implementations for correct map-based lookups, the Docker build relies on an unstable release candidate branch, and the AerospikeRecord list expansion is vulnerable to OOM attacks from malicious qualifiers. Additionally, the Dataflow pipeline requires improvements in temporary file handling, Bigtable batch flushing, and proper interruption handling in the BigtableWriteDoFn.
brandtnewton
left a comment
brandtnewton
left a comment
There was a problem hiding this comment.
LGTM, just waiting on internal process to merge
2 participants
This PR adds a Dataflow template that imports Aerospike backups from Google Cloud Storage into Cloud Bigtable.
To be merged after or instead of #185.