Skip to content

Pass storage_options to semantic dedup parquet grouping #2163

Description

@praateekmahajan

In semantic dedup KMeans, break_parquet_partition_into_groups accepts storage_options, but KMeansReadFitWriteStage.process_batch and _fit_pass call it without passing self.input_storage_options.

For S3-compatible inputs, FilePartitioningStage and _read_group can use read_kwargs["storage_options"], but the grouping metadata read may fall back to ambient AWS config instead.

Fix: pass storage_options=self.input_storage_options into the two break_parquet_partition_into_groups(...) calls in semantic/kmeans.py.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions