In semantic dedup KMeans, break_parquet_partition_into_groups accepts storage_options, but KMeansReadFitWriteStage.process_batch and _fit_pass call it without passing self.input_storage_options.
For S3-compatible inputs, FilePartitioningStage and _read_group can use read_kwargs["storage_options"], but the grouping metadata read may fall back to ambient AWS config instead.
Fix: pass storage_options=self.input_storage_options into the two break_parquet_partition_into_groups(...) calls in semantic/kmeans.py.
In semantic dedup KMeans,
break_parquet_partition_into_groupsacceptsstorage_options, butKMeansReadFitWriteStage.process_batchand_fit_passcall it without passingself.input_storage_options.For S3-compatible inputs,
FilePartitioningStageand_read_groupcan useread_kwargs["storage_options"], but the grouping metadata read may fall back to ambient AWS config instead.Fix: pass
storage_options=self.input_storage_optionsinto the twobreak_parquet_partition_into_groups(...)calls insemantic/kmeans.py.