perf(pts_association): simplify cluster + tune partitioning#186
Merged
Conversation
5 tasks
…park overrides Replaces the 25-property cluster definition with a Tier 1 minimal config relying on Dataproc 2.2 defaults (AQE on, dynamic allocation on, auto executor sizing). 1 master + 2 n2d-highmem-32 workers + otg-etl-25-secondary. Companion to PTS PR (algorithmic refactor of compute_novelty).
Add two targeted Spark properties to the pts_association cluster: - spark.sql.shuffle.partitions=2000 - spark.sql.adaptive.advisoryPartitionSizeInBytes=64MB Manual benchmark on the 26.03-ppp.1 dataset: - Tier 1 (defaults): 14m48s - Tier 2 (these properties): 11m41s (-22%) Heaviest stages (indirect by_datasource, indirect by_datatype, indirect overall) saved ~1min each from the finer partitioning, which also relieves memory pressure observed in the YARN node manager metrics.
DSuveges
force-pushed
the
association-perf-tier1
branch
from
76677dc to
bc50776
Compare
DSuveges
approved these changes
May 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Parallel solution to opentargets/issues#4375 (#185 is the other in-flight proposal). The companion algorithmic refactor lives in opentargets/pts#114. This PR is just the cluster definition.
Cluster simplification (Tier 1)
Replaces the previous 35-property block with a minimal definition that relies on Dataproc 2.2 defaults:
n2d-highmem-32otg-etl-25-secondaryautoscaling policy (existing)idle_delete_ttl: 3600,internal_ip_only: falseTargeted sizing overrides (Tier 2, second commit)
spark.sql.shuffle.partitions=2000spark.sql.adaptive.advisoryPartitionSizeInBytes=64MBThese two properties give finer-grained tasks, which generates more YARN backlog so the autoscaler ramps more aggressively, and reduces per-task memory footprint.
Manual benchmark on 26.03-ppp.1
pts_association(35 properties)Per-stage breakdown (Run-001 → Run-003) shows the indirect aggregations save ~1min each from the finer partitioning. Direct stages are essentially flat.
Test plan
open-targets-eu-dev).Companion PR
opentargets/pts#114 — algorithmic refactor + determinism fix in
_aggregate_associations. Should land together with this one.