Summary
The hardcoded workloads list in tests/unittest/benchmark_runner/common/template_operations/golden_files.py (GoldenFiles.__init__) overrides the canonical production list from environment_variables.environment_variables_dict['workloads']. This means every time a new workload is added to the production list in benchmark_runner/main/environment_variables.py, it must also be manually added to golden_files.py — a maintenance burden that is easy to miss, as evidenced by PR #1244 adding hammerdb_vm_mariadb_scale and hammerdb_vm_postgres_scale without a corresponding update here.
Required changes
tests/unittest/benchmark_runner/common/template_operations/golden_files.py
Remove the hardcoded override in GoldenFiles.__init__ and replace it with:
# Workloads excluded from golden-file regression tests.
# Add entries here (with a comment explaining why) only if a workload
# cannot reasonably produce golden files; do NOT maintain a separate allowlist.
_EXCLUDED_WORKLOADS: set = set()
environment_variables.environment_variables_dict['workloads'] = [
w for w in environment_variables.environment_variables_dict['workloads']
if w not in _EXCLUDED_WORKLOADS
]
Note: the list comprehension must be applied unconditionally — when _EXCLUDED_WORKLOADS is empty, every workload passes through unchanged. An outer if _EXCLUDED_WORKLOADS: guard would leave the old hardcoded list in place when the set is empty, defeating the purpose.
Golden files regeneration
After the code change, regenerate the golden files by running:
python tests/unittest/benchmark_runner/common/template_operations/generate_golden_files.py
Then commit the updated golden_files/ directory. New workloads (including all _scale, _lso, _ephemeral, etc. variants) will automatically be covered.
Acceptance criteria
References
Summary
The hardcoded workloads list in
tests/unittest/benchmark_runner/common/template_operations/golden_files.py(GoldenFiles.__init__) overrides the canonical production list fromenvironment_variables.environment_variables_dict['workloads']. This means every time a new workload is added to the production list inbenchmark_runner/main/environment_variables.py, it must also be manually added togolden_files.py— a maintenance burden that is easy to miss, as evidenced by PR #1244 addinghammerdb_vm_mariadb_scaleandhammerdb_vm_postgres_scalewithout a corresponding update here.Required changes
tests/unittest/benchmark_runner/common/template_operations/golden_files.pyRemove the hardcoded override in
GoldenFiles.__init__and replace it with:Note: the list comprehension must be applied unconditionally — when
_EXCLUDED_WORKLOADSis empty, every workload passes through unchanged. An outerif _EXCLUDED_WORKLOADS:guard would leave the old hardcoded list in place when the set is empty, defeating the purpose.Golden files regeneration
After the code change, regenerate the golden files by running:
Then commit the updated
golden_files/directory. New workloads (including all_scale,_lso,_ephemeral, etc. variants) will automatically be covered.Acceptance criteria
golden_files.pyis removed.golden_files.pyusesenvironment_variables.environment_variables_dict['workloads']as the source of truth._EXCLUDED_WORKLOADSdenylist pattern is in place for any future exclusions.environment_variables.pyautomatically includes it in the golden-file test without any change togolden_files.py.References