Skip to content

Refactor golden_files.py: replace hardcoded workloads allowlist with canonical list + exclusion denylist #1245

Description

@coderabbitai

Summary

The hardcoded workloads list in tests/unittest/benchmark_runner/common/template_operations/golden_files.py (GoldenFiles.__init__) overrides the canonical production list from environment_variables.environment_variables_dict['workloads']. This means every time a new workload is added to the production list in benchmark_runner/main/environment_variables.py, it must also be manually added to golden_files.py — a maintenance burden that is easy to miss, as evidenced by PR #1244 adding hammerdb_vm_mariadb_scale and hammerdb_vm_postgres_scale without a corresponding update here.

Required changes

tests/unittest/benchmark_runner/common/template_operations/golden_files.py

Remove the hardcoded override in GoldenFiles.__init__ and replace it with:

# Workloads excluded from golden-file regression tests.
# Add entries here (with a comment explaining why) only if a workload
# cannot reasonably produce golden files; do NOT maintain a separate allowlist.
_EXCLUDED_WORKLOADS: set = set()

environment_variables.environment_variables_dict['workloads'] = [
    w for w in environment_variables.environment_variables_dict['workloads']
    if w not in _EXCLUDED_WORKLOADS
]

Note: the list comprehension must be applied unconditionally — when _EXCLUDED_WORKLOADS is empty, every workload passes through unchanged. An outer if _EXCLUDED_WORKLOADS: guard would leave the old hardcoded list in place when the set is empty, defeating the purpose.

Golden files regeneration

After the code change, regenerate the golden files by running:

python tests/unittest/benchmark_runner/common/template_operations/generate_golden_files.py

Then commit the updated golden_files/ directory. New workloads (including all _scale, _lso, _ephemeral, etc. variants) will automatically be covered.

Acceptance criteria

  • The hardcoded workloads list in golden_files.py is removed.
  • golden_files.py uses environment_variables.environment_variables_dict['workloads'] as the source of truth.
  • An explicit _EXCLUDED_WORKLOADS denylist pattern is in place for any future exclusions.
  • Golden files are regenerated and committed.
  • Adding a new workload to environment_variables.py automatically includes it in the golden-file test without any change to golden_files.py.

References

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions