Skip to content

Skip tests for features not enabled and show pytest results#485

Open
ehelms wants to merge 1 commit into
theforeman:masterfrom
ehelms:feature-gated-tests
Open

Skip tests for features not enabled and show pytest results#485
ehelms wants to merge 1 commit into
theforeman:masterfrom
ehelms:feature-gated-tests

Conversation

@ehelms
Copy link
Copy Markdown
Member

@ehelms ehelms commented May 4, 2026

Why are you introducing these changes? (Problem description, related links)

When certain features are not enabled, their tests are marked as failed which can make debugging confusing.

What are the changes introduced in this pull request?

  • Checks for feature enablement and skips tests if their feature is not enabled
  • Uses flavor + parameters.yaml to determine enabled features
  • Add get_enabled_features() and has_feature() helpers to conftest.py that replicate the Ansible enabled_features calculation by combining flavor_features with user features from parameters.yaml.
  • Use pytest.mark.skipif to gate tests on their required features: foreman-proxy, hammer, azure-rm, google, remote-execution, and iop.
  • Read the Foreman admin password from
    .var/lib/foremanctl/foreman-admin-init-passwd instead of hardcoding.
  • Update the test playbook to display pytest output before failing so pass/fail details are visible in Ansible output.

How to test this pull request

Steps to reproduce:

  • Run a deployment with some set of enabled features (e.g. foremanctl deploy --add-feature iop
  • Check which tests are skipped and ensure the list looks appropriate

Checklist

  • Tests added/updated (if applicable)
  • Documentation updated (if applicable)

Add get_enabled_features() and has_feature() helpers to conftest.py
that replicate the Ansible enabled_features calculation by combining
flavor_features with user features from parameters.yaml. Use
pytest.mark.skipif to gate tests on their required features:
foreman-proxy, hammer, azure-rm, google, remote-execution, and iop.

Read the Foreman admin password from
.var/lib/foremanctl/foreman-admin-init-passwd instead of hardcoding.

Update the test playbook to display pytest output before failing so
pass/fail details are visible in Ansible output.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Member

@ekohl ekohl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hadn't seen this. #508 works towards a more generalized mechanism to skip tests for disabled features.

I like the idea of leveraging the directory structure more. How about:

  • tests
    • feature
      • foreman
        • test_base.py
      • foreman-proxy
        • test_base.py
        • test_bmc.py
      • iop
        • test_advisor.py

Then taking your approach (and very similarly https://stackoverflow.com/questions/57031403/pytest-marks-mark-entire-directory-package) we can automate it. I've taken a stab at that in #509.

Comment thread tests/conftest.py
Comment on lines +86 to +90
def foreman_admin_password():
test_dir = os.path.dirname(os.path.abspath(__file__))
foremanctl_dir = os.path.dirname(test_dir)
passwd_file = os.path.join(foremanctl_dir, '.var', 'lib', 'foremanctl', 'foreman-admin-init-passwd')
with open(passwd_file) as f:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also ran into this. I'd love to see it in a dedicated commit/PR. Please then also use rootpath

Suggested change
def foreman_admin_password():
test_dir = os.path.dirname(os.path.abspath(__file__))
foremanctl_dir = os.path.dirname(test_dir)
passwd_file = os.path.join(foremanctl_dir, '.var', 'lib', 'foremanctl', 'foreman-admin-init-passwd')
with open(passwd_file) as f:
def foreman_admin_password(pytestconfig):
passwd_file = pytestconfig.rootpath / '.var' / 'lib' / 'foremanctl' / 'foreman-admin-init-passwd'
with passwd_file.open() as f:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants