Unfail anomaly detection tests

[nikie]

Issue #, if available:
There are no user-facing changes.

Description of changes:

• Unfail anomaly detection tests by removing pytest.mark.xfail marks and making tests pass, except for the test_anomalyDetector test which relates to a not implemented feature.
• Add assertions to the test_RelativeRateOfChangeStrategy test.
• Apply black formatting to anomaly detection tests.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 8c93b14f) — may not be fully accurate. Reply if this doesn't help.

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 8c93b14f) — may not be fully accurate. Reply if this doesn't help.

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 8c93b14f) — may not be fully accurate. Reply if this doesn't help.

Make sure all the Checks pass inorder for the PR to be merged.

[sudsali]

@nikie Make sure all the checks pass for the PR to be merged.

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 1b0eaabe) — may not be fully accurate. Reply if this doesn't help.

[github-actions]

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 1b0eaabe) — may not be fully accurate. Reply if this doesn't help.

Additional feedback:

tests/test_anomaly_detection.py:305 — The @pytest.mark.xfail condition os.environ.get("SPARK_VERSION") in ["3.1.3", "3.2"] will only match the exact string "3.2", but SPARK_VERSION is typically set to a full version like "3.2.0" or "3.2.1". The _extract_major_minor_versions in configs.py handles this for the library, but os.environ.get returns the raw string. You should use startswith or extract the major.minor version for comparison, e.g. os.environ.get("SPARK_VERSION", "").startswith("3.1") or os.environ.get("SPARK_VERSION", "").startswith("3.2").

[github-actions]

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 1b0eaabe) — may not be fully accurate. Reply if this doesn't help.

Inline feedback:

tests/test_anomaly_detection.py:295 — The xfail condition os.environ.get("SPARK_VERSION") in ["3.1.3", "3.2"] will not match versions like "3.2.0" or "3.2.1" because it does an exact string comparison. Consider using _get_spark_version() from pydeequ.configs (which extracts major.minor) or use startswith checks, e.g.:

os.environ.get("SPARK_VERSION", "").startswith(("3.1", "3.2"))

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 1b0eaabe) — may not be fully accurate. Reply if this doesn't help.

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 1b0eaabe) — may not be fully accurate. Reply if this doesn't help.

[github-actions]

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 1b0eaabe) — may not be fully accurate. Reply if this doesn't help.

Additional feedback:

tests/test_anomaly_detection.py:295 — os.environ.get("SPARK_VERSION").startswith(("3.1", "3.2")) will raise AttributeError: 'NoneType' object has no attribute 'startswith' if the SPARK_VERSION environment variable is not set (since get returns None). Use os.environ.get("SPARK_VERSION", "").startswith(("3.1", "3.2")) to provide a default empty string.

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 1b0eaabe) — may not be fully accurate. Reply if this doesn't help.

[nikie]

@nikie Make sure all the checks pass for the PR to be merged.

Hi, @sudsali ,
The 2 failing tests tests/test_anomaly_detection.py::TestAnomalies::test_BatchNormalStrategy and test_holtWinters were previously marked with xfail as follows:

    # TODO - Failing bcoz of
    # can not implement breeze.stats.DescriptiveStats, because it is not an interface
    # (breeze.stats.DescriptiveStats is in unnamed module of loader 'app')
    @pytest.mark.xfail(reason="TODO: breeze.stats.DescriptiveStats is in unnamed module of loader 'app'")

I do not fully understand the reason, but apparently the tests succeed starting from Spark 3.3.
Deequ may have been fixed in a way that the anomaly detectors still do not work in Spark 3.1 and 3.2 in Python.

Since it is not possible to fix the original issue in python-deequ, I have returned xfail marks to the tests making them conditional to Spark 3.1 and 3.2 versions and removed "TODO" comment:

    @pytest.mark.xfail(
        os.environ.get("SPARK_VERSION", "").startswith(("3.1", "3.2")),
        reason=(
            "Not supported in Spark < 3.3: breeze.stats.DescriptiveStats "
            "is in unnamed module of loader 'app'"
        )
    )

I have tested locally for all Spark versions, used in the CI matrix: 3.1.3, 3.2, 3.3, 3.5.
The 2 affected tests xfail for Spark 3.1.3, 3.2 and succeed for 3.3 and 3.5.

If this is fine, could you, please trigger the test workflow?

No issues found.