Add a tox matrix for local test runs (Python x Spark, conda-managed JDK)

[fdosani]

Summary

Add a tox.ini providing a local test matrix that mirrors .github/workflows/test-package.yml, so contributors can run the same Python x Spark combinations locally without hand-managing JDK installs or juggling multiple venvs.

Motivation

Today, reproducing CI locally means manually:

• Installing the right JDK 17 for Spark tests (per CLAUDE.md, non-conda JDKs newer than 17 break with py4j.protocol errors).
• Switching between pytest.ini / pytest-ansi.ini / pytest-connect.ini by hand.
• Re-installing pyspark[connect] at different pins (3.5.8 vs 4.1.2) to check both Spark tracks.
• Remembering that Spark Connect tests must run in their own pytest process, since starting a local Connect server sets SPARK_LOCAL_REMOTE and every later SparkSession.builder.getOrCreate() in that process returns the Connect session (a classic and a Connect session can't coexist in one run).

A tox matrix (using tox-conda so Java/PySpark can come from conda-forge instead of a preinstalled system JDK) would let contributors run tox -e py312-spark4 (or similar) and get a CI-equivalent environment in one command, and would give us a single place to keep the "which pytest config with which extras" logic in sync with CI.

Proposed environments

Based on the current CI matrix in .github/workflows/test-package.yml and the three pytest configs in the repo root:

• py{310,311,312,313}-nospark — basic install (tests, qa extras only), mirrors test-basic-install.
• py311-spark35 — Spark 3.5.8 classic + ANSI mode, mirrors test-with-spark-3-install.
• py311-spark35-connect — Spark 3.5 + Spark Connect via --packages (Ivy resolution), since the 3.5 wheel doesn't bundle the Connect server jar. Experimental / not currently covered by CI, so treat failures as informative rather than blocking.
• py{310,311,312,313}-spark4 — Spark 4.1.2 classic + ANSI mode + both Spark Connect suites, mirrors test-with-spark-4-install.

All spark envs install openjdk=17 via conda_deps rather than relying on a system JDK. Snowflake tests are left out of the matrix entirely and rely on pytest.importorskip("snowflake.snowpark") to skip when the snowflake extra isn't installed (matching how the existing CI jobs also don't install snowflake).

Open questions

• tox-conda 0.10.x only supports tox 3 (envlist, usedevelop, not tox 4's env_list/package = editable) — do we want to pin to tox 3, or is it worth checking whether a tox 4 + conda plugin combination is viable instead?
• Should this live in CI as an alternative execution path, or purely as a local dev convenience (this issue assumes the latter)?
• Do we want a snowflake env added later once local Snowpark testing mode is exercised in CI?

Draft config

A working draft is attached below for reference/discussion — not meant to be merged as-is:

tox.ini draft

; Tox configuration for datacompy's local test matrix, using conda-managed
; environments (tox-conda) so Java and PySpark can be pulled from conda-forge
; rather than requiring a preinstalled JDK.
;
; Runs the full pytest suite (pandas, polars, spark, base, report, comparator,
; utility) across Spark 3.5 and Spark 4.x. Snowflake tests are skipped
; automatically via ``pytest.importorskip("snowflake.snowpark")`` when the
; snowflake extra is not installed -- so this file simply omits it.
;
; tox-conda 0.10.x only supports tox 3 (``envlist`` not ``env_list``,
; ``usedevelop`` not ``package = editable``).
;
; Local usage:
;   pip install "tox<4" tox-conda
;   tox                       # runs every env whose Python interpreter is available
;   tox -e py312-spark4       # run a specific env
;   tox -e py311-spark35      # run Spark 3.5 against Python 3.11
;   tox -e py312-spark4 -- tests/test_pandas.py     # forward args to pytest

[tox]
requires =
    tox<4
    tox-conda
envlist =
    py{310,311,312,313}-nospark
    py311-spark35
    py311-spark35-connect
    py{310,311,312,313}-spark4

[testenv]
description = Run the datacompy test suite for {envname}
usedevelop = true
passenv =
    JAVA_HOME
    HADOOP_HOME
    SPARK_HOME
    SPARK_LOCAL_IP
    SPARK_LOCAL_REMOTE
    http_proxy
    https_proxy
    no_proxy
    HTTP_PROXY
    HTTPS_PROXY
    NO_PROXY
setenv =
    PYTHONDONTWRITEBYTECODE = 1
conda_channels =
    conda-forge
; ``--override-channels`` prevents conda from consulting the base channels
; (e.g. ``conda-shared``) that would otherwise resolve ``python=3.10`` to
; GraalPy, whose SSL module is JSSE-backed and breaks pip.
conda_create_args =
    --override-channels
conda_install_args =
    --override-channels

; Basic install: no pyspark, no snowflake. pandas/polars/base/report/comparator
; tests run; spark and snowflake tests are skipped via ``importorskip``.
[testenv:py{310,311,312,313}-nospark]
extras =
    qa
    tests
commands =
    pytest --cov=datacompy --cov-report=term-missing {posargs}

; Spark 3.5 track. pyspark is pinned to 3.5.8 to match CI. openjdk 17 comes
; from conda-forge. Spark Connect is NOT exercised here -- see
; ``py311-spark35-connect`` below for that.
[testenv:py311-spark35]
extras =
    qa
    tests
    tests-spark
conda_deps =
    openjdk=17
deps =
    pyspark[connect]==3.5.8
commands =
    pytest --cov=datacompy --cov-report=term-missing {posargs}
    pytest -c pytest-ansi.ini --cov=datacompy --cov-report=term-missing --cov-append {posargs}

; Spark 3.5 + Spark Connect. Experimental: Spark 3.5 does not bundle the
; Connect server jar in the pyspark wheel (4.x does), so we ask the JVM to
; resolve it at gateway-launch time via Ivy by setting ``PYSPARK_SUBMIT_ARGS``
; -- PySpark's ``launch_gateway`` uses that env var to build the ``spark-submit``
; command, and ``--packages`` there triggers Ivy resolution before the Connect
; plugin class is loaded. The first run downloads the jar from Maven Central
; and caches it under ``~/.ivy2``; subsequent runs are offline. CI has never
; exercised this combination, so treat failures here as informative rather
; than blocking.
[testenv:py311-spark35-connect]
extras =
    qa
    tests
    tests-spark
conda_deps =
    openjdk=17
deps =
    pyspark[connect]==3.5.8
setenv =
    {[testenv]setenv}
    PYSPARK_SUBMIT_ARGS = --packages org.apache.spark:spark-connect_2.12:3.5.8 pyspark-shell
commands =
    pytest -c pytest-connect.ini tests/test_spark.py tests/comparator/
    pytest -m spark_connect tests/test_spark_connect.py

; Spark 4 track. Includes classic sessions (default + ANSI mode) and the two
; Spark Connect suites. Connect must run in its own pytest process because
; starting a local Connect server sets SPARK_LOCAL_REMOTE, after which every
; later ``SparkSession.builder.getOrCreate()`` returns the Connect session --
; so a classic and a Connect session cannot coexist in one run.
[testenv:py{310,311,312,313}-spark4]
extras =
    qa
    tests
    tests-spark
conda_deps =
    openjdk=17
deps =
    pyspark[connect]==4.1.2
commands =
    pytest --cov=datacompy --cov-report=term-missing {posargs}
    pytest -c pytest-ansi.ini --cov=datacompy --cov-report=term-missing --cov-append {posargs}
    pytest -c pytest-connect.ini tests/test_spark.py tests/comparator/
    pytest -m spark_connect tests/test_spark_connect.py

Acceptance criteria

• tox.ini added at the repo root with the environments above (or an agreed variant).
• Contributor-facing docs (README or CONTRIBUTING) mention tox as an optional way to run the local matrix, alongside the existing pytest instructions.
• Confirmed no conflict with the edgetest extra/workflow already used for dependency-bound testing.