[kevin9327]

Symptom

Running the dataset inspector on a multi-config dataset gives a different answer every time, and sometimes just fails. Same command, five consecutive runs on upstream/main (f3186ef):

$ for i in 1 2 3 4 5; do uv run skills/huggingface-llm-trainer/scripts/dataset_inspector.py \
    --dataset nyu-mll/glue --split train; done

Config 'default' not found, trying 'qnli'...   Config: qnli
Config 'default' not found, trying 'cola'...   Config: cola
Config 'default' not found, trying 'mrpc'...   Config: mrpc
Config 'default' not found, trying 'stsb'...   ERROR: API request failed: 502 Bad Gateway
Config 'default' not found, trying 'wnli'...   Config: wnli

skills/huggingface-vision-trainer/scripts/dataset_inspector.py is the same script and behaves the same way:

Config 'default' not found, trying 'rte'...    Config: rte
Config 'default' not found, trying 'rte'...    Config: rte
Config 'default' not found, trying 'cola'...   ERROR: API request failed: The read operation timed out
Config 'default' not found, trying 'cola'...   Config: cola
Config 'default' not found, trying 'mrpc'...   Config: mrpc

This matters because the skill's documented workflow is "inspect the dataset, read the compatibility verdict, then train" (SKILL.md: "Step 2: Validate dataset format (run dataset_inspector.py)"). The verdict is for whichever config the run happened to land on, and nothing in the output says the choice was arbitrary.

Cause

available_configs is a set, and the fallback takes list(...)[0]:

if not split_found and available_configs:
    config_to_use = list(available_configs)[0]

Set iteration order for strings depends on per-process hash randomisation, so the "first available" config is a different one on each invocation. Two lines further down the same set is already printed with sorted(available_configs), so the ordering problem is only in the selection.

Separately, the fallback never checks whether the chosen config actually has the requested split. On nyu-mll/glue the ax config only has a test split, so when the draw landed there the run died with a 500 from the rows endpoint rather than inspecting anything.

Fix

Track which configs carry the requested split and pick the lowest-sorted one, falling back to the previous "any config" behaviour when no config has that split. 5 added / 2 removed lines, applied identically to both copies of the script.

After

Ten runs, five per script, deterministic and landing on a config that actually has a train split:

Config 'default' not found, trying 'cola'...   Config: cola   Split: train      (x5, llm-trainer)
Config 'default' not found, trying 'cola'...   Config: cola   Split: train      (x5, vision-trainer)

(Two of those ten runs hit an unrelated transient read operation timed out from datasets-server; the config selection line was cola in every one of the ten.)

Cases pinned — unchanged before and after

Command	before	after
--dataset trl-lib/Capybara --split train (single config named default)	Config: default	Config: default
--dataset nyu-mll/glue --config sst2 --split train (config given explicitly)	Config: sst2	Config: sst2
--dataset rishitdagli/cppe-5 --split train (vision-trainer)	Config: default	Config: default
--dataset nyu-mll/glue --split nope (split exists in no config)	arbitrary config	same behaviour, now deterministic

None of these reach the changed branch except the last, which keeps the old "any available config" semantics.

Verification

uv run scripts/generate_cursor_plugin.py --check      # Cursor plugin artifacts are up to date.
uv run scripts/plugin_versions.py check               # in sync: 1.0.29
uv run scripts/plugin_versions.py check-bump <base>   # Version bumped: 1.0.28 -> 1.0.29
uv run scripts/build_skills_index.py --repo-root . --out-dir out/skills-index/latest --branch main
uv run scripts/test_build_skill_distribution.py       # Ran 2 tests ... OK
uv run scripts/build_skill_distribution.py --skills-dir skills --out-dir out/skills-distribution/latest --uri-prefix skill://

Version bumped to 1.0.29 and manifests regenerated per scripts/publish.sh; if that collides with another PR it is reproducible with uv run scripts/plugin_versions.py bump patch && ./scripts/publish.sh.