Course Lab Benchmark: Add a Check for Reference Solution Script

[tareknaser]

As discussed in today’s meeting, we now require a reference solution script for all labs. This PR is a draft and should not be merged yet since it will break CI because one task (cmu_15-213__data_lab) is missing a reference solution. PR #98 should be merged first to address this.

[Copilot]

Pull request overview

Adds an explicit requirement that every Course Lab task includes a reference solution script (sol.sh), and documents this requirement in the benchmark README.

Changes:

• Add a schema test that asserts each task folder contains sol.sh
• Update the benchmark README to list sol.sh as a required task artifact and update “Adding New Labs” instructions

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
benchmarks/courselab_bench/tests/test_data_schema.py	Adds a new test to enforce presence of sol.sh in every task directory.
benchmarks/courselab_bench/README.md	Documents sol.sh as a required part of each lab task and updates authoring instructions accordingly.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

sol.sh is later executed/read as a script; checking only exists() can pass for a directory, broken symlink, or an empty placeholder file. Consider asserting sol_path.is_file() and that it has non-empty content (and optionally that it’s readable) to avoid false positives.

Suggested change

	assert sol_path.exists(), f"{task_folder.name} missing sol.sh reference solution"
	assert sol_path.is_file(), f"{task_folder.name}: sol.sh must exist and be a regular file"
	assert sol_path.stat().st_size > 0, f"{task_folder.name}: sol.sh must not be empty"

[Copilot]

This requirement overlaps with test_required_files_exist (which maintains the list of required task files). To keep a single source of truth and avoid re-traversing all task folders, add sol.sh to the existing required_files list and drop this separate test.

Suggested change

	def test_reference_solution_exists(self):
	task_folders = get_task_folders(DATA_DIR)
	for task_folder in task_folders:
	sol_path = task_folder / "sol.sh"
	assert sol_path.exists(), f"{task_folder.name} missing sol.sh reference solution"

[tareknaser]

That's actually a very good point 😃