Add models and evaluation option#30
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds support for the GPT-5.3 Codex model and introduces a configurable judge version option to support multiple toolchain versions (201907, 202301, 202510) with their corresponding language support and library documentation.
Changes:
- Added GPT-5.3 Codex model configuration and pricing
- Introduced
judge_versionparameter throughout the evaluation pipeline to support multiple judge toolchains (201907, 202301, 202510) - Added support for 16 additional programming languages beyond the original 5, with judge-version-specific library documentation and CE code files
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/ale_bench_eval/language_config.py | New module defining language/judge version configurations, validation, and compatibility mappings |
| src/ale_bench_eval/prompts/texts.py | Refactored language/library documentation to be judge-version-aware; added support for 16+ languages |
| src/ale_bench_eval/prompts/builder.py | Updated to use judge-version-aware language documentation and modified code extraction to respect judge version |
| src/ale_bench_eval/scaffolds.py | Propagated judge_version parameter to evaluation calls |
| src/ale_bench_eval/evaluate.py | Extended CE code handling for all new languages; propagated judge_version to private evaluation |
| src/ale_bench_eval/selection.py | Updated type annotations to use new StoredCodeLanguage type |
| src/ale_bench_eval/data_types.py | Updated Solution model to use StoredCodeLanguage type |
| src/ale_bench_eval/safe_generation.py | Added vllm-chat and vllm-responses provider support |
| src/ale_bench_eval/codes/* | Added CE code files for 11 new languages (bash, csharp, fish, fortran, go, haskell, javascript, julia, lean, ocaml, perl, pypy, typescript) |
| src/ale_bench_eval/calc_cost.py | Added pricing information for GPT-5.3 Codex model |
| src/ale_bench_eval/main.py | Added judge_version parameter, validation, and backward compatibility handling |
| scripts/run_eval.sh | Enhanced with option parsing for judge_version, code_language, and prompt_language |
| docs/evaluation.md | Updated documentation with judge version usage, new language support, and script examples |
| llm_configs/gpt-5.3-codex-xhigh.json | New model configuration file for GPT-5.3 Codex |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| - Microsoft.ML.LightGbm = 4.0.2 | ||
| - ac-library-csharp = 3.9.2-atcoder1""", | ||
| "fish": _NO_LIBRARY_DOC_MESSAGE, | ||
| "fortran": """ac-library-fortran = |
There was a problem hiding this comment.
The Fortran library string is missing the leading dash for the first library entry. All other language library strings in this file start each library with a dash. The first library should be "- ac-library-fortran =" to maintain consistency.
| "fortran": """ac-library-fortran = | |
| "fortran": """- ac-library-fortran = |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 27 out of 27 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -269,6 +271,13 @@ def main( | |||
| """Main entry point for running LLM benchmarking evaluation.""" | |||
| start_time = get_now_utc() | |||
|
|
|||
| # NOTE: | |||
| # python-fire may parse numeric-looking CLI args (e.g. 202510) as int. | |||
| # Normalize all enum-like CLI parameters to strings at the entry point. | |||
| code_language = cast("EvalCodeLanguage", str(code_language).strip()) | |||
| judge_version = cast("EvalJudgeVersion", str(judge_version).strip()) | |||
| prompt_language = cast('Literal["en", "ja"]', str(prompt_language).strip()) | |||
|
|
|||
| physical_cores = cpu_count(logical=False) | |||
| if physical_cores is None: | |||
| warnings.warn( | |||
| @@ -281,8 +290,11 @@ def main( | |||
| ) | |||
| raise ValueError(msg) | |||
|
|
|||
| validate_code_language_for_judge(code_language, judge_version) | |||
|
|
|||
There was a problem hiding this comment.
code_language defaults to cpp20 even when callers override judge_version. If a user runs ale_bench_eval with --judge_version 201907/202510 but forgets to also pass --code_language, validate_code_language_for_judge() will fail because the default language may be unsupported for that judge version. Consider defaulting code_language based on judge_version at runtime (e.g., if the CLI argument is omitted/empty) using get_default_code_language(judge_version) to make the new judge_version option ergonomic.
| # Compile error will cause by `Meta.parse` | ||
| # Some syntax errors will not cause compile error, but runtime error. |
There was a problem hiding this comment.
Grammar in this comment is off ("will cause by" / "compile error"). Consider rephrasing to something like "A compilation error is caused by Meta.parse" to keep these helper files readable.
| # Compile error will cause by `Meta.parse` | |
| # Some syntax errors will not cause compile error, but runtime error. | |
| # A compilation error is caused by `Meta.parse`. | |
| # Some syntax errors will not cause a compilation error, but will instead cause a runtime error. |
ale_bench_evaland support more coding languages without changing previous behavior