Skip to content

Commit e787014

Browse files
viratzzsviratzzs
committed
Initial commit: parsewave/ProgramBench-experiments
Fork of facebookresearch/programbench v1.0.1, with the parsewave agent harness, prompts, scripts, and rollout traces overlaid. Headline experiment: best-of-5 evaluation of Claude Opus 4.7 on the easy-10 ProgramBench subset (framework-C doctrine). Average pass rate 99, 2 confirmed full solves (cmatrix 508/508, blake3 647/647). See example-outputs/README-best-of-5.md for the scoreboard and example-outputs/cc-opus-easy10-v2-C-{1..5}/ for the 50 task traces. Internal Parsewave infrastructure references (proxy IPs, hostnames, internal repo paths) in the trace data and harness are redacted to placeholders like <REDACTED_PROXY_IP_1>.
0 parents  commit e787014

874 files changed

Lines changed: 1912497 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/CODE_OF_CONDUCT.md‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
2+
# Contributor Covenant Code of Conduct
3+
4+
## Our Pledge
5+
6+
We as members, contributors, and leaders pledge to make participation in our
7+
community a harassment-free experience for everyone, regardless of age, body
8+
size, visible or invisible disability, ethnicity, sex characteristics, gender
9+
identity and expression, level of experience, education, socio-economic status,
10+
nationality, personal appearance, race, caste, color, religion, or sexual
11+
identity and orientation.
12+
13+
We pledge to act and interact in ways that contribute to an open, welcoming,
14+
diverse, inclusive, and healthy community.
15+
16+
## Our Standards
17+
18+
Examples of behavior that contributes to a positive environment for our
19+
community include:
20+
21+
* Demonstrating empathy and kindness toward other people
22+
* Being respectful of differing opinions, viewpoints, and experiences
23+
* Giving and gracefully accepting constructive feedback
24+
* Accepting responsibility and apologizing to those affected by our mistakes,
25+
and learning from the experience
26+
* Focusing on what is best not just for us as individuals, but for the overall
27+
community
28+
29+
Examples of unacceptable behavior include:
30+
31+
* The use of sexualized language or imagery, and sexual attention or advances of
32+
any kind
33+
* Trolling, insulting or derogatory comments, and personal or political attacks
34+
* Public or private harassment
35+
* Publishing others' private information, such as a physical or email address,
36+
without their explicit permission
37+
* Other conduct which could reasonably be considered inappropriate in a
38+
professional setting
39+
40+
## Enforcement Responsibilities
41+
42+
Community leaders are responsible for clarifying and enforcing our standards of
43+
acceptable behavior and will take appropriate and fair corrective action in
44+
response to any behavior that they deem inappropriate, threatening, offensive,
45+
or harmful.
46+
47+
Community leaders have the right and responsibility to remove, edit, or reject
48+
comments, commits, code, wiki edits, issues, and other contributions that are
49+
not aligned to this Code of Conduct, and will communicate reasons for moderation
50+
decisions when appropriate.
51+
52+
## Scope
53+
54+
This Code of Conduct applies within all community spaces, and also applies when
55+
an individual is officially representing the community in public spaces.
56+
Examples of representing our community include using an official email address,
57+
posting via an official social media account, or acting as an appointed
58+
representative at an online or offline event.
59+
60+
## Enforcement
61+
62+
Instances of abusive, harassing, or otherwise unacceptable behavior may be
63+
reported to the community leaders responsible for enforcement at
64+
[INSERT CONTACT METHOD].
65+
All complaints will be reviewed and investigated promptly and fairly.
66+
67+
All community leaders are obligated to respect the privacy and security of the
68+
reporter of any incident.
69+
70+
## Enforcement Guidelines
71+
72+
Community leaders will follow these Community Impact Guidelines in determining
73+
the consequences for any action they deem in violation of this Code of Conduct:
74+
75+
### 1. Correction
76+
77+
**Community Impact**: Use of inappropriate language or other behavior deemed
78+
unprofessional or unwelcome in the community.
79+
80+
**Consequence**: A private, written warning from community leaders, providing
81+
clarity around the nature of the violation and an explanation of why the
82+
behavior was inappropriate. A public apology may be requested.
83+
84+
### 2. Warning
85+
86+
**Community Impact**: A violation through a single incident or series of
87+
actions.
88+
89+
**Consequence**: A warning with consequences for continued behavior. No
90+
interaction with the people involved, including unsolicited interaction with
91+
those enforcing the Code of Conduct, for a specified period of time. This
92+
includes avoiding interactions in community spaces as well as external channels
93+
like social media. Violating these terms may lead to a temporary or permanent
94+
ban.
95+
96+
### 3. Temporary Ban
97+
98+
**Community Impact**: A serious violation of community standards, including
99+
sustained inappropriate behavior.
100+
101+
**Consequence**: A temporary ban from any sort of interaction or public
102+
communication with the community for a specified period of time. No public or
103+
private interaction with the people involved, including unsolicited interaction
104+
with those enforcing the Code of Conduct, is allowed during this period.
105+
Violating these terms may lead to a permanent ban.
106+
107+
### 4. Permanent Ban
108+
109+
**Community Impact**: Demonstrating a pattern of violation of community
110+
standards, including sustained inappropriate behavior, harassment of an
111+
individual, or aggression toward or disparagement of classes of individuals.
112+
113+
**Consequence**: A permanent ban from any sort of public interaction within the
114+
community.
115+
116+
## Attribution
117+
118+
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
119+
version 2.1, available at
120+
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
121+
122+
Community Impact Guidelines were inspired by
123+
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
124+
125+
For answers to common questions about this code of conduct, see the FAQ at
126+
[https://www.contributor-covenant.org/faq][FAQ].
127+
128+
[homepage]: https://www.contributor-covenant.org
129+
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
130+
[Mozilla CoC]: https://github.com/mozilla/diversity
131+
[FAQ]: https://www.contributor-covenant.org/faq

‎.github/bump_version.py‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
#!/usr/bin/env python3
2+
"""Bump the programbench version in pyproject.toml."""
3+
4+
import re
5+
import subprocess
6+
import sys
7+
from pathlib import Path
8+
9+
PYPROJECT = Path(__file__).resolve().parents[1] / "pyproject.toml"
10+
VERSION_RE = re.compile(r'^version\s*=\s*"([^"]+)"', re.MULTILINE)
11+
12+
13+
def main() -> None:
14+
text = PYPROJECT.read_text()
15+
match = VERSION_RE.search(text)
16+
if not match:
17+
print("Error: could not find version in", PYPROJECT)
18+
sys.exit(1)
19+
20+
current = match.group(1)
21+
print(f"Current version: {current}")
22+
23+
new = input("New version: ").strip()
24+
if not new:
25+
print("No version entered, aborting.")
26+
sys.exit(1)
27+
28+
updated = text[: match.start(1)] + new + text[match.end(1) :]
29+
PYPROJECT.write_text(updated)
30+
print(f"Updated {PYPROJECT.relative_to(Path.cwd())}: {current} -> {new}")
31+
32+
tag = f"v{new}"
33+
34+
subprocess.run(["git", "add", str(PYPROJECT)], check=True)
35+
subprocess.run(["git", "commit", "-m", "Bump version"], check=True)
36+
print("Added version bump as commit")
37+
38+
subprocess.run(["git", "tag", tag], check=True)
39+
print(f"Created tag {tag}")
40+
41+
print(f"\nTo push:\n git push && git push origin {tag}")
42+
43+
44+
if __name__ == "__main__":
45+
main()

‎.github/dependabot.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
version: 2
2+
updates:
3+
# Maintain dependencies for GitHub Actions
4+
- package-ecosystem: "github-actions"
5+
directory: "/"
6+
schedule:
7+
interval: "weekly"

‎.github/mlc_config.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"ignorePatterns": [
3+
{
4+
"pattern": "https://github.com/?.*"
5+
},
6+
{
7+
"pattern": "https://platform.openai.com/docs/.*"
8+
},
9+
{
10+
"pattern": ".*localhost.*"
11+
},
12+
{
13+
"pattern": "^(?!https).*(?<!\\.md)$"
14+
}
15+
]
16+
}

‎.github/workflows/check-links-periodic.yaml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
name: Check Markdown links full
2+
3+
on:
4+
workflow_dispatch:
5+
push:
6+
branches:
7+
- main
8+
- check-links*
9+
- fix-links*
10+
schedule:
11+
- cron: "0 0,12 * * *"
12+
13+
jobs:
14+
markdown-link-check:
15+
runs-on: ubuntu-latest
16+
steps:
17+
- uses: actions/checkout@v6
18+
- uses: tcort/github-action-markdown-link-check@v1
19+
with:
20+
use-verbose-mode: 'yes'
21+
config-file: .github/mlc_config.json

‎.github/workflows/check-links-pr.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
on: [pull_request]
2+
name: Check Markdown links in modified files
3+
jobs:
4+
markdown-link-check:
5+
runs-on: ubuntu-latest
6+
steps:
7+
- uses: actions/checkout@v6
8+
- uses: tcort/github-action-markdown-link-check@v1
9+
with:
10+
use-verbose-mode: 'yes'
11+
check-modified-files-only: 'yes'
12+
base-branch: 'main'
13+
config-file: .github/mlc_config.json

‎.github/workflows/release.yaml‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
name: Release to PyPI
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
test_pypi:
7+
description: 'Upload to Test PyPI instead of PyPI'
8+
required: false
9+
default: false
10+
type: boolean
11+
release:
12+
types: [published]
13+
14+
jobs:
15+
build-and-publish:
16+
runs-on: ubuntu-latest
17+
environment: release
18+
19+
steps:
20+
- name: Checkout code
21+
uses: actions/checkout@v6
22+
23+
- name: Set up Python
24+
uses: actions/setup-python@v6
25+
with:
26+
python-version: '3.13'
27+
28+
- name: Check version matches release tag
29+
if: github.event_name == 'release'
30+
run: |
31+
# Extract version from pyproject.toml using regex (no imports needed)
32+
PACKAGE_VERSION=$(grep -oP '^version\s*=\s*"\K[^"]+' pyproject.toml)
33+
34+
# Extract tag name (remove 'v' prefix if present)
35+
TAG_NAME="${{ github.event.release.tag_name }}"
36+
TAG_VERSION="${TAG_NAME#v}"
37+
38+
echo "Package version: $PACKAGE_VERSION"
39+
echo "Release tag: $TAG_NAME"
40+
echo "Tag version (normalized): $TAG_VERSION"
41+
42+
if [ "$PACKAGE_VERSION" != "$TAG_VERSION" ]; then
43+
echo "ERROR: Version mismatch!"
44+
echo "Package version ($PACKAGE_VERSION) does not match release tag ($TAG_VERSION)"
45+
echo "Please update version in pyproject.toml to match the release tag"
46+
exit 1
47+
fi
48+
49+
echo "Version check passed: $PACKAGE_VERSION matches $TAG_VERSION"
50+
51+
- name: Install uv
52+
run: |
53+
curl -LsSf https://astral.sh/uv/install.sh | sh
54+
- name: Install build dependencies
55+
run: |
56+
uv pip install --python ${Python_ROOT_DIR} build twine
57+
58+
- name: Build package
59+
run: python -m build
60+
61+
- name: Check package
62+
run: python -m twine check dist/*
63+
64+
- name: Upload to Test PyPI
65+
if: ${{ github.event.inputs.test_pypi == 'true' }}
66+
uses: pypa/gh-action-pypi-publish@release/v1
67+
with:
68+
password: ${{ secrets.TEST_PYPI_API_TOKEN }}
69+
repository-url: https://test.pypi.org/legacy/
70+
verbose: true
71+
72+
- name: Upload to PyPI
73+
if: ${{ github.event.inputs.test_pypi != 'true' }}
74+
uses: pypa/gh-action-pypi-publish@release/v1
75+
with:
76+
password: ${{ secrets.PYPI_API_TOKEN }}
77+
verbose: true
78+
79+
smoke-test:
80+
runs-on: ubuntu-latest
81+
needs: build-and-publish
82+
83+
steps:
84+
- name: Set up Python
85+
uses: actions/setup-python@v6
86+
with:
87+
python-version: '3.13'
88+
89+
- name: Install uv and pipx
90+
run: |
91+
curl -LsSf https://astral.sh/uv/install.sh | sh
92+
uv pip install --python ${Python_ROOT_DIR} pipx
93+
94+
- name: Wait for package availability
95+
run: |
96+
echo "Waiting for package to be available on PyPI..."
97+
sleep 60
98+
99+
- name: Test pipx run programbench --help (Test PyPI)
100+
if: ${{ github.event.inputs.test_pypi == 'true' }}
101+
run: pipx run --pip-args="--index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/" programbench --help
102+
103+
- name: Test pipx run programbench --help (PyPI)
104+
if: ${{ github.event.inputs.test_pypi != 'true' }}
105+
run: pipx run programbench --help
106+
107+
- name: Test uvx programbench --help (Test PyPI)
108+
if: ${{ github.event.inputs.test_pypi == 'true' }}
109+
run: uvx --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ programbench --help
110+
111+
- name: Test uvx programbench --help (PyPI)
112+
if: ${{ github.event.inputs.test_pypi != 'true' }}
113+
run: uvx programbench --help
114+
115+
- name: Test pip install and CLI (Test PyPI)
116+
if: ${{ github.event.inputs.test_pypi == 'true' }}
117+
run: |
118+
uv pip install --python ${Python_ROOT_DIR} --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ programbench
119+
programbench --help
120+
121+
- name: Test pip install and CLI (PyPI)
122+
if: ${{ github.event.inputs.test_pypi != 'true' }}
123+
run: |
124+
uv pip install --python ${Python_ROOT_DIR} programbench
125+
programbench --help

‎.github/workflows/test.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
name: Test
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
8+
jobs:
9+
pytest:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- uses: actions/checkout@v6
13+
- uses: actions/setup-python@v6
14+
with:
15+
python-version: "3.11"
16+
- run: pip install -e ".[dev]" && pip install pytest
17+
- run: pytest
18+
19+
pre-commit:
20+
runs-on: ubuntu-latest
21+
steps:
22+
- uses: actions/checkout@v6
23+
- uses: actions/setup-python@v6
24+
with:
25+
python-version: "3.11"
26+
- uses: pre-commit/action@v3.0.1

0 commit comments

Comments
 (0)