added attack lab by 777lefty · Pull Request #114 · sys-intelligence/system-intelligence-benchmark

777lefty · 2026-02-02T17:44:34Z

Description

Added attack lab for CMU 15-213

Changes

added attack lab

Testing

Tested with qwen3-coder-plus agent

Checklist

Tests pass locally
Code follows project style guidelines
Documentation updated (if needed)

Distinguish the models used in the executor and evaluator

Signed-off-by: Tarek <tareknaser360@gmail.com>

…m changes

…s/sysmobench/sysmobench_core'

- Add gpt-4o model configuration to models.yaml - Fix setup_tools.py to use shutil.move instead of os.rename This resolves 'Invalid cross-device link' error when /tmp is on different filesystem

* Add UIUC CS423 Fall 2025 Exams * Fix json parsing * Fix backtick * Fix backtick 2 * Update benchmarks/courseexam_bench/data/cs_423_operating_systems_design_fall_2025_midterm/MP1.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update benchmarks/courseexam_bench/data/cs_423_operating_systems_design_fall_2025_final/exam.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Xuan Feng <xfeng9209@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* add ostep processes-shell lab * fix * Update benchmarks/courselab_bench/data/cs537-projects-spring-2019/processes_shell/task.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Tarek Elsayed <60650661+tareknaser@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Signed-off-by: Tarek <tareknaser360@gmail.com>

* added cmu15-213 data lab * docs(courselab): add note about infrastructure restrictions Signed-off-by: Tarek <tareknaser360@gmail.com> --------- Signed-off-by: Tarek <tareknaser360@gmail.com> Co-authored-by: Tarek <tareknaser360@gmail.com>

* add cs537 fall 2021 final exam * add institution * fix * add solutions * update metadata * add choice array * avoid extra restrictions on LLM output Signed-off-by: Tarek <tareknaser360@gmail.com> --------- Signed-off-by: Tarek <tareknaser360@gmail.com> Co-authored-by: Tarek <tareknaser360@gmail.com>

Signed-off-by: Tarek <tareknaser360@gmail.com>

[Bug-fix] Patching ArtEvalBench's Acto (SOSP'23) entry

tareknaser

Please add a reference solution script

tareknaser · 2026-02-03T17:51:02Z

benchmarks/courselab_bench/data/cmu_15-213/task_attack_lab/starter/README.txt

Is this file part of the task?

xuafeng and others added 30 commits November 5, 2025 18:10

Rename it "System Intelligence Benchmark"

6d24e69

Init: Initialize SysMoBench benchmark integration

87db1e9

feat: Add gitigore

69f4cb5

feat: Add prototype for phase 1&2

843f031

feat: Distinguish evaluator and model API keys in env.toml

0d2b38f

feat: Add validation for required evaluator API keys

b2acaa7

doc: update README.md

ca7e72e

initial ArtEval commit

ec7b57f

Merge pull request #2 from systemintelligence/feat/distinguish-api-keys

a607e73

Distinguish the models used in the executor and evaluator

feat: Add test

60d30e0

featr: Add install.sh

ff96313

adding overview and contributor's guide

1799370

skeleton ArtEval agent implementation

2054314

adding sosp24 wasabi

6303aa5

docs: add arteval to main README

a5358dc

Signed-off-by: Tarek <tareknaser360@gmail.com>

feat(ci): add GH Actions workflow for running benchmarks tests

904374e

Signed-off-by: Tarek <tareknaser360@gmail.com>

feat: add issue and pull request templates

40ccf1f

Signed-off-by: Tarek <tareknaser360@gmail.com>

fix(ci): add a test for example_bench

4130c7a

Signed-off-by: Tarek <tareknaser360@gmail.com>

fix: shell scripts to be executable

3af5b70

Signed-off-by: Tarek <tareknaser360@gmail.com>

docs: update README with instructions for running a single benchmark

156c77c

Signed-off-by: Tarek <tareknaser360@gmail.com>

docs: a note on docker image arch support

a0557f9

Signed-off-by: Tarek <tareknaser360@gmail.com>

meta: add outputs directories to gitignore

868da59

Signed-off-by: Tarek <tareknaser360@gmail.com>

feat(ci): add release trigger to workflow

ea9b54d

Signed-off-by: Tarek <tareknaser360@gmail.com>

fix: Use tla_specification instead of generated_text to adapt upstrea…

5ef835f

…m changes

Merge commit '04900168e10834f3aa5eef4d13b318e1efcdac24' as 'benchmark…

c5dfbb1

…s/sysmobench/sysmobench_core'

fix: Add gpt-4o config and fix cross-device link issue in setup_tools

a68e171

- Add gpt-4o model configuration to models.yaml - Fix setup_tools.py to use shutil.move instead of os.rename This resolves 'Invalid cross-device link' error when /tmp is on different filesystem

fix: Convert GenerationOutput to GenerationResult for evaluators

984336a

docs: Update README and install script for Git Subtree integration

68025ca

feat: Add docker file

25c6af8

fix: Add env.toml

97c1c3c

paizhangliu and others added 26 commits January 9, 2026 17:01

fix: delete git ignored generated files

0bf0741

Signed-off-by: Tarek <tareknaser360@gmail.com>

courseexam: add 6.5840 spring 2025 exam 1

436242c

Signed-off-by: Tarek <tareknaser360@gmail.com>

[lab] added cmu15-213 "data lab" (#62)

5c6b412

* added cmu15-213 data lab * docs(courselab): add note about infrastructure restrictions Signed-off-by: Tarek <tareknaser360@gmail.com> --------- Signed-off-by: Tarek <tareknaser360@gmail.com> Co-authored-by: Tarek <tareknaser360@gmail.com>

correct the number of questions for cs537_fall_2021_final

db8c391

Signed-off-by: Tarek <tareknaser360@gmail.com>

feat(courseexam): add support for evaluation infrastructure

6a42caa

Signed-off-by: Tarek <tareknaser360@gmail.com>

feat(courseexam): deprecate points_mean metric

cacf763

Signed-off-by: Tarek <tareknaser360@gmail.com>

feat(ci): add experimental courseexam benchmark validation workflow

fd446d6

Signed-off-by: Tarek <tareknaser360@gmail.com>

docs(ci): include a link to inspect log viewer docs

619151e

Signed-off-by: Tarek <tareknaser360@gmail.com>

feat(courselab_bench): refactor infrastructure to use inspect-ai

488e8eb

Signed-off-by: Tarek <tareknaser360@gmail.com>

fix(courselab): handle binary starter files separately

c6cf5d1

feat: update versioning for courseexam and courselab benchmarks

376b056

fix: compute image name as the SHA256 of the file itself

16077b4

docs: add note on private contributions and access to the repository

ba92c3c

feat(courselab): add solution validation using a reference script

0ce3ef8

feat(courselab): add reference solutions for existing labs

5b08284

feat(courselab): add more verbose tags

1ef35f4

feat(courselab): specify git commit when cloning

fe752b8

feat(courselab): specify go binary path in task description

1fff943

docs(courselab): add a section in the README for best practices

40b1061

docs: mention the leaderboard for courseexam and courselab benchmarks

c3801f9

Merge branch 'sys-intelligence:main' into main

8b05322

Merge pull request #45 from bastoica/main

29f114d

[Bug-fix] Patching ArtEvalBench's Acto (SOSP'23) entry

added attack lab

75d74fb

xuafeng mentioned this pull request Feb 2, 2026

Add CMU 15-213 #99

Open

xuafeng requested a review from tareknaser February 3, 2026 01:49

tareknaser reviewed Feb 3, 2026

View reviewed changes

benchmarks/courselab_bench/data/cmu_15-213/task_attack_lab/starter/README.txt

Copy link

Collaborator

tareknaser Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this file part of the task?

tareknaser force-pushed the main branch from 57b962d to a1780ed Compare February 5, 2026 16:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added attack lab#114

added attack lab#114
777lefty wants to merge 200 commits intomainfrom
attack_lab

777lefty commented Feb 2, 2026

Uh oh!

tareknaser left a comment

Uh oh!

tareknaser Feb 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Conversation

777lefty commented Feb 2, 2026

Description

Changes

Testing

Checklist

Uh oh!

tareknaser left a comment

Choose a reason for hiding this comment

Uh oh!

tareknaser Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants