Skip to content

Add a learner-local task mode - #259

Open
jserv wants to merge 4 commits into
mainfrom
tasktrial
Open

jserv wants to merge 4 commits into
mainfrom
tasktrial

Conversation

@jserv

@jserv jserv commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Task mode lets an instructor publish a PIN-encrypted problem set on a static HTTPS site, and a learner work one task with Jim on a locally run CodeTrial using their own Gemini and LiveKit credentials. The learner signs in with GitHub (device flow), enters the PIN, chooses one of the languages the task allows, accepts the announced rules, calibrates the camera and starts in fullscreen. Leaving fullscreen or the page, or looking away past the calibrated limit, ends the attempt as invalid; a camera, detector or connection failure ends it as interrupted; Finish or the deadline completes it with a learning review that carries no hiring verdict. Everything runs on a machine the learner controls, so the encryption and the rules deter rather than prevent, and every page and export says so. Interviews are unchanged, and the contract is docs/task-mode.md.

The series is four commits: stopping a dropped Gemini session's reader, which interviews benefit from as well; raising the mutation shard ceiling to 48, which a diff this size needs; task mode itself; and letting a task name the languages a learner may choose (Python when it names none; C, C++ and Java run on Compiler Explorer, as in interviews, and the page says so).

Automated checks

./scripts/test.sh covers task mode with no credentials. The packaging tests need the cryptography package, so point PYTHON at an interpreter that has it:

python3 -m venv target/task-tools
target/task-tools/bin/pip install -r scripts/requirements-task.txt
PYTHON=target/task-tools/bin/python ./scripts/test.sh

On the rebased tip this passes with 1512 Rust tests, 1080 browser tests, the Python packaging and contract suites, clippy and formatting, and each commit builds on its own. The task mode suites are tests/unit/tasks/, tests/unit/livekit/tasks.rs, tests/web/tasks.rs, tests/browser/task-*.test.js, tests/test_task_package.py and tests/test_task_mode.py; the Rust and Python sides validate the same fixtures under tests/fixtures/task-mode/ and decrypt each other's packages.

Trying it by hand

Building a set needs no credentials. A bank is a directory with problems.json, judges.json and variants.json in the catalog's format, plus optional task-set.json (set rules) and task-mode.json (per-task sidecars, where languages lives). Two catalog problems are enough:

mkdir bank
python3 - <<'EOF'
import json
ids = ["two-sum", "valid-parentheses"]
read = lambda name: json.load(open(f"problem-bank/{name}.json"))
json.dump([p for p in read("problems") if p["id"] in ids], open("bank/problems.json", "w"))
for name in ("judges", "variants"):
    json.dump({i: read(name)[i] for i in ids}, open(f"bank/{name}.json", "w"))
d = read("task-defaults")
json.dump({"two-sum": {"promptVersion": 1, "interactionPrompt": d["interactionPrompt"],
    "completionTargets": [], "understandingChecks": d["understandingChecks"],
    "requiredCases": [], "maxHintRungs": 3, "rubricProfile": "task-engagement-v1",
    "languages": ["python", "javascript", "cpp"]}}, open("bank/task-mode.json", "w"))
EOF
PY=target/task-tools/bin/python
$PY scripts/task-package.py validate --bank bank
$PY scripts/task-package.py preview --bank bank --task-id two-sum
$PY scripts/task-package.py render-prompt --bank bank --task-id two-sum --language cpp
$PY scripts/task-package.py build --bank bank --set-id demo --version 1 \
    --site https://USER.github.io/REPO --output publish

build asks for a six-digit PIN without echo and writes publish/ with a landing page per version. Publish publish/ as the root of a GitHub Pages site (any static HTTPS host works; the server refuses plain HTTP and redirects), so --site names where it is served.

Taking the task needs a Gemini API key, LiveKit credentials and a GitHub OAuth App with device flow enabled, whose public client ID goes in GITHUB_CLIENT_ID (without one, the page falls back to the browser sign-in, which also needs GITHUB_CLIENT_SECRET). Run cargo run -- web on the default loopback address, then open the Open in CodeTrial link from the landing page, which is http://localhost:3000/t/demo/two-sum?site=...&version=1. A first run goes through Setup and returns to the link. Then, in desktop Chromium or Firefox with a camera and microphone:

  • sign in, enter the PIN, and on two-sum switch the language to see the starter, the editor label and the Compiler Explorer notice follow; valid-parentheses offers Python alone;
  • accept the rules, allow the devices, calibrate (a calibration without the typed line is refused) and Start;
  • run the public tests, ask for a hint, discuss a revision, and Finish to get the learning review and its export;
  • in further attempts, leave fullscreen, switch tabs, or look away past the limit (each ends as invalid), and stop the camera or reload the page (each ends as interrupted).

Task mode is off on a server reachable from elsewhere, one that records, or one that shares a fixed room; its routes then answer loopback_required.

Not done here: the recorded live pilot run the issue's gate asks for, against real LiveKit and Gemini with a published Pages set and real fullscreen and camera behaviour; docs/task-mode.md lists what it must cover. Release builds offer device sign-in only once a maintainer sets the repository variable CODETRIAL_GITHUB_CLIENT_ID to a GitHub OAuth App's public client ID; without it they fall back to the browser sign-in.

Close #242


Summary by cubic

Adds a learner-local task mode where an instructor publishes a PIN-encrypted problem set on a static HTTPS site and a learner works one task with Jim on a locally run CodeTrial using their own Gemini and LiveKit credentials. Interviews are unchanged, and the contract is docs/task-mode.md.

  • A task's sidecar lists the languages its learners may choose, defaulting to Python when it names none; the learner picks one before calibration, and the starter, targets, editor, runs, interviewer prompt, and result all follow it for the attempt. C, C++, and Java compile on Compiler Explorer, so the page says their runs leave the machine, and a server with it turned off offers only the task's other languages.
  • Breaking a rule (leaving fullscreen or the page, or looking away past the calibrated limit) ends the attempt as invalid; a camera, detector, or connection failure ends it as interrupted; Finish or the deadline completes it with a learning review that carries no hiring verdict.
  • One attempt runs per process, its result is held only for its own room and expires, and page actions route through the session so recovery sees that room's state.
  • Everything runs on a machine the learner controls, so the encryption and the rules deter rather than prevent, and every page and export says so. The instructor's tool starts a bank from a shipped example or a catalog problem, so a task is edited rather than written from nothing.
  • Release builds enable device sign-in only when a maintainer sets the repository variable CODETRIAL_GITHUB_CLIENT_ID; without it they fall back to browser sign-in. GitHub's signed-in user id is reported as the learner's identity only when GitHub issued it.
  • The first commit stops a dropped Gemini session's reader, which interviews benefit from too; CI installs the packaging tools in a venv outside the cached target directory, and the mutation shard ceiling rises to 48 so a diff of this size still meets the per-shard mutant budget.
  • Still missing from the issue's gate: a recorded live pilot run against real LiveKit and Gemini with a published set and real fullscreen and camera behaviour.

Written for commit a69ab16. Summary will update on new commits.

View guided diff Turn on auto-fix

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

@jserv
jserv force-pushed the tasktrial branch 2 times, most recently from 5f5bd5f to c5b0b93 Compare October 9, 2026 15:41
cubic-dev-ai[bot]

This comment was marked as resolved.

jserv added 4 commits October 10, 2026 07:19
A JoinHandle going away does not cancel its task, so a live session
dropped without shutdown, on an early return or a replaced socket, left
its reader parked on the socket, holding the connection and its
provider slot open. Dropping the session now aborts the reader;
shutdown stays the orderly close.
Sixteen shards put a diff that adds a whole subsystem far past the
design load of 25 mutants a shard, and every shard ran out of its 35
minute budget before reporting anything. Forty-eight keeps the design
load for diffs up to twelve hundred mutants; shards beyond the
account's concurrent jobs wait in the queue rather than fail. The
planning comment still said the cap was four.
Every session today is a hiring interview: the client picks the problem
and duration, and the report judges toward hire or no hire. Task mode
lets an instructor publish a PIN-encrypted problem set on a static site
and a learner work one task with Jim on their own CodeTrial, ending in
a learning review with no hiring verdict. Everything runs on a machine
the learner controls, so the encryption and the announced rules
(fullscreen, page visibility and a calibrated look-away limit) deter
rather than prevent, and every page and export says so; an attempt that
breaks a rule or loses its page is recorded as invalid or interrupted
without asking a model. The instructor's tool starts a bank from a
shipped example or a catalog problem, so a task is edited rather than
written from nothing, and builds it for a site with each task's link.
Interviews are unchanged. The contract is docs/task-mode.md.

Close #242
Task mode admitted Python alone, though the catalog ships starters in
five languages and the page can run all of them. A task's sidecar now
lists the languages its learners may choose from, Python when it names
none. The learner picks one before calibration connects the room, and
the starter, the targets offered, the editor, the runs, the
interviewer's prompt and the result all follow it for the attempt. C,
C++ and Java compile on Compiler Explorer, as in interviews, so the
page says their runs leave the machine, and a server with it turned off
offers only the task's other languages. The task page now loads the
runtime configuration, without which it could not tell, and the
instructor's tool takes the list with a --languages option, dropping
targets marked in a language it leaves out.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a task mode for guided programming exercises with a learning review

1 participant