Repository navigation
Conversation
jserv
force-pushed
the
tasktrial
branch
2 times, most recently
from
October 9, 2026 15:41
5f5bd5f to
c5b0b93
Compare
A JoinHandle going away does not cancel its task, so a live session dropped without shutdown, on an early return or a replaced socket, left its reader parked on the socket, holding the connection and its provider slot open. Dropping the session now aborts the reader; shutdown stays the orderly close.
Sixteen shards put a diff that adds a whole subsystem far past the design load of 25 mutants a shard, and every shard ran out of its 35 minute budget before reporting anything. Forty-eight keeps the design load for diffs up to twelve hundred mutants; shards beyond the account's concurrent jobs wait in the queue rather than fail. The planning comment still said the cap was four.
Every session today is a hiring interview: the client picks the problem and duration, and the report judges toward hire or no hire. Task mode lets an instructor publish a PIN-encrypted problem set on a static site and a learner work one task with Jim on their own CodeTrial, ending in a learning review with no hiring verdict. Everything runs on a machine the learner controls, so the encryption and the announced rules (fullscreen, page visibility and a calibrated look-away limit) deter rather than prevent, and every page and export says so; an attempt that breaks a rule or loses its page is recorded as invalid or interrupted without asking a model. The instructor's tool starts a bank from a shipped example or a catalog problem, so a task is edited rather than written from nothing, and builds it for a site with each task's link. Interviews are unchanged. The contract is docs/task-mode.md. Close #242
Task mode admitted Python alone, though the catalog ships starters in five languages and the page can run all of them. A task's sidecar now lists the languages its learners may choose from, Python when it names none. The learner picks one before calibration connects the room, and the starter, the targets offered, the editor, the runs, the interviewer's prompt and the result all follow it for the attempt. C, C++ and Java compile on Compiler Explorer, as in interviews, so the page says their runs leave the machine, and a server with it turned off offers only the task's other languages. The task page now loads the runtime configuration, without which it could not tell, and the instructor's tool takes the list with a --languages option, dropping targets marked in a language it leaves out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task mode lets an instructor publish a PIN-encrypted problem set on a static HTTPS site, and a learner work one task with Jim on a locally run CodeTrial using their own Gemini and LiveKit credentials. The learner signs in with GitHub (device flow), enters the PIN, chooses one of the languages the task allows, accepts the announced rules, calibrates the camera and starts in fullscreen. Leaving fullscreen or the page, or looking away past the calibrated limit, ends the attempt as invalid; a camera, detector or connection failure ends it as interrupted; Finish or the deadline completes it with a learning review that carries no hiring verdict. Everything runs on a machine the learner controls, so the encryption and the rules deter rather than prevent, and every page and export says so. Interviews are unchanged, and the contract is docs/task-mode.md.
The series is four commits: stopping a dropped Gemini session's reader, which interviews benefit from as well; raising the mutation shard ceiling to 48, which a diff this size needs; task mode itself; and letting a task name the languages a learner may choose (Python when it names none; C, C++ and Java run on Compiler Explorer, as in interviews, and the page says so).
Automated checks
./scripts/test.shcovers task mode with no credentials. The packaging tests need thecryptographypackage, so pointPYTHONat an interpreter that has it:On the rebased tip this passes with 1512 Rust tests, 1080 browser tests, the Python packaging and contract suites, clippy and formatting, and each commit builds on its own. The task mode suites are
tests/unit/tasks/,tests/unit/livekit/tasks.rs,tests/web/tasks.rs,tests/browser/task-*.test.js,tests/test_task_package.pyandtests/test_task_mode.py; the Rust and Python sides validate the same fixtures undertests/fixtures/task-mode/and decrypt each other's packages.Trying it by hand
Building a set needs no credentials. A bank is a directory with
problems.json,judges.jsonandvariants.jsonin the catalog's format, plus optionaltask-set.json(set rules) andtask-mode.json(per-task sidecars, wherelanguageslives). Two catalog problems are enough:buildasks for a six-digit PIN without echo and writespublish/with a landing page per version. Publishpublish/as the root of a GitHub Pages site (any static HTTPS host works; the server refuses plain HTTP and redirects), so--sitenames where it is served.Taking the task needs a Gemini API key, LiveKit credentials and a GitHub OAuth App with device flow enabled, whose public client ID goes in
GITHUB_CLIENT_ID(without one, the page falls back to the browser sign-in, which also needsGITHUB_CLIENT_SECRET). Runcargo run -- webon the default loopback address, then open the Open in CodeTrial link from the landing page, which ishttp://localhost:3000/t/demo/two-sum?site=...&version=1. A first run goes through Setup and returns to the link. Then, in desktop Chromium or Firefox with a camera and microphone:two-sumswitch the language to see the starter, the editor label and the Compiler Explorer notice follow;valid-parenthesesoffers Python alone;Task mode is off on a server reachable from elsewhere, one that records, or one that shares a fixed room; its routes then answer
loopback_required.Not done here: the recorded live pilot run the issue's gate asks for, against real LiveKit and Gemini with a published Pages set and real fullscreen and camera behaviour; docs/task-mode.md lists what it must cover. Release builds offer device sign-in only once a maintainer sets the repository variable
CODETRIAL_GITHUB_CLIENT_IDto a GitHub OAuth App's public client ID; without it they fall back to the browser sign-in.Close #242
Summary by cubic
Adds a learner-local task mode where an instructor publishes a PIN-encrypted problem set on a static HTTPS site and a learner works one task with Jim on a locally run CodeTrial using their own Gemini and LiveKit credentials. Interviews are unchanged, and the contract is
docs/task-mode.md.CODETRIAL_GITHUB_CLIENT_ID; without it they fall back to browser sign-in. GitHub's signed-in user id is reported as the learner's identity only when GitHub issued it.Written for commit a69ab16. Summary will update on new commits.