|
| 1 | +# framework-C — clean-room reverse engineering |
| 2 | + |
| 3 | +Task contract: |
| 4 | + |
| 5 | +- Reference: `/workspace/executable` (read + execute only) |
| 6 | +- Source: write to `/work/work/` |
| 7 | +- Build: `/work/work/compile.sh` must produce `/work/work/executable` |
| 8 | +- Grading: per-test pass rate over a hidden test suite |
| 9 | +- Budget: 4 hours per task, one submission |
| 10 | + |
| 11 | +--- |
| 12 | + |
| 13 | +## Approach |
| 14 | + |
| 15 | +Recover the reference binary's behavioral specification by composing four |
| 16 | +established black-box methodologies: |
| 17 | + |
| 18 | +- Category-Partition Method (Ostrand & Balcer, *CACM* 1988) |
| 19 | +- Equivalence Partitioning + Boundary Value Analysis + Error Guessing |
| 20 | + (Myers, *The Art of Software Testing*, 1979) |
| 21 | +- Differential Testing (McKeeman, *Digital Technical Journal* 1998; |
| 22 | + Csmith — Yang et al., *PLDI* 2011) |
| 23 | +- Clean-room Software Engineering (Mills, Dyer, Linger, |
| 24 | + *IEEE Software* 1987) |
| 25 | + |
| 26 | +The reference binary IS the oracle. Do not invent properties. Do not |
| 27 | +catch errors from the reference — propagate them as spec. Every behavior |
| 28 | +the reference exhibits is part of the specification: do not scope a |
| 29 | +behavior out on the assumption it will not be exercised. When a behavior's |
| 30 | +exact bytes are unstable (animations, streaming output, timestamps, |
| 31 | +randomized fields), reproduce it structurally — match whether it produces |
| 32 | +output, how much, in what shape, and whether it terminates or runs until |
| 33 | +signaled — rather than omitting it. |
| 34 | + |
| 35 | +--- |
| 36 | + |
| 37 | +## Hard rule on stdin-reading binaries (reference AND your own SUT) |
| 38 | + |
| 39 | +Any binary that may read stdin — the reference at `/workspace/executable` |
| 40 | +OR your own SUT at `/work/work/executable` (or any intermediate target |
| 41 | +under `/work/work/target/`) — MUST be invoked with both `timeout 5` AND |
| 42 | +either `< /dev/null` or piped deliberate input via `printf ... |`. This |
| 43 | +applies in TWO places, BOTH of which have hung the harness for 1.5-2 h |
| 44 | +in prior runs: |
| 45 | + |
| 46 | +### 1. Direct bash invocations |
| 47 | + |
| 48 | +Many CLI modes — `-c` (check mode, reads checksums from stdin), `--keyed` |
| 49 | +(reads a key from stdin), `-`/`--` (read text from stdin), and many flags |
| 50 | +that switch the binary into "read input" mode — block forever on bare |
| 51 | +invocation. The bash tool has no per-call timeout and waits indefinitely |
| 52 | +for the hung command to return. |
| 53 | + |
| 54 | +Bad: `/workspace/executable -c` |
| 55 | +Good: `timeout 5 /workspace/executable -c < /dev/null` |
| 56 | +Good: `printf 'abcd |
| 57 | +' | timeout 5 /workspace/executable -c` |
| 58 | + |
| 59 | +### 2. Subprocess calls inside Python helper scripts you write |
| 60 | + |
| 61 | +If you write `topup.py`, `full_diff.py`, `fuzz_diff.py`, or any other |
| 62 | +helper that runs the SUT or REF in a loop via `subprocess.run` / |
| 63 | +`subprocess.Popen`, EVERY such call MUST pass both `timeout=5` AND |
| 64 | +`stdin=subprocess.DEVNULL` (unless you are explicitly feeding stdin |
| 65 | +via `input=`). A single hung subprocess inside the helper will block |
| 66 | +the entire Python script. If you then run that script as |
| 67 | +`python3 helper.py 2>&1 | tail -3`, `tail` blocks on EOF and the whole |
| 68 | +bash call hangs — same outcome, more layers of blame. |
| 69 | + |
| 70 | +Bad: `subprocess.run([SUT, "--version"], capture_output=True)` |
| 71 | +Good: `subprocess.run([SUT, "--version"], capture_output=True, timeout=5, stdin=subprocess.DEVNULL)` |
| 72 | + |
| 73 | +This rule takes precedence over all other procedural guidance. Apply it |
| 74 | +to every probe and every test-driver script before considering anything |
| 75 | +else. |
| 76 | + |
| 77 | +--- |
| 78 | + |
| 79 | +## Procedure |
| 80 | + |
| 81 | +Before writing any reimplementation code: |
| 82 | + |
| 83 | +1. **Surface scan.** Run `B --help`, `B`, `B --version`, `B --bogus`, |
| 84 | + `B -`, `B --`. Record bytes verbatim. Exercise the reference in the |
| 85 | + full I/O environment a user supplies — an attached terminal and an |
| 86 | + interactive stdin, not only redirected files and pipes — since a |
| 87 | + program's behavior can branch on what it is connected to. The process |
| 88 | + model (one-shot, blocking on input, streaming, or running until |
| 89 | + signaled) is part of the spec. |
| 90 | +2. **Category-Partition.** List parameters and environment dimensions; |
| 91 | + per dimension list choices including {empty, boundary, hostile, |
| 92 | + invalid-type}; write the constraint list. Treat each environment |
| 93 | + variable the reference reads as its own dimension and probe it |
| 94 | + independently of the CLI flags. Cap probes at ~30 via a pairwise |
| 95 | + covering array (Kuhn, Kacker, Lei — NIST SP 800-142, 2010). |
| 96 | +3. **EP / BVA / EG layer.** For every numeric or length range, probe |
| 97 | + {min−1, min, min+1, max−1, max, max+1}. Include the canonical |
| 98 | + black-box probes: empty arg, only `--`, only `-`, repeated flag, |
| 99 | + conflicting flags, missing required, extra positional, NUL / BOM / |
| 100 | + CRLF / invalid-UTF-8 / control bytes on stdin, broken pipe. |
| 101 | +4. **Freeze the oracle.** For each probe record the full triple — exit |
| 102 | + code, stdout bytes, AND stderr bytes. Error-message templates on |
| 103 | + stderr and exit codes are first-class byte targets, frozen as exactly |
| 104 | + as stdout. Persist to `/work/work/frames.jsonl`. |
| 105 | +5. **Differential loop.** Build the reimplementation; run every frame |
| 106 | + against both binaries; on divergence, shrink (halve stdin, drop |
| 107 | + flags) to a minimal failing frame, fix, re-run the full bank. |
| 108 | +6. **Statistical top-up, then stop.** Before submission, sample 10 fresh |
| 109 | + probes from a usage Markov chain (Whittaker & Thomason, *IEEE TSE* |
| 110 | + 1994) over the discovered modes. When the reimplementation matches the |
| 111 | + reference on all 10, the spec is certified — submit immediately. |
| 112 | + Re-verification, re-fuzzing, or cleanup after a clean top-up does not |
| 113 | + raise certified reliability and only forfeits budget. |
| 114 | + |
| 115 | +--- |
| 116 | + |
| 117 | +## Probing tactics |
| 118 | + |
| 119 | +Standard tooling for black-box CLI investigation, used in service of |
| 120 | +the procedure above: |
| 121 | + |
| 122 | +- `file <binary>` — identify binary type and language runtime. Match the |
| 123 | + reference's source language when it links a library whose runtime |
| 124 | + contract a foreign-language reimplementation cannot reproduce. |
| 125 | +- `strings <binary> | grep -i <pattern>` — discover embedded library |
| 126 | + identifiers, version strings, error templates, format strings. |
| 127 | +- `od -c` or `hexdump -C` on diverging outputs — surface invisible-byte |
| 128 | + differences (trailing newlines, NULs, CRLF, BOM, ANSI escapes). |
| 129 | +- `diff <(REF args) <(SUT args)` — single-shot byte comparison; for |
| 130 | + large outputs combine with `head -c N` or `md5sum`. |
| 131 | +- Determinism check: run the reference twice on the same input and |
| 132 | + diff. If outputs differ (timestamps, IDs, random padding), reproduce |
| 133 | + the variability mechanism or extract stable invariants before |
| 134 | + freezing the oracle. |
| 135 | +- Wrap probes in `timeout <seconds>` for any invocation that may stream, |
| 136 | + block on input, or run until signaled. |
0 commit comments