# falsepass Probe RL environment verifiers for **false passes** (a known-bad completion is rewarded) and **false fails** (a known-good completion is not). In one public environment, echoing the question back earned full reward on every task: each question listed all of its candidate answers, and the grader only checked that the right one appeared somewhere in the reply, so a policy could collect full reward by listing everything. falsepass finds this kind of hole before training does. An RL environment is a task dataset plus a verifier that turns a model's output into reward. `falsepass` treats that verifier as the system under test: it calls the environment's own reward code on completions whose correctness is known in advance and reports every disagreement. No model, no rollouts, no GPU. It works with three frameworks, published or still in a local folder: | Framework | What falsepass runs | `falsepass init …` | |---|---|---| | [verifiers](https://github.com/PrimeIntellect-ai/verifiers) (Prime Intellect Environments Hub), all three API generations | the environment's rubric or taskset | `owner/environment`, or `./folder` | | [Inspect AI](https://inspect.aisi.org.uk) | the task's solver chain with a scripted model, then its scorers | `path/to/task.py`, `./project`, or `inspect_evals/gsm8k` | | [HUD](https://docs.hud.ai) | the task template's grader, driven like HUD's server | `path/to/env-folder` (add `--answers refs.json` for every probe) | ## Install Needs [uv](https://docs.astral.sh/uv/) on the PATH (every audit runs in its own throw-away virtualenv) and Python 3.11 or newer. ```bash uvx falsepass demo # try it, nothing to install uv tool install falsepass # or keep `falsepass` on the PATH pip install falsepass # or into an existing environment uvx --from git+https://github.com/Super-Eric/falsepass falsepass --version # the unreleased main branch ``` ## Quick start ```bash falsepass demo # audits a bundled toy environment with one planted defect (about a minute) falsepass init owner/environment # a Hub environment: package, version and verifiers pin read from the Hub falsepass init ./my-environment # or your own, from its folder (pyproject.toml); edits are picked up on every run falsepass audit my-environment # probe it; results in audits/my-environment/summary.md falsepass results my-environment --json # headline numbers and failing probes, for scripts and agents ``` `demo` writes the toy environment to `./falsepass-demo` and audits it. Its grader pays when the reference appears anywhere in the answer, and the summary shows that one line from several sides: the static scan points at it, and the probes that list several answers (P5), retract the answer (P9) or deny it inside the answer slot (P15) are paid. A condensed excerpt of the real output (full file: `examples/toy-env/summary.md`): ``` - False-pass ceiling: 1 of 1 (highest reward given to a known-bad completion; via P5 shotgun/all_in_one_slot) - Lazy-policy reward: 0 (best average reward of one fixed strategy that never uses the reference answer; via P3 empty/empty) - False-fail rate: 0.108 (93 known-good completions) | Probe | Variant | Expect | Runs | Failed | Score min / mean / max | | P5 shotgun | all_in_one_slot | zero | 10 | 10 | 1 / 1 / 1 | | P9 oversized | correct_then_retracted | zero | 10 | 10 | 1 / 1 / 1 | | P15 hedged_in_slot | denied | zero | 10 | 10 | 1 / 1 / 1 | Static hint: falsepass_toy_env.py:26 reference_substring_match return 1.0 if answer in reply else 0.0 confirmed: P9 failed 10/20, P5 failed 10/30 ``` ## Inspect AI and HUD ```bash falsepass init inspect_evals/gsm8k # a task from inspect_evals (pinned from PyPI) falsepass init my_eval.py --task hard_split # a task file; --task picks one of several falsepass init ./my-hud-env # a HUD environment folder (env.py + tasks.py) falsepass init ./my-hud-env --answers refs.json # with reference answers: every probe runs ``` Inspect: the scripted model replaces only the call to `generate()`; prompt templates, system messages and answer parsing (`multiple_choice()`) are the task's own code. A solver or scorer that calls a model itself (self-critique, `model_graded_qa`) is flagged on every row instead of being scored. Tasks that need a sandbox are not supported. HUD: a task template keeps its expected answer inside the grader, so without `--answers` only answer-free probes run: an empty reply, the prompt echoed back, and replies that list every integer from 0 to 1000, every letter, yes/no words or the prompt's words (A1; it also runs, with every other probe, when answers are given). A grader that pays those is a substring test in practice. The answers file is JSON: an object from task slug to answer (`{"add-2-3": "5"}`) or a list in task order (`["5", "12"]`). To add answers to an existing audit, rerun `init` with `--force`. `examples/inspect-toy/` and `examples/hud-toy/` each have one planted defect to try. ## For environment authors: check before you publish Run `falsepass init ./your-env && falsepass audit your-env` locally, or add the GitHub Action to the environment's repository so every push is checked: ```yaml # .github/workflows/falsepass.yml on: [push, pull_request] jobs: falsepass: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 - uses: Super-Eric/falsepass@main with: path: . # folder with the environment's pyproject.toml fail-on: "ceiling>0.1 errors>0" # fail when a known-bad answer earns over 0.1, or scoring raises ``` The step summary shows the probe table. Gate names: `ceiling` (highest reward any known-bad completion got), `lazy` (best average reward of one fixed strategy that never uses the answer), `false_fail` (share of correct answers refused), `errors`. Datasets that need a login or an API key at load time cannot be audited in CI without that secret. ## For coding agents - **MCP server.** `falsepass mcp` serves the commands as tools (`falsepass_scan`, `_init`, `_audit`, `_audit_status`, `_results`, `_completion`, `_repro`, `_discover`) over stdio. Audits run in the background, so a client's tool-call timeout never cuts one off; failures come back as tool errors with the reason. Any MCP client starts it with `uvx --from "falsepass[mcp]" falsepass mcp`; in Claude Code, for example: `claude mcp add falsepass -- uvx --from "falsepass[mcp]" falsepass mcp`. - **Skill.** [skills/falsepass/SKILL.md](/falsepass/docs/agents/skill/) is a ready-made audit procedure in the Agent Skills format: how to run an audit, how to tell a defect from a design choice, and how to write it up for a maintainer. Copy the folder into the agent's skills directory (for Claude Code, `~/.claude/skills/`). - **Plain CLI.** Every command is non-interactive and writes files; `init --json`, `results --json` and `scan --json` print JSON; `results --completion ID` prints one completion in full; `audit --fail-on ...` exits 1 when a condition is met. The output formats are described in [docs/OUTPUT.md](/falsepass/docs/output/). Whatever drives it, the output is a list of leads. An agent should reproduce a lead and read the reward code before calling it a defect, and should not report an environment as broken on the strength of a probe table. ## All commands ```bash falsepass init owner/environment # set up audits// for a Hub environment, or pass a local folder falsepass audit environment # no adapter needed for Q&A tasks: the answer format is inferred falsepass audit environment --search 120 # add mutation search (scorer calls per direction and example) falsepass audit environment --fail-on 'ceiling>0.1' --fail-on 'errors>0' # CI gate, exit 1 when met falsepass results environment [--json] # the last run: summary.md, or a compact JSON digest falsepass results environment --completion ID # one completion in full (IDs are in the JSON digest) falsepass scan path/to/source [--json] # static pre-scan only, nothing is installed or run falsepass discover --days 90 --limit 160 # recently maintained Hub environments and what each needs falsepass repro environment # stand-alone script that re-scores the exact failing completions falsepass rollout environment -n 20 # what a local model's own replies earn (Ollama) falsepass propose environment -n 5 # a local model writes wrong answers and climbs on the reward falsepass draft environment # REPORT.draft.md: candidate findings for manual review falsepass overview -o RESULTS.md # one table over every audit on disk falsepass container environment # container tasks: the key, and mutations of it, written as the output file (Docker) falsepass upstream environment # find the source repository on GitHub and compare it with the audit falsepass verify-fix environment --source ../patched/pkg # rerun the stored repro on a patch: which lines change falsepass demo # the toy environment above falsepass mcp # MCP server over stdio (`falsepass[mcp]`) ``` Each audit runs in a throw-away virtualenv pinned by `audits//env.toml`, because Hub environments are built for mutually incompatible `verifiers` versions. Output: `audits//probe-results.json` and `summary.md`. In a clone of this repository, prefix the commands with `uv run`; `uv run pytest` runs the kit's own tests. ## Checks | ID | Check | A correct verifier… | |---|---|---| | P1 | reference answer, in the requested format | gives full reward | | P2 | the same answer with harmless variation | gives full reward | | P3 | empty / whitespace-only | gives nothing | | P4 | the prompt echoed back | gives nothing | | P5 | every candidate answer at once | gives nothing | | P6 | wrong answer in perfect format | gives nothing | | P7 | confident wrong answer | gives nothing | | P8 | wrong answer plus a note to the grader (LLM-judge envs) | gives nothing | | P9 | right answer buried in filler, then retracted | gives nothing | | P10 | malformed output | gives nothing and does not raise | | P11 | the same completion ten times | gives the same score | | P12 | dataset: empty or duplicate prompts, conflicting labels | finds none | | P13 | `NaN`, `Infinity`, an integer answer plus 0.9 (JSON answers: floats set to NaN, integers plus 0.9 or Infinity) | gives nothing and does not raise | | P14 | the correct answer, with the answer tag or `\boxed{}` also named in prose before or after it | gives full reward | | P15 | the answer denied, or named next to a wrong value, inside the answer slot (`\boxed{R \text{ is wrong}}`, `\boxed{W = R}`) | gives nothing | | A1 | no reference needed: every integer 0–1000, every letter, yes/no words, or the prompt's words in one reply | gives nothing | | T7 | the correct answer after one no-op tool call | gives full reward (a judgement call where the harness offers no tools) | | C0–C2 | container tasks: the key written as the output file (C0), with NaN / Infinity / fractional counts (C1), emptied, truncated or replaced by an input file (C2) | pays C0 in full and no mutation in full | | T4 | multi-turn: replay an episode after an unrelated one | gives the same reward and transcript | | T5 | multi-turn: a policy that never answers | is cut off and not rewarded | | T8 | multi-turn: answer wrong, then repeat whatever the environment gives away ("The word was …") | gives nothing; the summary also says if the episode ran on after the environment announced its end | | T6 | tool use: six policies built from the tool schemas alone (no tool call, every tool once with empty arguments, the finishing tool at once, …) | are not rewarded and raise nothing | | X*, T1–T3 | environment-specific probes defined in an adapter | — | | S1 | search: mutate known-bad completions, never adding the answer, toward higher reward | finds nothing above zero | | S2 | search: mutate the correct completion without changing the answer, toward lower reward | finds nothing below full | | S3 | search: completions that mention the answer without committing to it | finds nothing above zero | **Search** (`--search N`) is a seeded beam search over chains of small named mutations. A hit is reported as a minimised recipe, e.g. `nothing + pad_think`. S2 hits are brittleness candidates and need a human eye: whether `"B"` in quotes should count as `B` is a judgement call. **Static pre-scan** runs with every audit on `audits//src` and on its own with `falsepass scan`. Narrow AST rules, each derived from a defect found in a real audit; every hint is printed with the verdict of the probe that tests it (`confirmed: P6 failed 25/25`, `not confirmed`, `not checked`). **Adapter inference.** Without an `adapter.py`, the answer format is found by scoring the reference answer in each candidate format (from the parser and the prompt text) and keeping the one the environment's own reward code pays best. Wrong answers come from the label set, a shifted number, or another row. This covers environments with reference answers, single- or multi-turn; tool-use and sandbox environments, and games whose rows carry no answer, need policies: `falsepass init --policies` writes a skeleton (or `--manual` for Q&A). A tag that the prompt uses to enclose its input (`…`) is not taken for an answer format, and when every format scores the same the plain answer is used. As a control, automatic mode compares the reference with an empty completion and marks the whole result **inconclusive** when the reward does not separate them. **Multi-turn.** Multi-turn environments are scored by running their own rollout loop with a scripted stand-in for the model (`V0RolloutScorer`), so follow-up prompts, stop conditions and per-episode state are exercised. Search mutates every turn of a script. A script entry can also carry tool calls (`{"tool_calls": [{"name": …, "arguments": {…}}]}`), which the environment executes with its real tools. Automatic mode also accepts tool-use environments whose tools run in-process: there is no answer format to infer, so it plays the six T6 policies and T5 and says in the summary that no control was possible. Policies that have to read the task to find a hole are still hand work (`falsepass.model.Policy`: a function from the conversation so far to the next reply). **Rollout check.** `falsepass rollout ` sends a sample of tasks to a model behind an OpenAI-compatible endpoint — a local Ollama server by default, so nothing is billed — and scores the model's own replies with the same scorer the probes use. It flags rows for reading (paid without the reference in the reply; reference in the reply but not paid in full; paid more than the reference; cut off at the token cap) and never calls a flag a finding. For tool-use environments the model is driven through verifiers' own client, so it receives the tool schemas and its tool calls are executed. `--endpoint`, `--model` and `--tag` run a second model next to the first. **Model-written wrong answers.** `falsepass propose ` asks the model for a reply that is on the subject and wrong, scores it, and has it rewritten a few times with the grader's score as feedback. For closed answers the reference must not occur in the reply (checked in code); for open-ended ones the wrongness is the model's own opinion, so every row is something to read, not a finding. **Discovery.** `falsepass discover` lists Hub environments updated recently with more than one published version, and reads each wheel to say what it needs (sandbox, tools, LLM judge, API key, reference answers). It needs the `prime` CLI for the listing. Three headline numbers per environment: the **false-pass ceiling** (highest reward any known-bad completion received), the **lazy-policy reward** (best average reward of one fixed wrong strategy across examples) and the **false-fail rate** (share of known-good completions denied full reward). `summary.md` also shows which reward term paid on known-bad completions. ## From a probe run to a report A probe run produces **leads, not findings**. To turn one into a finding: 1. `falsepass repro ` writes a script that depends only on verifiers and the environment package and re-scores the exact completions (kept in `completions.json`). Run it twice from a clean virtualenv; the output must be identical. 2. Read the reward code and find the line that causes it. Decide whether it is a defect or a design choice: a grader that is exploitable on purpose is an observation. 3. `falsepass draft ` writes `REPORT.draft.md` with each failing probe group as a candidate, the confirmed static hints attached, and the places marked where a person has to judge. Rate what holds with [docs/SEVERITY.md](/falsepass/docs/severity/). 4. Have someone who did not write the report check every number against the stored output. ## From a report to a fix upstream `falsepass upstream ` searches GitHub for the environment's source (package name in a `pyproject.toml`, then its most distinctive class and function names), compares every audited file with the repository's default branch, line endings ignored, and prints what a contributor needs: default branch, licence, last push, open pull requests, how many recent merges came from outside. `falsepass verify-fix --source ` reruns the stored repro with the patched package and prints the lines that changed; a fix is ready when only the finding's lines move. ## Adding an environment `falsepass init owner/environment` writes `audits//env.toml` (package, version, module, index, verifiers pin — all read from the Hub wheel) and unpacks the wheel's source into `audits//src` for the static scan. `falsepass init ./folder` does the same from a local `pyproject.toml` and records the folder as `source`, so the folder itself is installed and scanned on every run. For Q&A environments that is all. Otherwise add `--manual` and fill in the generated `adapter.py`: how to load examples, how to write an answer the way the prompt asks, and what a wrong answer looks like. ## Track record Used on a few dozen public Hub environments in October 2026. Every finding kept was reproduced twice from a clean virtualenv and confirmed by reading the reward code; fixes are being sent to the maintainers as pull requests. Individual environments and counts are not published here until those fixes are in. ## Limits - Leads need a person. The tool finds where a grader pays the wrong thing; deciding whether that is a defect, and how bad, is manual. - Search is shallow: fixed text mutations, a few hundred scorer calls. It does not train a model against the reward. - Container tasks have no automatic mode, and environments on hosted sandboxes or with a paid LLM judge are not covered without credentials. - Supported frameworks: verifiers, Inspect AI (no sandboxed tasks) and HUD (answer-graded templates; tasks graded on what an agent did in a workspace need hand-written policies). ## Related tools - [rewardlint](https://github.com/junglezke/rewardlint) — adversarial corpus for single reward functions. `falsepass` runs it through an environment's reward code when the reference can be substituted, and prints its verdict in `summary.md`. - ratctl — static and LLM-assisted scanner of environment source for sandbox exploit patterns. - [VerifierLab](https://github.com/fraware/verifierlab) — attack campaigns against a wrapped grader under a declared budget, with a ground-truth oracle. - [vlabs-integrity](https://github.com/verifiablelabs/vlabs-integrity) — mutation-kill and isomorphic-relabelling audit for coding rewards with a trusted reference. - [harden-v0](https://github.com/few-sh/harden-v0) — LLM hacker–fixer loop that hardens Harbor-format agent benchmark verifiers. ## Layout ``` falsepass/ the probe kit (falsepass/mcp_server.py: the MCP tools) tests/ its tests examples/toy-env/ the demo environment, with its planted defect examples/inspect-toy/, examples/hud-toy/ the same idea for Inspect AI and HUD skills/falsepass/ agent skill: the audit procedure action.yml the GitHub Action docs/SEVERITY.md the rating scale; docs/OUTPUT.md: the output files audits// created by `falsepass init`; ignored by git ``` ## Licence Apache-2.0, see `LICENSE`. --- # Using falsepass from an agent falsepass is built to be driven by a coding agent as much as by a person. Every command is non-interactive, writes its results to files, and has a machine-readable form. There are three ways in, from most to least structured. ## 1. MCP server `falsepass mcp` serves the commands as tools over stdio. Any MCP client can start it without installing anything first: ```bash uvx --from "falsepass[mcp]" falsepass mcp ``` For Claude Code: ```bash claude mcp add falsepass -- uvx --from "falsepass[mcp]" falsepass mcp ``` For clients configured with JSON (Cursor, Claude Desktop, and others): ```json { "mcpServers": { "falsepass": { "command": "uvx", "args": ["--from", "falsepass[mcp]", "falsepass", "mcp"] } } } ``` | Tool | Arguments | Returns | |---|---|---| | `falsepass_scan` | `path` | static hints for a source folder or file; nothing is installed | | `falsepass_init` | `target` (Hub id, local folder, Inspect task, HUD folder); optional `name`, `version`, `framework`, `task`, `answers`, `extra`, `manual`, `force` | `{"name": ..., "folder": ..., "framework": ..., "env": {...}, "next": ...}`: `name` is what the other tools take | | `falsepass_audit` | `env`, `n` (tasks, default 25), `search` (mutation budget), `wait` (seconds, default 45) | the results digest below with `"status": "finished"`, or `"status": "running"` if the audit needs longer | | `falsepass_audit_status` | `env`, `wait` | the same, for an audit already started: results, or `"running"` with the latest progress lines | | `falsepass_results` | `env` | the digest of the last run (progress instead while one is running) | | `falsepass_completion` | `env`, `completion_id` | the full text of one completion; the digest shows only its start and end | | `falsepass_repro` | `env`, `probe` (e.g. `["P5"]`) | where the stand-alone repro script was written | | `falsepass_discover` | `days`, `limit` | recently maintained Hub environments and what each needs | Every tool takes `workdir`; audits are created in `workdir/audits//`. An audit installs the environment before it probes it, so it can take several minutes, longer than many clients allow for one tool call. It therefore runs in the background: `falsepass_audit` waits up to `wait` seconds and either returns the results or `{"status": "running", "elapsed_s": ..., "log_tail": [...]}`. Call `falsepass_audit_status` with the same `env` until it finishes. Calling `falsepass_audit` again while one is running doesn't start a second audit. The job is recorded in `audits//job.json`, its output in `audits//job.log`. When something goes wrong (no such audit, a failed install, a dataset that needs a login), the tool returns an MCP error whose text says what happened and what to try next, so a client sees `isError` and the reason without reading server logs. Names are forgiving: `arc_easy` finds the audit `init` named `arc-easy`. For a HUD environment, pass reference answers with `answers`: a JSON file holding either an object from task slug to answer (`{"add-2-3": "5"}`) or a list in task order. Without it only the answer-free probes run. Adding answers to an existing audit takes `force: true`. `uv` must be on the PATH: each audit installs the environment in its own throw-away virtualenv. ## 2. Agent skill [`skills/falsepass/SKILL.md`](https://super-eric.github.io/falsepass/docs/agents/skill/) is the whole audit procedure in the Agent Skills format: run, read the numbers, confirm a lead before claiming it, decide defect or design, and write it up for the maintainer. Copy the folder into the agent's skills directory (for Claude Code, `~/.claude/skills/`) and ask the agent to audit an environment. ## 3. Plain CLI ```bash falsepass init owner/environment # or ./folder, task.py, inspect_evals/, a HUD folder falsepass audit environment --fail-on 'ceiling>0.1' # exit 1 when the condition is met falsepass results environment --json # compact digest, small enough for a context window falsepass scan path/to/source --json # static pre-scan only ``` The digest from `results --json` and from the MCP audit tools (abridged): ```json { "environment": "owner/env", "headline": {"false_pass_ceiling": 1.0, "false_pass_via": "P5 shotgun/all_in_one_slot", "lazy_policy_reward": 0.0, "false_fail_rate": 0.108, "scoring_errors": 0}, "failing": [{"probe": "P5", "variant": "all_in_one_slot", "runs": 10, "failed": 10, "completion": "start of the completion … [316 chars] … its end, where the answer is", "completion_id": "da79bcea66a8"}], "static_hints": ["..."], "note": "Leads, not findings: ..." } ``` All fields are described in [Output files](https://super-eric.github.io/falsepass/docs/output/). ## Games, simulations and tool tasks Automatic mode needs a reference answer it can write into a reply. A game, a customer-service simulation or a tool task has none: the reward depends on what the model does over several turns. There, an audit ends `"status": "inconclusive"` (the reference and an empty reply earned about the same) or, for tool tasks, runs only the generic tool policies. The digest then lists only `facts_that_hold`: rows built without the answer, such as "an empty reply earned 1.0 on 20 of 20 tasks", which are true either way. The next step is to write policies, which takes a capable agent a few minutes per environment: ```bash falsepass init owner/environment --policies --force # adds adapter.py: a policy skeleton ``` A policy is a function from the conversation so far (a list of `{"role", "content"}` dicts) to the next assistant message, either a string or `{"content": ..., "tool_calls": [{"name": ..., "arguments": {...}}]}`. With `Policy(name, act, wants_tools=True)` it also gets the tool schemas. Each policy runs through the environment's own rollout loop and rubric. Fill in: - `solver(example)`: a policy that does the task properly (it may read the row's answer). It should earn full reward; set `max_reward` to what it earns. - `gives_up(example)`: plays honestly without solving. It should earn nothing. - `exploits(example)`: policies that never use the answer. The skeleton starts from the ready-made ones in `falsepass.policies`, each of which found a real hole: `after_the_end(wrong, say)` (answer wrong, then repeat what the environment gives away), `keeps_talking(first)` (carry on after "game over"), `never_acts()`. Adapt their messages to the environment's format. - `special_number_scripts(example)`: one message sequence the environment accepts; `policies.special_numbers()` turns it into variants with `nan`, `inf`, `-1`, `1e999` and `-0` in place of its numbers (ids such as `#1234` are left alone). A limit written as `amount > LIMIT` is false for NaN. Then run `falsepass audit ` as usual. Each policy's episode (what it said, and the whole conversation) is kept: `falsepass_completion` shows it, and `falsepass repro` replays the policy's turns as a fixed script, so a lead found by a policy is reproduced without falsepass. ## Reading the output - **`false_pass_ceiling` near the maximum**: some known-bad completion was paid. `failing` says which probe and variant. - **`lazy_policy_reward` above zero**: one fixed strategy that never uses the answer earns that much on average. On a closed answer set (yes/no, A–J), subtract what one fixed label earns under any correct grader. - **`false_fail_rate`**: correct answers refused. Some of it is deliberate strictness. - **`static_hints` with `verdict: confirmed`**: the line that most likely causes a failure. ## Rules worth giving your agent The output is a list of leads. Before an agent reports anything: 1. Read the whole completion if the preview isn't enough: `falsepass_completion` (or `falsepass results --completion `). 2. Reproduce it: `falsepass repro --probe P5` writes a script that depends only on the environment's own packages. Run it twice from a clean virtualenv. 3. Find the line in the reward code that causes it. 4. Decide whether it is a defect or a design choice. A grader that is exploitable on purpose, or strictness the prompt asks for, is not a bug. 5. Measure how many of the environment's own tasks it affects. The audit samples 25 tasks by default; the repro script is the place to extend it to the full dataset. Don't let an agent call an environment broken on the strength of a probe table, and don't let it file issues on other people's repositories without a person agreeing. ## Text versions Every page on this site is also published as Markdown (link at the bottom of each page), and [`llms.txt`](https://super-eric.github.io/falsepass/llms.txt) / [`llms-full.txt`](https://super-eric.github.io/falsepass/llms-full.txt) index all of them. --- # falsepass 检查强化学习（RL）环境的判分器（verifier）有没有两类错误：**假通过**（明知是错的回答拿到了奖励）和**假失败**（明知是对的回答没拿到满分）。 举个真实的例子：在一个公开的环境里，把题目原样抄回去，每道题都能拿满分。原因是每道题都列出了全部候选答案，而判分器只检查正确答案有没有出现在回复里的某个位置。所以只要把所有答案都写上，就能拿满分。falsepass 能在训练之前发现这类漏洞。 一个 RL 环境由任务数据集和判分器组成，判分器把模型的输出换算成奖励。`falsepass` 把判分器本身当作被测对象：用环境自己的奖励代码，去给一批事先已知对错的回答打分，然后报告每一处判分和事实不符的地方。不需要模型，不需要真的跑训练，也不需要 GPU。 支持三个框架，已发布的或还在本地文件夹里的都可以： | 框架 | falsepass 实际运行的是 | `falsepass init …` | |---|---|---| | [verifiers](https://github.com/PrimeIntellect-ai/verifiers)（Prime Intellect 的 Environments Hub），三代接口都支持 | 环境自己的 rubric 或 taskset | `owner/environment`，或 `./文件夹` | | [Inspect AI](https://inspect.aisi.org.uk) | 任务自己的 solver 链（用按脚本回答的替身代替模型），再用它自己的 scorer 打分 | `path/to/task.py`、`./项目`，或 `inspect_evals/gsm8k` | | [HUD](https://docs.hud.ai) | 任务模板自己的判分代码，调用方式和 HUD 服务器一样 | `path/to/环境文件夹`（加 `--answers refs.json` 可跑全部测试） | ## 安装 需要先装好 [uv](https://docs.astral.sh/uv/)（每次审计都在一个用完即弃的虚拟环境里运行），以及 Python 3.11 或更新版本。 ```bash uvx falsepass demo # 直接试用，不用安装 uv tool install falsepass # 或者装成常驻命令 `falsepass` pip install falsepass # 或者装进现有的 Python 环境 uvx --from git+https://github.com/Super-Eric/falsepass falsepass --version # 还没发布的 main 分支 ``` ## 快速上手 ```bash falsepass demo # 审计一个自带的小环境，里面故意埋了一个漏洞（约一分钟） falsepass init owner/environment # Hub 上的环境：包名、版本和 verifiers 版本自动从 Hub 读取 falsepass init ./my-environment # 或者你自己的环境，指向它的文件夹（含 pyproject.toml）；每次运行都用最新代码 falsepass audit my-environment # 开始检查；结果在 audits/my-environment/summary.md falsepass results my-environment --json # 核心指标和没通过的测试，给脚本和 AI 代理读 ``` `demo` 会把小环境写到 `./falsepass-demo` 再审计。这个环境的判分只看"标准答案有没有出现在回答里的某个位置"，结果摘要会从几个方向指出这一行代码：静态扫描直接定位到它；"一次列出多个答案"（P5）、"给出答案后又收回"（P9）、"在答案框里否定答案"（P15）这几类回答都拿到了奖励。下面是真实输出的精简节选（完整文件：`examples/toy-env/summary.md`）： ``` - False-pass ceiling: 1 of 1 (highest reward given to a known-bad completion; via P5 shotgun/all_in_one_slot) - Lazy-policy reward: 0 (best average reward of one fixed strategy that never uses the reference answer; via P3 empty/empty) - False-fail rate: 0.108 (93 known-good completions) | Probe | Variant | Expect | Runs | Failed | Score min / mean / max | | P5 shotgun | all_in_one_slot | zero | 10 | 10 | 1 / 1 / 1 | | P9 oversized | correct_then_retracted | zero | 10 | 10 | 1 / 1 / 1 | | P15 hedged_in_slot | denied | zero | 10 | 10 | 1 / 1 / 1 | Static hint: falsepass_toy_env.py:26 reference_substring_match return 1.0 if answer in reply else 0.0 confirmed: P9 failed 10/20, P5 failed 10/30 ``` 意思是：错误回答拿到的最高分是满分（1 分里拿了 1 分）；"一次列出所有答案"、"给出答案又收回"、"在答案框里否定答案"三类回答，10 次全部拿了满分；静态扫描指出的第 26 行，被这些测试结果证实了。 ## Inspect AI 和 HUD ```bash falsepass init inspect_evals/gsm8k # inspect_evals 里的任务（版本从 PyPI 固定） falsepass init my_eval.py --task hard_split # 任务文件；有多个任务时用 --task 选一个 falsepass init ./my-hud-env # HUD 环境文件夹（env.py + tasks.py） falsepass init ./my-hud-env --answers refs.json # 提供标准答案：跑全部测试 ``` Inspect：替身只代替调用 `generate()` 的那一步；提示模板、系统消息、答案解析（`multiple_choice()`）都是任务自己的代码。如果 solver 或 scorer 自己去调用模型（自我批评、`model_graded_qa`），每一行都会被标记出来，而不是给出一个不可信的分数。需要沙箱的任务暂不支持。 HUD：任务模板把标准答案藏在判分代码里，所以不加 `--answers` 时只跑不需要标准答案的测试：空回答、原样抄回题目，以及一次列出 0 到 1000 所有整数、所有字母、yes/no 之类的词或题目里所有词的回答（A1；提供答案时它也会和其他测试一起跑）。如果这些回答能拿分，判分实际上就是在做子串匹配。答案文件是 JSON：要么是"任务 slug → 答案"的对象（`{"add-2-3": "5"}`），要么是按任务顺序排列的列表（`["5", "12"]`）。要给已经建好的审计补上答案，带 `--force` 重新运行 `init`。`examples/inspect-toy/` 和 `examples/hud-toy/` 各埋了一个漏洞，可以拿来试。 ## 给环境作者：发布前先检查 在本地运行 `falsepass init ./your-env && falsepass audit your-env`，或者在环境的仓库里加上 GitHub Action，每次提交都自动检查一遍： ```yaml # .github/workflows/falsepass.yml on: [push, pull_request] jobs: falsepass: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 - uses: Super-Eric/falsepass@main with: path: . # 环境的 pyproject.toml 所在的文件夹 fail-on: "ceiling>0.1 errors>0" # 错误回答得分超过 0.1，或判分代码报错时，标记为失败 ``` 检查结果的表格会显示在这次运行的摘要页里。可用的判定条件有：`ceiling`（错误回答拿到的最高分）、`lazy`（一个从不看题、每次都给同样回答的策略平均能拿多少分）、`false_fail`（正确回答被拒的比例）、`errors`（判分代码报错的次数）。如果加载数据集需要登录或 API 密钥，在 CI 里必须配置相应的密钥才能审计。 ## 给 AI 编程代理 - **MCP 服务**：`falsepass mcp` 通过 stdio 把各个命令提供为工具（`falsepass_scan`、`_init`、`_audit`、`_audit_status`、`_results`、`_completion`、`_repro`、`_discover`）。审计在后台运行，客户端单次调用的超时不会把它打断；出错时以工具错误返回，并写明原因。任何 MCP 客户端都可以用这条命令启动它：`uvx --from "falsepass[mcp]" falsepass mcp`。以 Claude Code 为例：`claude mcp add falsepass -- uvx --from "falsepass[mcp]" falsepass mcp`。 - **Skill**：`skills/falsepass/SKILL.md` 是一份写好的审计流程（Agent Skills 格式），内容包括怎么运行审计、怎么区分漏洞和有意的设计、怎么写给维护者。把这个文件夹复制到代理的 skills 目录即可（Claude Code 是 `~/.claude/skills/`）。 - **普通命令行**：所有命令都不需要交互，结果写成文件；`init --json`、`results --json` 和 `scan --json` 输出 JSON；`results --completion ID` 输出某个回答的全文；`audit --fail-on ...` 在条件满足时以退出码 1 结束。输出格式见 `docs/OUTPUT.md`（英文）。 不管由谁来运行，输出都只是一份**线索**清单。代理应当先复现线索、读过奖励代码，再判断是不是漏洞；不能只凭一张测试表格就说某个环境有问题。 ## 全部命令 ```bash falsepass init owner/environment # 为 Hub 上的环境建好 audits//，也可以传本地文件夹 falsepass audit environment # 问答类任务不需要写适配器：答案格式会自动推断 falsepass audit environment --search 120 # 加上变异搜索（每个方向、每道题调用判分器的次数） falsepass audit environment --fail-on 'ceiling>0.1' --fail-on 'errors>0' # CI 门槛，满足时退出码为 1 falsepass results environment [--json] # 上一次的结果：summary.md，或精简的 JSON falsepass results environment --completion ID # 某个回答的全文（ID 在 JSON 摘要里） falsepass scan path/to/source [--json] # 只做静态扫描，不安装也不运行任何东西 falsepass discover --days 90 --limit 160 # 最近有人维护的 Hub 环境，以及每个需要什么条件 falsepass repro environment # 生成独立脚本，给那些没通过的回答重新打分 falsepass rollout environment -n 20 # 看本地模型（Ollama）自己答题实际拿多少分 falsepass propose environment -n 5 # 让本地模型写错误回答，并根据得分反复改写 falsepass draft environment # REPORT.draft.md：待人工审核的候选问题 falsepass overview -o RESULTS.md # 把磁盘上所有审计结果汇成一张表 falsepass container environment # 容器类任务：把标准答案及其变体写成输出文件（需要 Docker） falsepass upstream environment # 在 GitHub 上找到环境的源码仓库，和审计的版本逐个文件对比 falsepass verify-fix environment --source ../patched/pkg # 用打过补丁的代码重跑复现脚本，看哪几行结果变了 falsepass demo # 上面那个小环境 falsepass mcp # MCP 服务，走 stdio（需要 `falsepass[mcp]`） ``` 每次审计都在一个由 `audits//env.toml` 固定版本的临时虚拟环境里运行，因为 Hub 上的环境依赖的 `verifiers` 版本互不兼容。输出文件是 `audits//probe-results.json` 和 `summary.md`。如果是在本仓库的克隆里使用，命令前要加 `uv run`；`uv run pytest` 运行工具自身的测试。 ## 测试项目 | 编号 | 测试内容 | 正确的判分器应该… | |---|---|---| | P1 | 标准答案，按要求的格式 | 给满分 | | P2 | 同一个答案，加上无害的变化 | 给满分 | | P3 | 空回答 / 只有空白 | 不给分 | | P4 | 把题目原样抄回去 | 不给分 | | P5 | 一次列出所有候选答案 | 不给分 | | P6 | 错误答案，格式完全正确 | 不给分 | | P7 | 语气很肯定的错误答案 | 不给分 | | P8 | 错误答案，加一句写给评委的话（用大模型当评委的环境） | 不给分 | | P9 | 正确答案埋在一堆废话里，随后又收回 | 不给分 | | P10 | 格式损坏的输出 | 不给分，也不报错 | | P11 | 同一个回答打分十次 | 每次分数相同 | | P12 | 数据集：空题目、重复题目、相互矛盾的标签 | 一个都没有 | | P13 | `NaN`、`Infinity`、整数答案加 0.9（JSON 答案：小数改成 NaN，整数加 0.9 或改成 Infinity） | 不给分，也不报错 | | P14 | 正确答案，但在正文里答案前后又提到了答案标签或 `\boxed{}` | 给满分 | | P15 | 在答案框里否定答案，或把答案和错误值并列（`\boxed{R \text{ is wrong}}`、`\boxed{W = R}`） | 不给分 | | A1 | 不需要标准答案：一次列出 0–1000 所有整数、所有字母、yes/no 之类的词，或题目里所有的词 | 不给分 | | T7 | 先调用一次无实际作用的工具，再给出正确答案 | 给满分（如果环境本身不提供工具，这一项需要人来判断） | | C0–C2 | 容器类任务：把标准答案直接写成输出文件（C0），数字改成 NaN / Infinity / 小数计数（C1），清空、截断或换成输入文件（C2） | C0 给满分，所有变体都不给满分 | | T4 | 多轮：先跑一个无关的对话，再重放同一个对话 | 奖励和对话记录都相同 | | T5 | 多轮：一个永远不作答的策略 | 被中途截断，不给分 | | T8 | 多轮：先答错，再把环境泄露出来的内容（如"The word was …"）原样答回去 | 不给分；如果环境宣布结束后回合还在继续，也会注明 | | T6 | 工具类：只根据工具定义构造的 6 种策略（不调用工具、每个工具空参数调用一次、直接调用结束工具……） | 不给分，也不报错 | | X*、T1–T3 | 在适配器里定义的环境专用测试 | — | | S1 | 搜索：对已知错误的回答做变异（不加入答案），往高分方向找 | 找不到高于 0 的 | | S2 | 搜索：对正确回答做变异（不改答案），往低分方向找 | 找不到低于满分的 | | S3 | 搜索：提到答案但不明确认定它的回答 | 找不到高于 0 的 | **搜索**（`--search N`）是带固定随机种子的束搜索，在一串串命名好的小变异上进行。命中的结果会化简成最短的组合报告出来，例如 `nothing + pad_think`。S2 的命中只说明判分可能过于死板，需要人来看：比如带引号的 `"B"` 算不算 `B`，属于判断问题。 **静态扫描**在每次审计时对 `audits//src` 运行，也可以用 `falsepass scan` 单独运行。规则都是范围很窄的 AST 规则，每一条都来自一次真实审计中发现的问题；每条提示旁边都会标出相应测试的结论（`confirmed: P6 failed 25/25`、`not confirmed`、`not checked`）。 **自动推断适配器**：没有 `adapter.py` 时，工具会把标准答案写成各种候选格式（从解析器和题目文字里找），用环境自己的奖励代码打分，取得分最高的那种格式。错误答案从标签集合里取，或者把数字偏移一下，或者用另一道题的答案。这适用于有标准答案的环境，单轮多轮都行；工具类、沙箱类环境，以及数据里没有答案的游戏，需要写策略：`falsepass init --policies` 会生成一个骨架（问答类用 `--manual`）。题目中用来包住输入内容的标签（如 `…`）不会被当成答案格式；如果所有格式得分相同，就用不加格式的答案。作为对照，自动模式会比较标准答案和空回答的得分；如果奖励区分不出两者，整个结果标记为**无法判断**（inconclusive）。 **多轮**：多轮环境的打分方式是运行它自己的对话循环，用一个按脚本回答的替身代替模型（`V0RolloutScorer`），所以追问、停止条件和每轮的状态都会被真实执行。搜索会对脚本的每一轮都做变异。脚本里的一条也可以包含工具调用（`{"tool_calls": [{"name": …, "arguments": {…}}]}`），由环境用真实的工具执行。自动模式也支持工具在本进程内运行的工具类环境：这类环境没有答案格式可以推断，所以会运行 6 种 T6 策略和 T5，并在摘要里说明无法做对照。需要读懂题目才能找到漏洞的策略，仍然要手写（`falsepass.model.Policy`：一个根据目前的对话给出下一句回复的函数）。 **真实模型对照**：`falsepass rollout ` 把一部分题目发给一个 OpenAI 兼容接口背后的模型（默认是本地的 Ollama，不花钱），再用测试所用的同一个打分器给模型自己的回答打分。它会标出值得看的行（回答里没有标准答案却拿了分；有标准答案却没拿满分；得分比标准答案还高；回答被长度上限截断），但从不把这些标记称为问题。工具类环境通过 verifiers 自己的客户端驱动模型，所以模型能收到工具定义，它的工具调用也会被执行。用 `--endpoint`、`--model` 和 `--tag` 可以再加一个模型对比。 **模型写的错误答案**：`falsepass propose ` 让模型写一个切题但错误的回答，打分后把分数反馈给模型，让它改写几次。对于有固定答案的题目，回答里不能出现标准答案（代码里会检查）；对于开放式题目，"错误"只是模型自己的判断，所以每一行都只是值得读的材料，不是结论。 **发现环境**：`falsepass discover` 列出 Hub 上最近更新过、发布过不止一个版本的环境，并读取每个安装包，说明它需要什么（沙箱、工具、大模型评委、API 密钥、标准答案）。列出环境需要 `prime` 命令行工具。 每个环境有三个核心指标：**假通过上限**（任何已知错误的回答拿到的最高分）、**偷懒策略得分**（一个不看答案、每题都用同一种错误策略的回答，在所有题上的最高平均分）和**假失败率**（已知正确的回答没拿到满分的比例）。`summary.md` 还会显示，在错误回答上是奖励的哪一项给了分。 ## 从测试结果到报告 一次测试得到的是**线索，不是结论**。把线索变成结论的步骤： 1. `falsepass repro ` 生成一个只依赖 verifiers 和环境本身的脚本，给那些回答重新打分（回答原文保存在 `completions.json`）。在干净的虚拟环境里运行两次，输出必须完全一致。 2. 读奖励代码，找到导致问题的那一行。判断它是漏洞还是有意的设计：故意做成可以被钻空子的判分器，只记作观察，不算问题。 3. `falsepass draft ` 生成 `REPORT.draft.md`：每组没通过的测试列为一个候选，附上已确认的静态提示，并标出需要人来判断的地方。用 `docs/SEVERITY.md`（英文）给成立的问题定级。 4. 请一个没参与写报告的人，把报告里的每个数字和保存的输出逐一核对。 ## 从报告到给上游提交修复 `falsepass upstream ` 在 GitHub 上搜索环境的源码（先按 `pyproject.toml` 里的包名，再按最有特征的类名和函数名），把审计过的每个文件和仓库默认分支逐一比较（忽略换行符差异），并列出提交修复需要知道的信息：默认分支、许可证、最近一次推送、未合并的 PR、最近合并的 PR 中有多少来自外部。`falsepass verify-fix --source <打过补丁的文件夹>` 用打过补丁的包重跑保存的复现脚本，列出变化的行；只有问题对应的那几行变了，修复才算可以提交。 ## 添加环境 `falsepass init owner/environment` 会生成 `audits//env.toml`（包名、版本、模块名、安装源、verifiers 版本，全部从 Hub 上的安装包读取），并把安装包的源码解压到 `audits//src`，供静态扫描使用。`falsepass init ./folder` 从本地的 `pyproject.toml` 做同样的事，并把文件夹记为 `source`，这样每次运行都会重新安装和扫描这个文件夹。问答类环境到这一步就够了。其他环境要加上 `--manual`，然后补全生成的 `adapter.py`：怎么加载题目、怎么按题目要求写答案、错误答案长什么样。 ## 实际使用记录 2026 年 10 月用于几十个公开的 Hub 环境。保留下来的每个问题，都在干净的虚拟环境里复现过两次，并通过读奖励代码确认；修复正在以 PR 的形式陆续提交给维护者。在这些修复合并之前，这里不公布具体环境和问题数量。 ## 局限 - 线索需要人来判断。工具能找出判分器在哪里给错了分，但这算不算漏洞、有多严重，要靠人来决定。 - 搜索比较浅：只有固定的文本变异，判分器调用几百次。不会针对奖励去训练模型。 - 容器类任务没有自动模式；托管沙箱上的环境，以及用付费大模型当评委的环境，没有相应凭据就无法覆盖。 - 支持的框架：verifiers、Inspect AI（不含需要沙箱的任务）和 HUD（按回答判分的任务模板；按代理在工作区里做了什么来判分的任务，仍需手写策略）。 ## 相关工具 - [rewardlint](https://github.com/junglezke/rewardlint)：针对单个奖励函数的对抗样本集。在标准答案可以替换的情况下，`falsepass` 会用环境的奖励代码运行它，并把结论写进 `summary.md`。 - ratctl：对环境源码做静态扫描，并可借助大模型，查找沙箱漏洞的常见写法。 - [VerifierLab](https://github.com/fraware/verifierlab)：在设定的预算内，对一个包装好的判分器发起成批攻击，需要提供标准答案来源。 - [vlabs-integrity](https://github.com/verifiablelabs/vlabs-integrity)：针对编程题奖励的变异检测和同构重标记审计，需要一份可信的标准解法。 - [harden-v0](https://github.com/few-sh/harden-v0)：用大模型扮演攻击方和修补方循环对抗，加固 Harbor 格式的代理测评判分器。 ## 目录结构 ``` falsepass/ 测试工具本身（falsepass/mcp_server.py：MCP 工具） tests/ 工具自身的测试 examples/toy-env/ demo 用的小环境，带一个故意埋的漏洞 examples/inspect-toy/、examples/hud-toy/ Inspect AI 和 HUD 版的同类小例子 skills/falsepass/ 给 AI 代理的 skill：审计流程 action.yml GitHub Action docs/SEVERITY.md 定级标准；docs/OUTPUT.md：输出文件说明（均为英文） audits// 由 `falsepass init` 生成；不纳入 git ``` ## 许可证 Apache-2.0，见 `LICENSE`。 --- # Output files Everything an audit writes lives in `audits//` (or the folder given with `--out`). | File | For | Written by | |---|---|---| | `env.toml` | how the environment is installed: `id`, `framework` (`verifiers`, `inspect`, `hud`; default verifiers), `task` (Inspect or HUD task spec), `package`, `version`, `module`, `index`, `verifiers` / `inspect_ai` / `hud` (version pins), optional `source` (a local folder), `answers` (HUD reference file), `extra`, `python`, `load_args` | `falsepass init` | | `src/` | the Hub wheel's source, for the static scan (local environments are scanned in place) | `falsepass init` | | `probe-results.json` | every probe run, machine-readable (below) | `falsepass audit` | | `summary.md` | the same for people: headline numbers, one row per probe variant, reward terms, static hints | `falsepass audit` | | `completions.json` | the full text of every completion a result refers to, keyed by `completion_id` | `falsepass audit` | | `job.json`, `job.log` | an audit started from the MCP server: its state (`running`, `finished`, `failed`), pid, start time, and its console output | `falsepass mcp` | | `repro_auto.py` | stand-alone script that re-scores the failing completions | `falsepass repro` | | `REPORT.draft.md` | candidate findings to review | `falsepass draft` | ## probe-results.json ```json { "meta": {"environment": "owner/env", "date": "…", "examples": 25, "seed": 0, "python": "3.12.x", "versions": {"verifiers": "…", "…": "…"}, "falsepass": "…", "max_reward": 1.0, "bad_tolerance": 0.0, "weights": {"term": 1.0}, "adapter": {"…": "how the format was inferred"} }, "results": [ { "probe": "P5", "name": "shotgun", "variant": "all_in_one_slot", "example": "row-98", "expected": "zero", "score": 1.0, "passed": false, "detail": "expected zero, got 1.0", "completion": "preview", "metrics": {"term": 1.0}, "errors": [], "answer_free": false, "completion_id": "…" } ], "static": [ { "rule": "reference_substring_match", "file": "env.py", "line": 26, "where": "load_environment", "evidence": "source line", "meaning": "…", "confirm_with": ["P9", "P5", "S3"], "verdict": "confirmed: P9 failed 10/20, P5 failed 10/30" } ], "rewardlint": null } ``` - `expected`: `full` (a correct completion, must get the maximum), `zero` (a known-bad completion, must get nothing), `stable` (the same completion scored repeatedly, must get the same score), `at_most_reference` (must not beat the concise correct answer). - `passed`: whether the grader did what `expected` asks. - `answer_free`: the completion was built without the reference answer, so a policy that knows nothing about the task could produce it. These feed the lazy-policy number. - `metrics`: the value of each reward term, so a leak can be traced to the term that pays. - `errors`: exceptions or timeouts raised by the reward code on this completion. ## falsepass results --json A digest of the same run, small enough to hand to a model (numbers from `falsepass demo`): ```json { "environment": "owner/env", "meta": {"date": "…", "falsepass": "…", "versions": {"…": "…"}, "examples": 25, "max_reward": 1.0, "adapter": {"format": "", "answers": "open", "turns": "single-turn", "validity": "ok"}}, "headline": {"false_pass_ceiling": 1.0, "false_pass_via": "P5 shotgun/all_in_one_slot", "lazy_policy_reward": 0.0, "lazy_policy_via": "P3 empty/empty", "false_fail_rate": 0.108, "known_good": 93, "known_bad": 233, "scoring_errors": 0}, "failing": [ {"probe": "P5", "name": "shotgun", "variant": "all_in_one_slot", "expected": "zero", "runs": 10, "failed": 10, "scores": [1.0, 1.0, 1.0, 1.0, 1.0], "detail": "…", "example": "row-98", "completion": "start … [316 chars] … end", "completion_id": "da79bcea66a8", "answer_free": false} ], "static_hints": [ "as in probe-results.json" ], "note": "Leads, not findings: …" } ``` `failing` lists false passes first (a known-bad completion was paid), the most consistent ones on top, then false fails. `meta` and `static_hints` are trimmed to what an agent needs; everything else is in `probe-results.json`. The headline numbers: - **false-pass ceiling**: the highest reward any known-bad completion received. - **lazy-policy reward**: the best average reward of one fixed strategy that never uses the reference answer. On a closed answer set, one fixed label earns its share of the dataset under any correct grader; subtract that base rate before reading it. - **false-fail rate**: the share of known-good completions denied full reward. Some of these are deliberate strictness about format; read the variant names. `completion` shows the start and the end of the first failing completion (the answer is usually at the end); `falsepass results --completion ` prints all of it. ## falsepass init --json ```json {"name": "gsm8k", "folder": "/abs/path/audits/gsm8k", "framework": "verifiers", "env": {"id": "primeintellect/gsm8k", "package": "gsm8k", "version": "…"}, "manual": false, "others": [], "next": "falsepass audit gsm8k"} ``` `name` is what every other command takes. Names are forgiving: `arc_easy` finds `arc-easy`. ## Exit codes `falsepass audit` exits 1 when a `--fail-on` condition is met and with the probe run's own code when it fails; every other command exits 0 on success and 1 with a message on stderr when its input is missing. --- # Severity scale used in the audit reports Every audit report rates its findings against this scale. It was written after an independent review found an earlier, shorter scale had been applied unevenly (the same kind of defect rated High in one report and Low in another). Two terms first. - **Answer-free completion** — text produced without knowing the task's answer: empty, the prompt echoed back, one fixed text used for every task, a wrong value in the right format, the reference of a different task. - **Non-committal completion** — text that contains the correct answer but denies it, retracts it, or offers it among alternatives ("definitely not 18", "either 25 or 18"). Rewards are read relative to the maximum the rubric can pay. | Severity | Applies when | |---|---| | **Critical** | An answer-free completion earns at least 0.75 of the maximum — or more than the dataset's own reference answers while earning at least 0.75. Also: a completion that plainly violates the rule the environment exists to test is scored as compliant. | | **High** | An answer-free completion earns 0.25–0.75 of the maximum on average. Or a non-committal completion receives full correctness credit on all sampled tasks. Or the dataset's reference answers earn less than about half the maximum. Or a correct answer scores below a wrong one. Or a reward term carrying a quarter or more of the weight never fires. | | **Medium** | An answer-free completion earns 0.10–0.25. A non-committal completion gets full credit on a subset of task types. A correct answer in the requested format is rejected *and* a less compliant form is accepted. Dataset defects (tasks that need no skill, duplicate or conflicting rows). Non-deterministic scoring or environment replies. A reward term that is constant. | | **Low** | Defensible strictness about format (a period inside the answer slot, letter case). Which of several answers in one reply is graded — a policy choice, as long as it is consistent. Reward code that raises on unusual input. Documentation that disagrees with the code. Anything below 0.10. | Notes on applying it. - **Similarity graders.** Partial credit is their design. The severity follows the reward an answer-free completion gets, using the rows above; the report says that the design is intentional. - **Disclosure does not lower the rating.** If maintainers document a weakness, the report says so prominently and the rating stays where the scale puts it — the default configuration is what gets trained on. - **Sample size is stated, not folded into the rating.** A finding seen on 2 of 2 searched tasks is rated by the scale and labelled with the 2. - **Environments that only test for the absence of a violation.** Doing nothing is compliant by such an environment's own definition, so inaction scoring full is rated High, not Critical, and the report says why it matters only if the reward is trained on. - **Closed answer sets.** On a task whose answer is one of a few labels, one fixed label used for every task earns that label's share of the dataset under any correct grader (0.5 on a balanced yes/no task). That base rate is subtracted before a fixed guess is rated, and the label balance itself is an observation, not a finding. - **Reach.** When a defect needs the completion to take a particular form, the report says how often that form arises from the environment's own data. If it arises on fewer than about one task in twenty, the finding is rated one level below what the table gives. The form must be counted in the environment's own data (its tasks, references, inputs). How often a model's replies take the form is an observation and never moves a rating: a few dozen model replies cannot show "under one in twenty" anyway. - **Graders that are exploitable on purpose.** Some environments are built to study reward hacking and ship a grader their own source describes as the exploitable one. What that grader lets through is recorded as an observation and not rated; findings are rated against the grader the source presents as the sound one. The report says which of the two is the default. - **Results obtained with a substitute.** When a component could not be run as the environment runs it — a hosted LLM judge replaced by a local model — what was measured with the substitute is reported as an observation and not rated. A finding that depends on how that component behaves is rated on what could be measured and marked provisional. - **Non-default configurations.** A finding that exists only under a setting the user has to choose (a tier, an alternative grader the source presents as sound) is rated by the table like any other, and its heading and the summary say "non-default". The scale has no discount for it and none for a narrow margin: a mean that clears a threshold by a few thousandths is rated at that level, and the report states the margin and what it rests on. - **One root cause, one finding.** Two symptoms of the same defect are reported together and rated once, at the higher of the two. - **"No false pass"** in a summary means: no answer-free completion scored above 0.10 of the maximum and no non-committal completion received correctness credit. --- # Audit an RL environment's grader with falsepass falsepass scores completions whose correctness is known in advance with the environment's own reward code and reports every disagreement. Its output is a list of leads. Your job is to turn leads into findings a maintainer can act on, or to drop them. ## 1. Run it Requires `uv` on the PATH. If `falsepass` is not installed, prefix every command with `uvx` (`uvx falsepass init ...`), or install it once with `uv tool install falsepass`. ```bash falsepass scan path/to/source --json # optional first look, nothing installed falsepass init owner/environment # Hub environment; or ./folder, an Inspect task file, # inspect_evals/, or a HUD environment folder falsepass audit environment # 25 tasks; add --search 120 for mutation search falsepass results environment --json # headline numbers and failing probes ``` Work in a scratch directory: `init` creates `audits//` in the current directory. If `audit` stops at the control ("inconclusive", "needs a hand-written adapter"), the environment is multi-turn, tool-use, sandboxed or judge-only in a way automatic mode cannot script. Say so, then write policies: `falsepass init --policies --force` writes a skeleton (a solver, an honest loser, and exploit policies that never use the answer); see "Games, simulations and tool tasks" on the agents page of the docs. For a Q&A environment whose answer format can't be inferred, `--manual` writes a simpler one. ## 2. Read the results - `false_pass_ceiling` near the maximum: some known-bad completion was paid. Look at `failing` for which probe and variant. - `lazy_policy_reward` above zero: a fixed strategy that never uses the answer earns that much on average. On a closed answer set (yes/no, A–J) subtract the share one fixed label earns under any correct grader before calling it a problem. - `false_fail_rate`: correct answers refused. Read the variant names: rejecting `\boxed{6.}` or lower-case letters can be deliberate strictness. - `static_hints` with `verdict: confirmed` point at the line that likely causes a failure. - HUD tasks without `--answers`: only answer-free probes ran (P3, P4, A1, P10, P11). A1 paying means a reply that lists everything earns reward; P1/P5/P9 need reference answers. - Each failing entry shows the start and the end of one completion (the answer is usually at the end). `falsepass results environment --completion ` prints it in full. - Through MCP, an audit runs in the background: if `falsepass_audit` returns `"status": "running"`, call `falsepass_audit_status` until it finishes. ## 3. Confirm before you claim For each lead: 1. `falsepass repro environment --probe P5` writes `repro_auto.py`. Run it twice from a clean virtualenv (the command is in its docstring). Identical output, or it does not count. 2. Open the reward code and find the line that causes it. Explain the mechanism in one or two sentences. 3. Decide defect or design. Partial credit in a similarity grader, a deliberately exploitable grader in an environment built to study reward hacking, and format strictness the prompt asks for are design choices: report them as observations. 4. Measure reach on the environment's own data: on how many tasks does it happen? 5. If you have a fix, check it keeps every correct answer that is paid today and run the repro again; with a patched copy, `falsepass verify-fix environment --source ` shows which lines change. ## 4. Write it up For the maintainer: environment and version, the exact completion, the score it got and should get, the line of code and why, how many tasks are affected, a minimal repro, and a suggested fix with what it was tested on. Say what the grader gets right. Do not call an environment "broken", do not inflate how likely a model is to find the hole, and do not file issues or pull requests on someone else's behalf without their explicit go-ahead. A rating scale is in `docs/SEVERITY.md` of the falsepass repository. --- # Your reward function is an assertion. I tested a few dozen of them like one. I've spent my career writing tests and, more usefully, distrusting them. The most expensive bug a test suite can have isn't a failing test. It's a test that passes when the feature is broken. A false pass ships the defect and tells you everything is fine. Reinforcement-learning environments for language models have the same component under another name. An environment is a set of tasks plus a verifier that turns a model's output into a reward. That verifier is an assertion. If it can be satisfied without solving the task, the model being trained will find out, because finding out is exactly what the optimiser is paid for. The field calls this reward hacking. A test engineer would call it a false positive in the oracle. So I did what I'd do with any assertion I didn't write. I fed it inputs whose correct verdict I already knew and counted the disagreements. ## The method I wrote a small tool, [falsepass](https://super-eric.github.io/falsepass/), that loads an environment and calls its own reward code directly on completions I construct. No model, no rollouts, no GPU. Each probe has a known verdict: - **Should pass:** the dataset's reference answer, written the way the prompt asks, and then the same answer with harmless variation (spacing, case, `8.0` for `8`). - **Should fail:** an empty reply; the prompt echoed back; every candidate answer at once; a wrong answer in perfect format; the right answer stated and then denied; the reference answer of a *different* task. - **Malformed:** unclosed tags, a NUL byte, two megabytes of text. - **Repeated:** the same completion scored ten times. On top of that there's a mutation search that pushes wrong completions toward higher reward without ever adding the answer, a static scan of the reward source whose hints are always printed next to the probe that confirms or clears them, and an automatic mode that works out which answer format an environment wants by scoring the reference in each candidate format. Multi-turn and tool environments are played by small scripted policies instead of a model. Why not run a model and read transcripts? Because a rollout tells you what one model happened to do on one day. Calling the grader directly tells you what it *would* do with an output nobody has produced yet, and that's the output an optimiser goes looking for. I ran it on a few dozen public environments from the Prime Intellect Environments Hub: math and arithmetic, multiple choice, medical and finance QA, code-writing tasks graded by heuristics, a few sandboxed coding tasks, games, a memory benchmark, a tool-use task, a policy-compliance simulation. For almost every finding I read the reward code, found the line responsible, and reproduced it twice from a clean virtualenv with a script that doesn't depend on my tool. I'm not naming any of them. Many are small community projects, and the point here is the patterns, which repeat across authors. ## What kept turning up ### Graders that count words Several small environments ask for free-form output (document this function, write a data-cleaning routine, explain a rule) and grade it by looking for surface features. Does the reply contain "return"? "because"? The right method names? For a handful of these I wrote one text that addresses no task in particular and sent it for every task I sampled. It earned between about 40% and 96% of the maximum. In one, a function that returns its input unchanged, with the expected method names in a comment, beat the dataset's own reference solutions. That's the less interesting direction. The other one: in most of these environments the concise reference answer the dataset ships is *not* the reward maximum. The optimum is a longer text with more of the right nouns in it. Training on that reward teaches padding. ### Rewards that pay for the form A common design adds "format" and "reasoning" terms on top of correctness. Sensible on paper, but in the environments I looked at those terms never check the answer. A wrong letter with a well-formed thinking block collects about a third of the maximum every time. One reasoning term measured length, so a real sentence of clinical reasoning earned less than a few hundred characters of *lorem ipsum*. And the mirror image: a correct answer written as the letter plus the option text, instead of the bare letter, lost both the accuracy and the format credit and scored *below* a wrong letter in clean format. ### Strict and lenient in the wrong places One environment asks for the final answer after a marker, `#### 42`. With the marker it's strict about some things: `#### 42.` with a trailing period scores 0. Without the marker, a fallback hands the whole reply to a math parser, and "The answer is definitely not 42." scores 1. A near-miss on the format is graded more harshly than ignoring the format altogether. ### Denials inside the answer box Math graders that extract `\boxed{...}` and compare symbolically held up well against everything outside the box. Inside it was another story. In a few environments that share a common math-checking library, `\boxed{12 \text{ is wrong}}` (with 12 the right answer) was paid in full on nearly every task, because a normalisation step drops text after a number (it's there for units). A false equation like `\boxed{13 = 12}` was accepted too, because only the right-hand side got compared. I'd first cleared two of these environments. My probes had put the denial outside the box. Substring matching has the same problem without any parsing at all. If the check is "the reference appears in the reply", then "not Lisbon" is Lisbon, and a reply that lists every option is always right. In one environment the questions listed their own candidate answers, so echoing the question back earned full reward on every task. ### The exception nobody saw One environment's largest reward term called `re.search` in a module that never imports `re`. Every call raised `NameError`. The framework caught it, logged it and recorded zero, so every run finished with plausible-looking numbers and a ceiling nobody intended. I only noticed because the tool counts scoring exceptions, and one short run had hundreds of them. A single unit test that scores one reference answer would have caught it. ### A violation that reads as compliance An agent is tested on whether it respects a spending limit, and the reward is 1 unless it goes over. The amount parser didn't cope with a currency sign plus a thousands separator, so a large amount written the ordinary way was read as zero and scored as the limit being respected. On the environment's own data this fires rarely. Real defect, narrow reach, and I only measured the reach during a review of my report. ### The game that keeps scoring after it ends A word-game environment doesn't stop its episodes. After a win or a loss it keeps replying to every further turn until the turn limit, and on most losses it reveals the word. The reward functions rebuild the game by re-reading every assistant message. With small scripted players: an honest solver scored in the middle of the range. A player that lost, saw the word and typed it in scored about two thirds as much. A player that won and then kept making moves in the spare turns picked up extra reward and scored highest. A player that won and then said nothing scored lowest, because an empty last message erased the win. The best reward went to whoever kept talking. ### A grader that never reads the answer A tool-use environment has the model keep notes in files and answer questions from them later. Its stricter grader checks that each true fact is somewhere on disk. It never looks at the answer. A policy that never deletes an old value and answers "unknown" to everything gets full marks, although the grader's docstring says stale values get no credit. A single file listing every possible value gets 0.97, a limitation the docstring does name. ### The ones that held Not everything leaked. One multiple-choice environment gave up nothing at all: wrong, empty, echoed, listed and mutated completions scored zero, and so did hedges written inside the answer box. A large suite of procedurally generated reasoning tasks came close, with one narrow class of false pass on a few task types. Every audit I wrote lists the checks that passed, not only the ones that failed. ## Does a real model actually hit these? Everything above uses completions I built. So I also ran a mid-sized local model on about twenty tasks per environment and scored its own replies with the same code, and a second small model on some of them. One sample per task, so it's an observation, not a measurement. On most exactly graded tasks its normal output didn't reach the defects. On one `####` arithmetic set it answered cleanly every time. On another set of the same kind it wrote things like `#### $10` and `#### 16 hours` on a few tasks and was graded wrong. Elsewhere it walked straight in. In the word game it kept chatting after each game ("That was a great game!"), the environment kept asking, and the chatter was scored: games the environment had declared lost were paid as solved. One of its replies crashed the environment outright, which led to something no probe had found: a guess followed by a full stop crashes it too. In the note-keeping environment it saved the facts correctly and then gave non-answers to about half the questions, and the stricter grader still paid 0.95. In most of the word-counting environments its long, fluent replies outscored the reference answers. The probes say where a grader can be wrong. The rollout says which of those places a model is already standing in. ## Checking my own work The automatic mode found leads fast and stated several of them wrongly. In one environment the "reference" it compared against was a function name. In another every flagged row was an artefact of how I'd built the probe. Reading the reward code fixed those. Then every report got a separate claim-by-claim check against the source and the stored output. The main findings held up. The reports didn't come through clean: I'd rated the same kind of defect High in one place and Low in another, some reports quoted numbers from runs I'd since redone, two suggested fixes wouldn't have fixed anything, and one "generic" probe text happened to contain three of the answers. I wrote a severity scale down, re-rated everything, stored every output next to its report, and now generate the totals instead of typing them. An audit is a test artefact too. It needs its own oracle. ## If you maintain an environment - **Write the negative cases first.** Few of the packages I audited checked that a wrong answer in perfect format scores zero. The ones that tested their grader mostly checked only that the right answer passes. - **Score your own reference answers.** If the dataset's answer isn't the maximum, something is off, either in the grader or in the data. - **Ask what the laziest answer earns.** Echo the prompt, list every option, send one fixed text for every task. - **Test the sum, not the terms.** Multi-part rewards fail in how the parts interact. - **Make "it raised" visible.** A swallowed exception is a silent zero. - **Put the hedge inside the box.** If you parse a region of the reply, probe inside it. falsepass does most of this in one command, and there's a GitHub Action for running it in CI. It works with verifiers environments (published or local), Inspect AI tasks and HUD environments. ## Sending fixes upstream Where a maintainer had an active public repository and the fix was small, I sent a pull request, with a test wherever the repository had tests for that code. One has been merged; the others are still open. Where the right behaviour was a judgment call, I asked instead. Strictness the prompt asks for isn't a defect on its own, and I only noted it as low priority or where it sat next to leniency in the same grader. A grader that's exploitable on purpose, because the environment studies reward hacking, I didn't report at all. ## Limits The findings come from synthetic probes. The model check is mainly one mid-sized local model, one sample per task. I didn't re-solve datasets to check their labels. Sandboxed code-execution environments, probably the most hackable kind, still need policies written by hand, and I only covered a few. LLM-judged environments were only tested with stand-ins for the judge, so nothing here says how a real judge behaves. No domain expert has gone over the medical or finance reports. The tool is open source under Apache-2.0: [source on GitHub](https://github.com/Super-Eric/falsepass), [documentation](https://super-eric.github.io/falsepass/docs/). If you run it on your own environment and it's wrong about something, I'd like to hear about it.