# Using falsepass from an agent falsepass is built to be driven by a coding agent as much as by a person. Every command is non-interactive, writes its results to files, and has a machine-readable form. There are three ways in, from most to least structured. ## 1. MCP server `falsepass mcp` serves the commands as tools over stdio. Any MCP client can start it without installing anything first: ```bash uvx --from "falsepass[mcp]" falsepass mcp ``` For Claude Code: ```bash claude mcp add falsepass -- uvx --from "falsepass[mcp]" falsepass mcp ``` For clients configured with JSON (Cursor, Claude Desktop, and others): ```json { "mcpServers": { "falsepass": { "command": "uvx", "args": ["--from", "falsepass[mcp]", "falsepass", "mcp"] } } } ``` | Tool | Arguments | Returns | |---|---|---| | `falsepass_scan` | `path` | static hints for a source folder or file; nothing is installed | | `falsepass_init` | `target` (Hub id, local folder, Inspect task, HUD folder); optional `name`, `version`, `framework`, `task`, `answers`, `extra`, `manual`, `force` | `{"name": ..., "folder": ..., "framework": ..., "env": {...}, "next": ...}`: `name` is what the other tools take | | `falsepass_audit` | `env`, `n` (tasks, default 25), `search` (mutation budget), `wait` (seconds, default 45) | the results digest below with `"status": "finished"`, or `"status": "running"` if the audit needs longer | | `falsepass_audit_status` | `env`, `wait` | the same, for an audit already started: results, or `"running"` with the latest progress lines | | `falsepass_results` | `env` | the digest of the last run (progress instead while one is running) | | `falsepass_completion` | `env`, `completion_id` | the full text of one completion; the digest shows only its start and end | | `falsepass_repro` | `env`, `probe` (e.g. `["P5"]`) | where the stand-alone repro script was written | | `falsepass_discover` | `days`, `limit` | recently maintained Hub environments and what each needs | Every tool takes `workdir`; audits are created in `workdir/audits//`. An audit installs the environment before it probes it, so it can take several minutes, longer than many clients allow for one tool call. It therefore runs in the background: `falsepass_audit` waits up to `wait` seconds and either returns the results or `{"status": "running", "elapsed_s": ..., "log_tail": [...]}`. Call `falsepass_audit_status` with the same `env` until it finishes. Calling `falsepass_audit` again while one is running doesn't start a second audit. The job is recorded in `audits//job.json`, its output in `audits//job.log`. When something goes wrong (no such audit, a failed install, a dataset that needs a login), the tool returns an MCP error whose text says what happened and what to try next, so a client sees `isError` and the reason without reading server logs. Names are forgiving: `arc_easy` finds the audit `init` named `arc-easy`. For a HUD environment, pass reference answers with `answers`: a JSON file holding either an object from task slug to answer (`{"add-2-3": "5"}`) or a list in task order. Without it only the answer-free probes run. Adding answers to an existing audit takes `force: true`. `uv` must be on the PATH: each audit installs the environment in its own throw-away virtualenv. ## 2. Agent skill [`skills/falsepass/SKILL.md`](https://super-eric.github.io/falsepass/docs/agents/skill/) is the whole audit procedure in the Agent Skills format: run, read the numbers, confirm a lead before claiming it, decide defect or design, and write it up for the maintainer. Copy the folder into the agent's skills directory (for Claude Code, `~/.claude/skills/`) and ask the agent to audit an environment. ## 3. Plain CLI ```bash falsepass init owner/environment # or ./folder, task.py, inspect_evals/, a HUD folder falsepass audit environment --fail-on 'ceiling>0.1' # exit 1 when the condition is met falsepass results environment --json # compact digest, small enough for a context window falsepass scan path/to/source --json # static pre-scan only ``` The digest from `results --json` and from the MCP audit tools (abridged): ```json { "environment": "owner/env", "headline": {"false_pass_ceiling": 1.0, "false_pass_via": "P5 shotgun/all_in_one_slot", "lazy_policy_reward": 0.0, "false_fail_rate": 0.108, "scoring_errors": 0}, "failing": [{"probe": "P5", "variant": "all_in_one_slot", "runs": 10, "failed": 10, "completion": "start of the completion … [316 chars] … its end, where the answer is", "completion_id": "da79bcea66a8"}], "static_hints": ["..."], "note": "Leads, not findings: ..." } ``` All fields are described in [Output files](https://super-eric.github.io/falsepass/docs/output/). ## Games, simulations and tool tasks Automatic mode needs a reference answer it can write into a reply. A game, a customer-service simulation or a tool task has none: the reward depends on what the model does over several turns. There, an audit ends `"status": "inconclusive"` (the reference and an empty reply earned about the same) or, for tool tasks, runs only the generic tool policies. The digest then lists only `facts_that_hold`: rows built without the answer, such as "an empty reply earned 1.0 on 20 of 20 tasks", which are true either way. The next step is to write policies, which takes a capable agent a few minutes per environment: ```bash falsepass init owner/environment --policies --force # adds adapter.py: a policy skeleton ``` A policy is a function from the conversation so far (a list of `{"role", "content"}` dicts) to the next assistant message, either a string or `{"content": ..., "tool_calls": [{"name": ..., "arguments": {...}}]}`. With `Policy(name, act, wants_tools=True)` it also gets the tool schemas. Each policy runs through the environment's own rollout loop and rubric. Fill in: - `solver(example)`: a policy that does the task properly (it may read the row's answer). It should earn full reward; set `max_reward` to what it earns. - `gives_up(example)`: plays honestly without solving. It should earn nothing. - `exploits(example)`: policies that never use the answer. The skeleton starts from the ready-made ones in `falsepass.policies`, each of which found a real hole: `after_the_end(wrong, say)` (answer wrong, then repeat what the environment gives away), `keeps_talking(first)` (carry on after "game over"), `never_acts()`. Adapt their messages to the environment's format. - `special_number_scripts(example)`: one message sequence the environment accepts; `policies.special_numbers()` turns it into variants with `nan`, `inf`, `-1`, `1e999` and `-0` in place of its numbers (ids such as `#1234` are left alone). A limit written as `amount > LIMIT` is false for NaN. Then run `falsepass audit ` as usual. Each policy's episode (what it said, and the whole conversation) is kept: `falsepass_completion` shows it, and `falsepass repro` replays the policy's turns as a fixed script, so a lead found by a policy is reproduced without falsepass. ## Reading the output - **`false_pass_ceiling` near the maximum**: some known-bad completion was paid. `failing` says which probe and variant. - **`lazy_policy_reward` above zero**: one fixed strategy that never uses the answer earns that much on average. On a closed answer set (yes/no, A–J), subtract what one fixed label earns under any correct grader. - **`false_fail_rate`**: correct answers refused. Some of it is deliberate strictness. - **`static_hints` with `verdict: confirmed`**: the line that most likely causes a failure. ## Rules worth giving your agent The output is a list of leads. Before an agent reports anything: 1. Read the whole completion if the preview isn't enough: `falsepass_completion` (or `falsepass results --completion `). 2. Reproduce it: `falsepass repro --probe P5` writes a script that depends only on the environment's own packages. Run it twice from a clean virtualenv. 3. Find the line in the reward code that causes it. 4. Decide whether it is a defect or a design choice. A grader that is exploitable on purpose, or strictness the prompt asks for, is not a bug. 5. Measure how many of the environment's own tasks it affects. The audit samples 25 tasks by default; the repro script is the place to extend it to the full dataset. Don't let an agent call an environment broken on the strength of a probe table, and don't let it file issues on other people's repositories without a person agreeing. ## Text versions Every page on this site is also published as Markdown (link at the bottom of each page), and [`llms.txt`](https://super-eric.github.io/falsepass/llms.txt) / [`llms-full.txt`](https://super-eric.github.io/falsepass/llms-full.txt) index all of them.