Output files#

Everything an audit writes lives in audits/<env>/ (or the folder given with --out).

File	For	Written by
`env.toml`	how the environment is installed: `id`, `framework` (`verifiers`, `inspect`, `hud`; default verifiers), `task` (Inspect or HUD task spec), `package`, `version`, `module`, `index`, `verifiers` / `inspect_ai` / `hud` (version pins), optional `source` (a local folder), `answers` (HUD reference file), `extra`, `python`, `load_args`	`falsepass init`
`src/`	the Hub wheel's source, for the static scan (local environments are scanned in place)	`falsepass init`
`probe-results.json`	every probe run, machine-readable (below)	`falsepass audit`
`summary.md`	the same for people: headline numbers, one row per probe variant, reward terms, static hints	`falsepass audit`
`completions.json`	the full text of every completion a result refers to, keyed by `completion_id`	`falsepass audit`
`job.json`, `job.log`	an audit started from the MCP server: its state (`running`, `finished`, `failed`), pid, start time, and its console output	`falsepass mcp`
`repro_auto.py`	stand-alone script that re-scores the failing completions	`falsepass repro`
`REPORT.draft.md`	candidate findings to review	`falsepass draft`

probe-results.json#

{
  "meta": {"environment": "owner/env", "date": "…", "examples": 25, "seed": 0, "python": "3.12.x",
           "versions": {"verifiers": "…", "…": "…"}, "falsepass": "…", "max_reward": 1.0,
           "bad_tolerance": 0.0, "weights": {"term": 1.0}, "adapter": {"…": "how the format was inferred"} },
  "results": [ { "probe": "P5", "name": "shotgun", "variant": "all_in_one_slot", "example": "row-98",
                 "expected": "zero", "score": 1.0, "passed": false, "detail": "expected zero, got 1.0",
                 "completion": "preview", "metrics": {"term": 1.0}, "errors": [],
                 "answer_free": false, "completion_id": "…" } ],
  "static": [ { "rule": "reference_substring_match", "file": "env.py", "line": 26, "where": "load_environment",
                "evidence": "source line", "meaning": "…", "confirm_with": ["P9", "P5", "S3"],
                "verdict": "confirmed: P9 failed 10/20, P5 failed 10/30" } ],
  "rewardlint": null
}

expected: full (a correct completion, must get the maximum), zero (a known-bad completion, must get nothing), stable (the same completion scored repeatedly, must get the same score), at_most_reference (must not beat the concise correct answer).
passed: whether the grader did what expected asks.
answer_free: the completion was built without the reference answer, so a policy that knows nothing about the task could produce it. These feed the lazy-policy number.
metrics: the value of each reward term, so a leak can be traced to the term that pays.
errors: exceptions or timeouts raised by the reward code on this completion.

falsepass results --json#

A digest of the same run, small enough to hand to a model (numbers from falsepass demo):

{
  "environment": "owner/env",
  "meta": {"date": "…", "falsepass": "…", "versions": {"…": "…"}, "examples": 25, "max_reward": 1.0,
           "adapter": {"format": "<answer>", "answers": "open", "turns": "single-turn", "validity": "ok"}},
  "headline": {"false_pass_ceiling": 1.0, "false_pass_via": "P5 shotgun/all_in_one_slot",
               "lazy_policy_reward": 0.0, "lazy_policy_via": "P3 empty/empty",
               "false_fail_rate": 0.108, "known_good": 93, "known_bad": 233, "scoring_errors": 0},
  "failing": [ {"probe": "P5", "name": "shotgun", "variant": "all_in_one_slot", "expected": "zero",
                "runs": 10, "failed": 10, "scores": [1.0, 1.0, 1.0, 1.0, 1.0], "detail": "…", "example": "row-98",
                "completion": "start … [316 chars] … end", "completion_id": "da79bcea66a8",
                "answer_free": false} ],
  "static_hints": [ "as in probe-results.json" ],
  "note": "Leads, not findings: …"
}

failing lists false passes first (a known-bad completion was paid), the most consistent ones on top, then false fails. meta and static_hints are trimmed to what an agent needs; everything else is in probe-results.json.

The headline numbers:

false-pass ceiling: the highest reward any known-bad completion received.
lazy-policy reward: the best average reward of one fixed strategy that never uses the reference answer. On a closed answer set, one fixed label earns its share of the dataset under any correct grader; subtract that base rate before reading it.
false-fail rate: the share of known-good completions denied full reward. Some of these are deliberate strictness about format; read the variant names.

completion shows the start and the end of the first failing completion (the answer is usually at the end); falsepass results <env> --completion <completion_id> prints all of it.

falsepass init --json#

{"name": "gsm8k", "folder": "/abs/path/audits/gsm8k", "framework": "verifiers",
 "env": {"id": "primeintellect/gsm8k", "package": "gsm8k", "version": "…"}, "manual": false,
 "others": [], "next": "falsepass audit gsm8k"}

name is what every other command takes. Names are forgiving: arc_easy finds arc-easy.

Exit codes#

falsepass audit exits 1 when a --fail-on condition is met and with the probe run's own code when it fails; every other command exits 0 on success and 1 with a message on stderr when its input is missing.