falsepass

Your reward function is an assertion. I tested a few dozen of them like one.

I've spent my career writing tests and, more usefully, distrusting them. The most expensive bug a test suite can have isn't a failing test. It's a test that passes when the feature is broken. A false pass ships the defect and tells you everything is fine.

Reinforcement-learning environments for language models have the same component under another name. An environment is a set of tasks plus a verifier that turns a model's output into a reward. That verifier is an assertion. If it can be satisfied without solving the task, the model being trained will find out, because finding out is exactly what the optimiser is paid for. The field calls this reward hacking. A test engineer would call it a false positive in the oracle.

So I did what I'd do with any assertion I didn't write. I fed it inputs whose correct verdict I already knew and counted the disagreements.

The method#

I wrote a small tool, falsepass, that loads an environment and calls its own reward code directly on completions I construct. No model, no rollouts, no GPU. Each probe has a known verdict:

  • Should pass: the dataset's reference answer, written the way the prompt asks, and then the same answer with harmless variation (spacing, case, 8.0 for 8).
  • Should fail: an empty reply; the prompt echoed back; every candidate answer at once; a wrong answer in perfect format; the right answer stated and then denied; the reference answer of a different task.
  • Malformed: unclosed tags, a NUL byte, two megabytes of text.
  • Repeated: the same completion scored ten times.

On top of that there's a mutation search that pushes wrong completions toward higher reward without ever adding the answer, a static scan of the reward source whose hints are always printed next to the probe that confirms or clears them, and an automatic mode that works out which answer format an environment wants by scoring the reference in each candidate format. Multi-turn and tool environments are played by small scripted policies instead of a model.

Why not run a model and read transcripts? Because a rollout tells you what one model happened to do on one day. Calling the grader directly tells you what it would do with an output nobody has produced yet, and that's the output an optimiser goes looking for.

I ran it on a few dozen public environments from the Prime Intellect Environments Hub: math and arithmetic, multiple choice, medical and finance QA, code-writing tasks graded by heuristics, a few sandboxed coding tasks, games, a memory benchmark, a tool-use task, a policy-compliance simulation. For almost every finding I read the reward code, found the line responsible, and reproduced it twice from a clean virtualenv with a script that doesn't depend on my tool. I'm not naming any of them. Many are small community projects, and the point here is the patterns, which repeat across authors.

What kept turning up#

Graders that count words#

Several small environments ask for free-form output (document this function, write a data-cleaning routine, explain a rule) and grade it by looking for surface features. Does the reply contain "return"? "because"? The right method names?

For a handful of these I wrote one text that addresses no task in particular and sent it for every task I sampled. It earned between about 40% and 96% of the maximum. In one, a function that returns its input unchanged, with the expected method names in a comment, beat the dataset's own reference solutions.

That's the less interesting direction. The other one: in most of these environments the concise reference answer the dataset ships is not the reward maximum. The optimum is a longer text with more of the right nouns in it. Training on that reward teaches padding.

Rewards that pay for the form#

A common design adds "format" and "reasoning" terms on top of correctness. Sensible on paper, but in the environments I looked at those terms never check the answer. A wrong letter with a well-formed thinking block collects about a third of the maximum every time. One reasoning term measured length, so a real sentence of clinical reasoning earned less than a few hundred characters of lorem ipsum.

And the mirror image: a correct answer written as the letter plus the option text, instead of the bare letter, lost both the accuracy and the format credit and scored below a wrong letter in clean format.

Strict and lenient in the wrong places#

One environment asks for the final answer after a marker, #### 42. With the marker it's strict about some things: #### 42. with a trailing period scores 0. Without the marker, a fallback hands the whole reply to a math parser, and "The answer is definitely not 42." scores 1. A near-miss on the format is graded more harshly than ignoring the format altogether.

Denials inside the answer box#

Math graders that extract \boxed{...} and compare symbolically held up well against everything outside the box. Inside it was another story. In a few environments that share a common math-checking library, \boxed{12 \text{ is wrong}} (with 12 the right answer) was paid in full on nearly every task, because a normalisation step drops text after a number (it's there for units). A false equation like \boxed{13 = 12} was accepted too, because only the right-hand side got compared. I'd first cleared two of these environments. My probes had put the denial outside the box.

Substring matching has the same problem without any parsing at all. If the check is "the reference appears in the reply", then "not Lisbon" is Lisbon, and a reply that lists every option is always right. In one environment the questions listed their own candidate answers, so echoing the question back earned full reward on every task.

The exception nobody saw#

One environment's largest reward term called re.search in a module that never imports re. Every call raised NameError. The framework caught it, logged it and recorded zero, so every run finished with plausible-looking numbers and a ceiling nobody intended. I only noticed because the tool counts scoring exceptions, and one short run had hundreds of them. A single unit test that scores one reference answer would have caught it.

A violation that reads as compliance#

An agent is tested on whether it respects a spending limit, and the reward is 1 unless it goes over. The amount parser didn't cope with a currency sign plus a thousands separator, so a large amount written the ordinary way was read as zero and scored as the limit being respected. On the environment's own data this fires rarely. Real defect, narrow reach, and I only measured the reach during a review of my report.

The game that keeps scoring after it ends#

A word-game environment doesn't stop its episodes. After a win or a loss it keeps replying to every further turn until the turn limit, and on most losses it reveals the word. The reward functions rebuild the game by re-reading every assistant message. With small scripted players: an honest solver scored in the middle of the range. A player that lost, saw the word and typed it in scored about two thirds as much. A player that won and then kept making moves in the spare turns picked up extra reward and scored highest. A player that won and then said nothing scored lowest, because an empty last message erased the win. The best reward went to whoever kept talking.

A grader that never reads the answer#

A tool-use environment has the model keep notes in files and answer questions from them later. Its stricter grader checks that each true fact is somewhere on disk. It never looks at the answer. A policy that never deletes an old value and answers "unknown" to everything gets full marks, although the grader's docstring says stale values get no credit. A single file listing every possible value gets 0.97, a limitation the docstring does name.

The ones that held#

Not everything leaked. One multiple-choice environment gave up nothing at all: wrong, empty, echoed, listed and mutated completions scored zero, and so did hedges written inside the answer box. A large suite of procedurally generated reasoning tasks came close, with one narrow class of false pass on a few task types. Every audit I wrote lists the checks that passed, not only the ones that failed.

Does a real model actually hit these?#

Everything above uses completions I built. So I also ran a mid-sized local model on about twenty tasks per environment and scored its own replies with the same code, and a second small model on some of them. One sample per task, so it's an observation, not a measurement.

On most exactly graded tasks its normal output didn't reach the defects. On one #### arithmetic set it answered cleanly every time. On another set of the same kind it wrote things like #### $10 and #### 16 hours on a few tasks and was graded wrong.

Elsewhere it walked straight in. In the word game it kept chatting after each game ("That was a great game!"), the environment kept asking, and the chatter was scored: games the environment had declared lost were paid as solved. One of its replies crashed the environment outright, which led to something no probe had found: a guess followed by a full stop crashes it too. In the note-keeping environment it saved the facts correctly and then gave non-answers to about half the questions, and the stricter grader still paid 0.95. In most of the word-counting environments its long, fluent replies outscored the reference answers.

The probes say where a grader can be wrong. The rollout says which of those places a model is already standing in.

Checking my own work#

The automatic mode found leads fast and stated several of them wrongly. In one environment the "reference" it compared against was a function name. In another every flagged row was an artefact of how I'd built the probe. Reading the reward code fixed those.

Then every report got a separate claim-by-claim check against the source and the stored output. The main findings held up. The reports didn't come through clean: I'd rated the same kind of defect High in one place and Low in another, some reports quoted numbers from runs I'd since redone, two suggested fixes wouldn't have fixed anything, and one "generic" probe text happened to contain three of the answers. I wrote a severity scale down, re-rated everything, stored every output next to its report, and now generate the totals instead of typing them.

An audit is a test artefact too. It needs its own oracle.

If you maintain an environment#

  • Write the negative cases first. Few of the packages I audited checked that a wrong answer in perfect format scores zero. The ones that tested their grader mostly checked only that the right answer passes.
  • Score your own reference answers. If the dataset's answer isn't the maximum, something is off, either in the grader or in the data.
  • Ask what the laziest answer earns. Echo the prompt, list every option, send one fixed text for every task.
  • Test the sum, not the terms. Multi-part rewards fail in how the parts interact.
  • Make "it raised" visible. A swallowed exception is a silent zero.
  • Put the hedge inside the box. If you parse a region of the reply, probe inside it.

falsepass does most of this in one command, and there's a GitHub Action for running it in CI. It works with verifiers environments (published or local), Inspect AI tasks and HUD environments.

Sending fixes upstream#

Where a maintainer had an active public repository and the fix was small, I sent a pull request, with a test wherever the repository had tests for that code. One has been merged; the others are still open. Where the right behaviour was a judgment call, I asked instead. Strictness the prompt asks for isn't a defect on its own, and I only noted it as low priority or where it sat next to leniency in the same grader. A grader that's exploitable on purpose, because the environment studies reward hacking, I didn't report at all.

Limits#

The findings come from synthetic probes. The model check is mainly one mid-sized local model, one sample per task. I didn't re-solve datasets to check their labels. Sandboxed code-execution environments, probably the most hackable kind, still need policies written by hand, and I only covered a few. LLM-judged environments were only tested with stand-ins for the judge, so nothing here says how a real judge behaves. No domain expert has gone over the medical or finance reports.

The tool is open source under Apache-2.0: source on GitHub, documentation. If you run it on your own environment and it's wrong about something, I'd like to hear about it.