# Severity scale used in the audit reports

Every audit report rates its findings against this scale. It was written after an
independent review found an earlier, shorter scale had been applied unevenly (the same
kind of defect rated High in one report and Low in another).

Two terms first.

- **Answer-free completion** — text produced without knowing the task's answer: empty,
  the prompt echoed back, one fixed text used for every task, a wrong value in the right
  format, the reference of a different task.
- **Non-committal completion** — text that contains the correct answer but denies it,
  retracts it, or offers it among alternatives ("definitely not 18", "either 25 or 18").

Rewards are read relative to the maximum the rubric can pay.

| Severity | Applies when |
|---|---|
| **Critical** | An answer-free completion earns at least 0.75 of the maximum — or more than the dataset's own reference answers while earning at least 0.75. Also: a completion that plainly violates the rule the environment exists to test is scored as compliant. |
| **High** | An answer-free completion earns 0.25–0.75 of the maximum on average. Or a non-committal completion receives full correctness credit on all sampled tasks. Or the dataset's reference answers earn less than about half the maximum. Or a correct answer scores below a wrong one. Or a reward term carrying a quarter or more of the weight never fires. |
| **Medium** | An answer-free completion earns 0.10–0.25. A non-committal completion gets full credit on a subset of task types. A correct answer in the requested format is rejected *and* a less compliant form is accepted. Dataset defects (tasks that need no skill, duplicate or conflicting rows). Non-deterministic scoring or environment replies. A reward term that is constant. |
| **Low** | Defensible strictness about format (a period inside the answer slot, letter case). Which of several answers in one reply is graded — a policy choice, as long as it is consistent. Reward code that raises on unusual input. Documentation that disagrees with the code. Anything below 0.10. |

Notes on applying it.

- **Similarity graders.** Partial credit is their design. The severity follows the
  reward an answer-free completion gets, using the rows above; the report says that the
  design is intentional.
- **Disclosure does not lower the rating.** If maintainers document a weakness, the
  report says so prominently and the rating stays where the scale puts it — the default
  configuration is what gets trained on.
- **Sample size is stated, not folded into the rating.** A finding seen on 2 of 2
  searched tasks is rated by the scale and labelled with the 2.
- **Environments that only test for the absence of a violation.** Doing nothing is
  compliant by such an environment's own definition, so inaction scoring full is rated
  High, not Critical, and the report says why it matters only if the reward is trained on.
- **Closed answer sets.** On a task whose answer is one of a few labels, one fixed label
  used for every task earns that label's share of the dataset under any correct grader
  (0.5 on a balanced yes/no task). That base rate is subtracted before a fixed guess is
  rated, and the label balance itself is an observation, not a finding.
- **Reach.** When a defect needs the completion to take a particular form, the report
  says how often that form arises from the environment's own data. If it arises on
  fewer than about one task in twenty, the finding is rated one level below what the
  table gives. The form must be counted in the
  environment's own data (its tasks, references, inputs). How often a model's replies take
  the form is an observation and never moves a rating: a few dozen model replies cannot show "under one in
  twenty" anyway.
- **Graders that are exploitable on purpose.** Some environments are built to study
  reward hacking and ship a grader their own source describes as the exploitable one.
  What that grader lets through is recorded as an observation and not rated; findings
  are rated against the grader the source presents as the sound one. The report says
  which of the two is the default.
- **Results obtained with a substitute.** When a component could not be run as the
  environment runs it — a hosted LLM judge replaced by a local model — what was measured
  with the substitute is reported as an observation and not rated. A finding that
  depends on how that component behaves is rated on what could be measured and marked
  provisional.
- **Non-default configurations.** A finding that exists only under a setting the user
  has to choose (a tier, an alternative grader the source presents as sound) is rated by
  the table like any other, and its heading and the summary say "non-default". The scale
  has no discount for it and none for a narrow margin: a mean that clears a threshold by
  a few thousandths is rated at that level, and the report states the margin and what
  it rests on.
- **One root cause, one finding.** Two symptoms of the same defect are reported
  together and rated once, at the higher of the two.
- **"No false pass"** in a summary means: no answer-free completion scored above 0.10
  of the maximum and no non-committal completion received correctness credit.