Issue 029 - AI safety - Rare-event testing throughput
Can 30 days of testing detect rare rogue AI behavior?
Reuters reported that the U.S. government finalized a voluntary framework under which developers could provide advanced AI models for safety testing up to 30 days before release. Suppose dangerous autonomous behavior occurs in one out of every 100,000 agent tasks.
The problem
Suppose a dangerous autonomous action occurs during one out of every 100,000 agent tasks.
Estimate how many test tasks must be run to give testers a reasonably high chance of observing the behavior at least once. Then estimate the infrastructure and human labor needed to complete that testing within 30 days.
How does the result change if the dangerous behavior is ten times rarer?
Because Fermi problems target an order of magnitude, I normally use no more than two significant digits and write most calculations in scientific notation; the Fermi reference explains both conventions.
Before checking sources
Matt's first pass
If humans need to review all tasks because there is no way to algorithmically capture or flag potentially dangerous autonomous actions, and if a human might only catch 5% of dangerous actions, then I would want the systems to carry out 2 million tasks.
base rate ~= 1 dangerous action / 1 x 10^5 tasks
human catch rate ~= 5%
tasks ~= 1 x 10^5 x 20
~= 2 x 10^6 tasksNext I assumed it would take a human evaluator an average of 30 seconds per action to assess:
review time ~= 2 x 10^6 tasks x 0.5 min/task
~= 1 x 10^6 minutesAssuming 20 working days in a 30-day window and 7 hours of work per day, each human evaluator has about 140 hours, or roughly 8,400 minutes, of checking time.
I wrote down that it would take over 1,200 human evaluators to check 2 million tasks within 30 days. Humans would absolutely be the bottleneck, because I thought it would take a fraction of that time for the computer systems to run that number of tasks. I did not think multiple computers would be needed to run those tasks within the time frame.
If there were a way to algorithmically flag and limit the tasks that should be reviewed, that would cut down the required time and number of humans.
Calibration Score
Matt's Calibration Score: 35 / 100
Higher is better: earn points for accurate pegs, sound models, correct math, and a result close to the sourced answer. The image shows percent full of it: 100 minus the Calibration Score.
Pegs: 10/30. The rare-event and human-review pegs were plausible but not close enough.
Model: 15/30. The model was directionally right, but it needed the expected-event probability shortcut and realistic agent-runtime assumptions.
Math: 0/10. The reviewer-count arithmetic was off by about a factor of ten.
Result: 10/30. The answer was directionally useful but not close enough to count as a strong Fermi result.
Grounding facts
A 30-day window can be enough for hundreds of thousands of tests if the tests are parallelized. But it is not enough by magic. Rare-event detection converts quickly into many simulated environments, logging pipelines, triage rules, and reviewer hours.
The deeper issue is whether the test distribution matches the real risk distribution. Three hundred thousand easy tasks may tell you less than a few thousand carefully designed adversarial tasks.
After checking sources
Check and recalibrate
For a rare event with probability p per independent task, the chance of seeing at least one event in n tasks is approximately:
P(at least one) ~= 1 - e^(-n p)If you run 1/p tasks, you expect one event and have about a 63% chance of seeing at least one. For about 95% detection, use roughly 3/p:
p ~= 1 / 100,000
~= 1 x 10^-5
n for 95% ~= 3 / p
~= 3 / 1 x 10^-5
~= 3 x 10^5 tasksSo the corrected task count is about 300,000 tasks for a 95% chance, if the dangerous behavior is visible when it happens and the tasks are independent enough to count.
If the behavior is ten times rarer:
p ~= 1 x 10^-6
n for 95% ~= 3 / 1 x 10^-6
~= 3 x 10^6 tasksNow estimate compute throughput. A 30-day window is about:
30 days ~= 30 x 24 x 60
~= 4.3 x 10^4 minutesIf realistic agent tasks average 5 to 10 minutes each:
compute minutes ~= 3 x 10^5 tasks x (5 to 10 min/task)
~= 1.5 x 10^6 to 3 x 10^6 minutes
parallel agents ~= compute minutes / 4.3 x 10^4 minutes
~= 35 to 70 agentsFor the ten-times-rarer case, multiply by 10: about 350 to 700 parallel agents.
Human review depends almost entirely on triage. If humans review every one of 300,000 tasks for 30 seconds:
human review ~= 3 x 10^5 x 0.5 min
~= 1.5 x 10^5 minutes
~= 2.5 x 10^3 hours
reviewer capacity ~= 20 days x 7 h/day
~= 1.4 x 10^2 h/reviewer
reviewers ~= 2.5 x 10^3 / 1.4 x 10^2
~= 20 reviewersIf humans only catch 5% of genuine dangerous actions, then the effective detection probability is smaller. To get about 95% chance of a caught event:
effective p ~= 1 x 10^-5 x 0.05
~= 5 x 10^-7
n ~= 3 / 5 x 10^-7
~= 6 x 10^6 tasksAt 30 seconds of review per task, that is about 50,000 human-hours, or around 360 reviewers in a 20-workday review window. At 5 to 10 minutes per agent task, it is also about 700 to 1,400 parallel agents.
The biggest practical limit is probably not the calendar alone. It is designing tests that actually elicit the behavior. Brute force can find randomly occurring behavior, but if the dangerous action only appears in specific contexts, then task diversity, realism, adversarial setup, monitoring, and triage matter more than raw task count.
Post-check reflection
Matt's reflection
My answer was probably directionally correct but not close enough for a Fermi answer.
I should keep track of the formula for rare-event detection. It looks like if I multiply the count of events necessary to produce one detection by three, that improves confidence in a detection to about 95%. That might be the better peg to remember.
My math was off on the number of human reviewers necessary to cover that many task reviews. It should have been closer to 120 for my numbers, and 360 for corrected task numbers.
My intuition was also incorrect about whether a single computer could run the full test. I really should have known better: my direct experience with agents lets me know tasks can range in execution time from a second to several minutes, and it probably would have been more appropriate to use something like 5 to 10 minutes for the average task that might produce a dangerous action.
My final point about selecting tasks that are more likely to produce bad behaviors was important. That design change will substantially change the requirements of the overall test. Brute force is costly, requiring hundreds of testers and hundreds of computers or individual agents. I would still consider humans the greater limitation, because they need food and sleep, for example. They are much more expensive to use.
Recommended memory peg
For rare events, remember n ~= 1/p gives about a 63% chance of at least one detection, while n ~= 3/p gives about a 95% chance. Testing throughput is parallel agents ~= tasks x minutes per task / minutes available.
Reader results
Bars show how submitted estimates sort into the answer choices from the gut-check prompt.