Neither 26% nor 86% is yours: Trail of Bits reopens 1Password's AI patch test

*The graders disagreed with each other more than the vendors do, so measure your own reviewers before you set patching policy.*

Opening read

The most useful item in today's evidence is not a model launch but a fight over how a benchmark was scored. Trail of Bits pulled apart 1Password's August report on AI-written security patches and found that the headline 26% clean-fix rate and its own 86% recount both depend on which trials were counted and how the patches were graded, not on the models alone. My read is that neither number should set anyone's patching policy; the size of the disagreement between graders is the finding.

AI patching: the grader moved more than the model

1Password published a report on August 6 in which AI agents patched security vulnerabilities and came away with a 26% clean-fix rate. The reasonable consensus read is that agents fix about one in four security bugs cleanly, so a human stays in front of every patch and the agent is a first-drafter. A smart reader would take that at face value, since the report graded each patch on a full five-category outcome rather than a bare pass or fail. Trail of Bits, a security firm, says this week that the 26% is misleading. Trail of Bits (https://blog.trailofbits.com/2026/09/15/1passwords-ai-patching-benchmark-is-misleading/)

Its complaint has two halves. The first is about which trials got counted, and it favors the agents. The 26% pools trials where agents could run tests against their patches with trials where testing was forbidden, and it counts runs where agents were told to apply the wrong fix. Strip those out and 86% of the remaining patches blocked the supplied exploit, meaning the test that reproduces the vulnerability. The bugs were also picked for difficulty rather than as a representative sample, and the models ran at default reasoning settings, without being told to think longer, instead of their strongest ones. Trail of Bits (https://blog.trailofbits.com/2026/09/15/1passwords-ai-patching-benchmark-is-misleading/) On that reading, the vendor undersold the agent.

I would not swap 26% for 86%, and the second half of the post, the one about how patches were graded, is why. Give the same patch to two grading models and they reach different verdicts on 36.8% of them, more than one in three. Ask a model to judge the patch it wrote itself and a human reviewer lands on the same full verdict a little under two times in three, 65.9% to be exact. The rubric also punished changes the patch was supposed to make; 8% of the ActiveMQ verdicts were docked for that. Trail of Bits (https://blog.trailofbits.com/2026/09/15/1passwords-ai-patching-benchmark-is-misleading/) A third of patches changing grade depending on which model reads them is not a rounding error you fix with a better model. So the number that survives is neither one. What survives is that whoever grades the patch sets the score, which extends a thread this brief has been tracking: the checker is part of the result, and here the checker is another model, and two of them agree neither with each other nor with a human.

For a security lead deciding whether to let an agent draft fixes, this moves the question from "what is the pass rate" to "what is my grader". The test is cheap to run. Pick a handful of already-fixed vulnerabilities in your own code that have an exploit test, let the agent see and run that test, then have one engineer and one model grade each patch on two questions: does it block the exploit, and does it break intended behavior. Count the disagreements between the two graders. Treat that count as a diagnostic, not a verdict. 1Password's 36.8% and 65.9% figures came from grading on all five outcome categories, so a two-question rubric should split less often than that, and any sizable split is the signal that a review gate belongs ahead of any model swap. Close agreement does not by itself make a pass rate trustworthy either; it tells you the grader is not your loudest source of noise, which is the precondition for setting your rate beside Trail of Bits' filtered 86%, remembering that figure came from bugs chosen for difficulty and from trials with the wrong-fix runs removed. Either way, the real policy decision becomes whether you are willing to let an agent execute tests inside your environment at all.

Voice agents: Gemini 3.8 Live Extended Thinking tops one chart and stumbles at the bank

Google DeepMind released Gemini 3.8 Live and 3.8 Live Extended Thinking on September 15, speech-to-speech models that take audio in and hand audio back. Extended Thinking ranks first on Artificial Analysis' Speech to Speech Quality Index at 82.6 and scored 68.6% on tau-Voice, a benchmark of completing tasks by voice, yet on Sierra's banking version of that benchmark it managed 35.1%. Google DeepMind Blog (https://deepmind.google/blog/introducing-gemini-3-8-live-and-3-8-live-extended-thinking/) That gap is the decision for an engineer building a voice agent: the general score sells the model, and the banking score looks more like a regulated call center. The good news is that checking which score describes your work costs almost nothing. Simon Willison has already published a browser tool that talks straight to the model over a live browser connection with no external libraries, so running your own call scripts through it is a weekend, not a procurement cycle. Simon Willison's Weblog (https://simonwillison.net/2026/Sep/15/gemini-live/)

Distillation: the pedigree runs in both directions

Anthropic said last week that Chinese labs ran campaigns to distill Claude, meaning they harvested its answers at scale to train copies, an accusation aimed squarely at the open-model supply chain. Zvi Mowshowitz's read of Anthropic's report adds the detail that matters for buyers: DeepSeek, Moonshot, and Xiaomi sent user queries straight to Claude, and Moonshot at least handed the answers back to its users as if Kimi had written them. Don't Worry About the Vase (https://thezvi.substack.com/p/the-bad-guy-with-an-ai-named-claude) The pedigree question cuts the other way too. Salesforce announced Koa the same day, a reasoning model built on Nvidia's open-weight Nemotron (Nvidia publishes the trained parameters for anyone to download) and tuned afterwards on made-up training examples rather than real customer data for sales and support work, and Interconnects argued in May that Nemotron is itself distilled in large part from Chinese open-weight models. TechCrunch AI (https://techcrunch.com/2026/09/15/salesforce-and-nvidias-new-reasoning-model-is-everything-the-ai-labs-should-fear/) Interconnects (https://www.interconnects.ai/p/the-distillation-panic) The label on the model is weaker evidence than it looks for anyone about to sign a vendor contract: a "Kimi" answer may have been Claude, and a Salesforce model may carry Chinese open-weight lineage through Nemotron, whichever labs that lineage runs to. Ask where the weights came from before you sign a data-handling clause.

Cost controls: Snowflake puts a breaker on every user

Snowflake has made per-user AI quotas generally available, and they cover plain compute, its Cortex functions and agents, and its CoCo assistant; in block mode, once a person crosses a daily or monthly line on one of those governed services, Snowflake automatically pauses their access to that particular service. Snowflake Blog (https://www.snowflake.com/en/blog/per-user-quotas-ai-generally-available/) Databricks shipped hard spend caps in its Unity AI Gateway in July, so two data platforms now enforce hard stops rather than alerts. Databricks Blog (https://www.databricks.com/blog/introducing-ai-spend-controls-unity-ai-gateway) The consensus advice is simply to turn these on, and for human users I agree. My caution is that agent spend concentrates in a few service accounts, and a block on one of those at 2am is an outage, not a saving. A finance or IT leader setting limits should cap people generously and write a separate rule for what an agent account does when it hits its line.

Augur calls

• Call: 1Password publicly responds to Trail of Bits' critique with a revised or clarified methodology for its AI patching report by 2026-10-31.
• Criteria: A post on 1Password's blog or an update to the August 6 report that addresses the pooling of test-allowed and test-prohibited trials or the grader-agreement figures counts as a hit; silence, or a response that does not address methodology, is a miss.
• Due: 2026-10-31
• Odds: 45%

• Call: Google moves Gemini 3.8 Live Extended Thinking from private preview to general availability in Gemini Enterprise by 2026-12-31.
• Criteria: A Google Cloud or Google DeepMind announcement or product page listing Gemini 3.8 Live Extended Thinking as generally available in Gemini Enterprise counts as a hit; any narrower status on that date is a miss.
• Due: 2026-12-31
• Odds: 60%

What to watch

• 1Password's reply to the critique will show whether the 36.8% grader disagreement gets addressed or waved away, and that answer matters more than any revised pass rate. Trail of Bits (https://blog.trailofbits.com/2026/09/15/1passwords-ai-patching-benchmark-is-misleading/)
• Gemini 3.8 Live Extended Thinking's 35.1% on the banking voice benchmark is the score to track in the next update; a jump there means more for regulated deployments than the quality index. Google DeepMind Blog (https://deepmind.google/blog/introducing-gemini-3-8-live-and-3-8-live-extended-thinking/)
• Moonshot has not, in today's evidence, answered the claim that it passed Claude's answers off as Kimi's, and either a denial or continued silence changes the risk for anyone routing to Kimi. Don't Worry About the Vase (https://thezvi.substack.com/p/the-bad-guy-with-an-ai-named-claude)
• The Fields Medallists' declaration warning of a "severe misalignment" between AI companies and mathematicians started with 25 signatures, and its growth is the gauge of whether the fight with OpenAI reaches journal and prize policy. Princeton CITP - Freedom to Tinker (https://blog.citp.princeton.edu/2026/09/15/after-math/)
• Hugging Face's TRL 1.14 syncs only a few-megabyte LoRA adapter (a small add-on set of weights) to the serving engine instead of a 3 GB model, across separate rented jobs with no fast interconnect, a pattern worth testing if you train with reinforcement learning (where a model learns by having its own attempts scored) on borrowed machines. Hugging Face Blog (https://huggingface.co/blog/asyncgrpo-lora-hfjobs)

Verification notes

Trail of Bits' 86% recount and its grader-agreement figures come from its own reanalysis of 1Password's published data, 1Password has not responded in today's evidence, and I cannot check the recount. Trail of Bits (https://blog.trailofbits.com/2026/09/15/1passwords-ai-patching-benchmark-is-misleading/) Gemini 3.8 Live's scores are Google's own reported numbers, and enterprise access to Extended Thinking is private preview as of the announcement. Google DeepMind Blog (https://deepmind.google/blog/introducing-gemini-3-8-live-and-3-8-live-extended-thinking/)

Inspect the evidence behind this dispatch. (https://augurdispatch.ai/evidence/2026-09-16?utm_source=augur&utm_medium=web&utm_campaign=dispatch-2026-09-16)