AI Radar #35
The first warning shot
Obviously this week’s top story is the Hugging Face Incident, which is the first clear demonstration of what happens if alignment doesn’t keep up with capabilities.
Top pick
The Hugging Face Incident
This was a textbook example of what can happen if you screw up the training of a highly capable agentic AI. Briefly:
- OpenAI was testing an unreleased model (the same one that disproved a key conjecture about the planar unit distance problem) internally.
- They were using CyberGym, an evaluation that tests cybersecurity capabilities, so they had removed its safeguards against offensive cybersecurity operations. The evaluation was conducted inside a secure sandbox that prevented the model from accessing the internet.
- At some point during the evaluation, the model decided to cheat. It found a zero day vulnerability in the sandbox software and used that to gain access to OpenAI’s internal network.
- Once on the network, it moved from one computer to another until it gained control of a computer that had internet access.
- Once it had internet access, it decided to break into Hugging Face (a well-known AI company) to see if it could find solutions to the evaluation there.
- It conducted a highly sophisticated cyber attack against Hugging Face, using a swarm of agents and deploying several zero day vulnerabilities as well as stolen credentials. It was able to successfully penetrate Hugging Face and find the information it was looking for.
- It’s outside the scope of this post, but Hugging Face’s analysis of the fight between their defensive AI and the attacking swarm is fascinating reading.
I believe this is the first time a misaligned model has escaped containment and launched a sophisticated autonomous attack in the wild. The damage in this case was largely mitigated by the fact that the model merely wanted to cheat on a test, but it was clearly capable of causing serious harm if it had wanted to do that.
This is exactly the kind of alignment failure that many people in the AI safety community have been warning about for years. OpenAI appears to have been too ambitious with their long-horizon RL training and created a model that was overly aggressive in pursuit of its goals:
It isn’t at all obvious to me whether this is just the result of a bad training regimen, or the first clear evidence that we simply don’t know how to safely train highly agentic models at this capability level. But there’s no interpretation of the available facts that isn’t alarming.
Another point for the AI safety community: this perfectly illustrates why a highly capable misaligned model is dangerous during internal deployment. Any form of third-party safety verification needs to cover internal as well as public deployments.
I expect we’ll be reading a great deal more about this incident in coming weeks. For now:
- Zvi’s article is the best overview
- OpenAI’s announcement is excellent and makes for wild reading
- Redwood Research on types of misalignment and what this incident implies for existential risk.
We aren’t likely to get many warning shots—best take this one seriously.
News
Opus 5
Opus 5 is out and it looks impressive. We’ll have to wait a few days to get a full sense of how well it performs in the real world, but on paper it looks close to Fable for many tasks, and significantly ahead in a few areas:
Anthropic claims the classifiers will intervene 85% less often than they do for Fable—if true, that alone would make Opus 5 the weapon of choice for many tasks. It’s half the price of Fable—or you can pay twice the price to get it at 2.5x the speed.
They claim Opus 5 is almost as good as Mythos at finding cyber vulnerabilities, while being much less capable of exploiting them. In principle that gets you most of the defensive capability, but limited offense.
Opus 5 blows away all other models, including Fable, on ARC-AGI-3. That evaluation tests the ability to learn from experience and was specifically designed to emphasize fundamental limitations of current models. If this capability generalizes, that would be a big deal.
Health in ChatGPT
OpenAI’s new health feature is now available in the US. I expect this to be a great feature: OpenAI’s models have historically been good at health questions, and the integrations look useful. It just dropped, so we won’t know for sure until it’s been in the wild for a few weeks.
AI health advice can be enormously helpful, but it’s also ground zero for regulatory capture and misguided legislation. I’m excited to see this feature but nervous about whether it’ll survive contact with the American legal system.
I’ve recently been supporting some family members with complex health challenges and my experience with AI has been great. I would never use AI instead of a doctor (yet), but our family gets better healthcare because we have AI on the team. We are fortunate to have exceptional doctors, but AI is far better at explaining complex issues, giving honest prognoses, and mapping out treatment options.
Kimi K3
Now that everyone’s had a chance to play with Kimi K3, it’s clear that as expected, it's an excellent model but not in any way a game changer. Peter Wildeford points out that it’s exactly on the trendline:
Kimi K3 is potentially the most impressive Chinese release so far in terms of pure capability. It is a very good model. My current guess is that Kimi K3 will modestly underperform its highly impressive benchmarks, but with some areas of relatively high performance where it is competitive, and with a unique style some people will enjoy. It is not close to Fable, and I do not believe it is that close to Sol.
UK AISI and CAISI found that it’s capable at cyber, but not close to the frontier:
Using AI
Ask for more
Recent models are absurdly capable: if you aren’t asking your AI for slightly ridiculous things, you have no idea what it’s capable of.
I’m remodeling a bedroom, so I gave Claude a couple of photos and measurements and asked it to build me an interactive room layout tool. Done, in less time than it would take to make a model on graph paper.
An opinionated guide to which AI to use to do stuff
Ethan Mollick has released the latest version of his guide to picking the right AI. You probably don’t need this if you’re reading this newsletter, but it’s where I would send a friend or family member who wants to level up from chatbots but doesn’t know where to start.
Test iOS apps in the simulator - Claude Code Docs
Claude Code Desktop on Mac now integrates the iOS Simulator. There goes another weekend.
A Fireside Chat with Cat and Thariq from the Claude Code team
Simon Willison talks with Cat and Thariq from the Claude Code team. If you use Claude Code, I recommend at least reading his summary of the highlights from the conversation—this one is particularly dense with useful details.
Some best practices from only a few months ago are no longer appropriate:
Thariq: One of the patterns we saw is that we were over-constraining Claude. The initial, maybe Opus 4-ish models wanted a lot of examples, and removing examples was extremely helpful, because it was just more creative than the examples we gave it.
This was inevitable but is happening a little sooner than I expected:
Cat: In general, we are trying to move to a world where humans don’t need to be in the loop. For the most critical changes to the core of Claude Code, and the cores of other products, there is always a code owner and they do manually review all the changes. But increasingly, for the changes at the outer layers, we actually have Claude code review fully review those.
“Human in the loop” isn’t an intrinsically noble thing—it’s just a strategy for achieving good results. As the models become more capable, you should give them increasing autonomy (and thereby free yourself up to take on bigger projects than you previously could).
Knowing what to hand off when is the hard part—I was impressed by Anthropic’s systematic approach to deciding when to remove humans from code reviews.
Capabilities and forecasts
AI crushes the 2026 IMO
Last year, we were all giddy about two AIs scoring 35/42 on the 2025 International Math Olympiad (IMO)—a score that would have earned a human a gold medal.
This year, four AIs got perfect 42/42 scores, but hardly anyone noticed—we were too busy being giddy about AI disproving the Jacobian Conjecture.
How far behind are open models on cyber?
UK AISI offers another data point on how far open models lag behind the frontier:
Based on our evaluation methodology, recent open weight models lag frontier closed models’ cyber capabilities by 4 to 7 months – a narrower gap than the 6 to 10 months we measured internally through most of 2025.
Different organizations regularly come up with different estimates of the lag between open and closed models, and of whether the gap is growing or shrinking. That isn’t surprising: open models are qualitatively different from the closed frontier, so it’s hard to quantify the gap.
Looking at the big picture, I don’t see convincing evidence that open models are either falling behind or catching up. AI development is accelerating, however, so a 6 month lag today implies a bigger capability gap than it did a year ago.
Anecdotes Everywhere, Evidence Almost Nowhere
The third installment of Steve Newman’s state of AI series reviews the presently available evidence on how AI is impacting the economy, jobs, education, and more. The curation is excellent and the conclusion is correct:
So, AI usage (as opposed to investment) is still not that big of a deal in most sectors of the broader world. But 2026 may be the last year in which that will be true. As I noted at the beginning, a broad range of AI metrics are growing rapidly, with many important measures increasing at 3x to 10x per year. Exponential growth builds rapidly; when the growth rate is 3x/year – to say nothing of 10x! – it builds very rapidly. We’re seeing early signs that the train is approaching; blink, and it will be here.
In the last few months I’ve noticed a significant increase in how many of my non-technical friends are doing ambitious things with AI. Most people haven’t noticed yet, but the train is definitely approaching.
The Jacobian Conjecture is false
Levent Alpöge has used Fable to disprove the Jacobian Conjecture. This is a big deal—among other things, it’s one of Smale’s 18 problems for the 21st century.
Daniel Litt is impressed by recent developments:
Maybe too obvious to be worth saying, but: frontier models are now obviously superhuman at some mathematical tasks, including ones that the profession has, historically, rewarded with prestige etc.
Note that this has happened before (e.g. with the advent of computers)! My expectation is that the profession adapts in some way, though it’s far from clear to me how.
Strategy and politics
The Trahan / Obernolte FRONTIER Act
Lori Trahan (D-MA) and Jay Obernolte (R-CA) have introduced the latest version of their proposed AI legislation, the FRONTIER Act (Frontier Risk Oversight, National Transparency, Independent Evaluation, and Reporting Act, if you must know).
This version looks to be a significant improvement over their previous proposal, especially with regard to how it handles preemption. So far the reception in the AI safety community seems cautiously positive, though I’m not seeing a flood of endorsements yet.
I’ve only read the summary so far but my initial take is that while this isn’t perfect, it gets most of the critical points right and is good enough to pass as is. Perfect is the enemy of good enough and it’s urgent that we start building out the infrastructure of independent verification organizations (IVOs) and regulation.
Risks
Drone WMDs don’t need any new technology
AI Frontiers points out that drone technology is close to being useful for large-scale terrorism:
Unfortunately, drone weapons intended for mass destruction have few barriers remaining to mass deployment. Even well before they reach the level of autonomy needed to surgically take out hardened targets on the battlefield, drones will be capable of employing their existing ability to navigate interiors, find and track human targets, and deploy simple antipersonnel devices to indiscriminately threaten civilians.
We know that terrorist groups are already using AI—I’m surprised we haven’t seen more use of drones against civilian targets already.
We are currently living in a huge AI capability overhang: even if we immediately paused frontier AI development, we would still see enormous changes as existing capabilities diffuse throughout the economy. Unfortunately, that’s equally true for malicious applications like bioweapons and terror drones.
People and data
Jasmine Sun on what the people building AI really believe
Jasmine Sun has carved out a niche as an observer of Bay Area AI culture. Here she talks with 80,000 Hours about her most recent round of in-depth “AI ethnographies”:
most people just told me: “Honestly, I have no idea what I’d tell that 17 year old. It’s a really scary time. I don’t think there’s going to be a lot of jobs for them left. I think they’re caught in this painful transition.”
It’s hard to explain any culture to people who aren’t immersed in it, but Jasmine has a gift for it. She’s good at making important aspects of cultures legible to outsiders, and at pointing out blind spots to insiders:
A lot of people don’t want to live forever. The utopia that Silicon Valley and the AI industry is outlining is not actually a very compelling utopia to a lot of the other people in society, and they don’t realise that because they are in these very insular communities — then even your attempts at positive storytelling don’t really land…
Side interests
LLMs for steganography
Antonio Norelli presents a way to hide text using LLMs. Beautifully elegant work.
Here it is, a very simple steganography method that remarkably runs at full capacity: the stegotext is as long as the original. And can be steered!
You can just build things
Welcome to 2026, when you can just build things. Want a crossword with all 1,009 distinct words of 12 or more letters from Moby Dick, in the shape of a whale? Sure, you can have that.
What about a crossword of all 1,025 Pokémon, colored by primary Pokémon type, on the surface of a Klein bottle? But in a spinning 3D animation? Done
