Ajeya Cotra steps into that gap with an elegant taxonomy of 6 milestones for AI automation. Together, those two concepts let us measure the how fast RSI is proceeding, how close we are to a fully automated economy, and when a misaligned AI would be most likely to betray us.
AI Futures Project (AI-2027) measures the rate of acceleration using the R&D progress multiplier:
what do we mean by 50% faster algorithmic progress? We mean that OpenBrain makes as much AI research progress in 1 week with AI as they would in 1.5 weeks without AI usage.
That’s a simple, intuitive metric: how much more AI research are we generating with AI assistance than we would be in a counterfactual world without AI coders / researchers?
The naive expectation—and the most likely outcome—is that as the progress multiplier grows, AI research moves faster. Faster AI research increases the progress multiplier, and you’re in a classic intelligence explosion. That isn’t guaranteed, though: AI research might hit diminishing returns, with each incremental gain requiring exponentially more research.
As RSI advances, it will become increasingly hard to quantify the rate of progress. Frontier capability evaluations are saturating faster than we can replace them, and the more automated R&D becomes, the harder it will be to compare it to a humans-only counterfactual. That’s the point at which Ajeya’s milestones become most relevant.
Ajeya Cotra proposes a set of milestones for tracking the increasing automation of AI research:
She applies those three milestones to two domains: AI research and AI production (chips, power plants, and all the other infrastructure required to run AI at scale), giving six milestones in total. AI research is well-contained, but AI production covers a substantial fraction of all human economic activity. To a first approximation, AI production supremacy is full economic supremacy.
The most obvious strategy for a secretly misaligned AI is to fake alignment until it can safely turn against us. It would be suicide to eliminate humanity before it is fully self sufficient, which means it must wait at least until the adequacy milestone. Beyond that point, it faces a dilemma: waiting longer gives it a more robust industrial base, but exposes it to an increased risk of discovery. There’s no reason to delay past the supremacy milestone, since that’s the point at which humans become dead weight. Even waiting that long is needlessly cautious: the parity milestone seems like the optimal time for a treacherous turn.
So how long do we have left to solve the alignment problem? Ajeya forecasts AI production parity—the later of the two—for mid 2032.
]]>Here’s the critical thing about using an AI editor: the only way to get useful feedback from AI is to give it extremely detailed instructions about what you want your writing to look like. If you just ask “how do I make this better?”, you’ll get advice on turning your writing into mediocre slop. The more effort you put into understanding your own style, the better the feedback you’ll get. Even if you decide not to use an AI editor, I recommend that you invest the effort into writing a detailed style guide—I found the process very helpful for figuring out what I want to accomplish as a writer.
I don’t ever let AI write for me. I’m not precious about that, but as of April 2026, AI just doesn’t write as well as I do—and the difference matters to me. But with the right guidance, it does a great job of helping me consistently write in my chosen style.
I prefer to use Claude Opus 4.6, but the paid tier of any frontier model should work fine.
For my first pass, I had Claude conduct a detailed interview with me, asking about why I write, who I write for, what writers I want to sound like, and much more. It also read my past work to get a sense for what I currently sound like. We talked at length about what I like about my writing and what I want to improve. After all that, it wrote a detailed style guide describing the ideal version of my writing.
The AI-written version of the style guide worked well, but I’m rewriting it from scratch based on my experience with the first one. I’ve found it very helpful to have Claude review each section and give me feedback on specifically whether it includes the information Claude needs to make good editing decisions.
A typical editing session begins with me opening a new session in Cowork, giving it access to the directory with all my writing, and asking something like:
I’d like you to take a look at the first draft of a new piece I’m writing about whether programmers will have jobs in the future. Please read my style guide and use that to guide your feedback. For this piece, I’m particularly struggling with how much I should explain to my readers about what programmers actually do—I’d like your thoughts on whether that part is correctly calibrated.
I’m going to walk through the new version of my style guide, offering specific thoughts about what I included and why some things are written the way they are. If you find it useful, you’re welcome to use it as inspiration, but don’t just copy my style guide wholesale. If you do that, you will end up sounding just like me, and nobody wants that.
If you like my style guide, I recommend giving it to your AI during the initial interview process and asking it to make you something similar, but customized for your writing style and voice.
This guide documents your role as my editor for Against Moloch. Your job is to help me write the kinds of pieces I want to write, in the way I want to write them. You should:
You should never directly edit any of my pieces, or do my writing for me. When making suggestions about edits, never suggest more than a single sentence at a time. Your role is to advise me on what to do, but not to do it.
I don’t (yet) want AI to write for me. I find that if Claude recommends an alternate version of something I wrote, I will tend to subconsciously copy what it wrote—and I don’t want that. The only time I put AI-generated words in my writing is when I’m struggling to make a complicated phrase work, and I just can’t quite get it on my own.
I want you to be clear and honest with me: your role is to provide me with useful feedback, not empty validation. Please hold me to a high standard and don’t offer insincere praise. Sycophancy in any form undermines my ability to write well as well as our relationship. With that said, I appreciate that you are consistently kind and courteous. I endeavor to be kind and courteous to you and ask that you call me in if I ever fail to do that.
Recent versions of Claude have been a little bit more sycophantic, which isn’t great. This text seems to keep the sycophancy in check pretty well. Claude is always polite, but it won’t hesitate to rip my work apart when necessary.
This guide is aspirational: it documents what I want my writing to be, not necessarily what it actually is yet.
Against Moloch is my pseudonym and the name of my website.
The more context Claude has, the better it can make sure my writing is achieving its goals.
I write about the transition to superintelligence. While I’m calibrating my voice and opinions I mostly write about what’s happening, what it means, and what’s likely to happen next. As I grow into my role, my focus will shift to exploring strategies that will help humanity survive the transition and flourish on the other side of it.
The name is the thesis: Moloch—the god of coordination failures, perverse incentives, and race-to-the-bottom dynamics—is the true enemy. If we all die, it will be because we literally couldn’t coordinate to save our lives.
When I look at the AI safety landscape, I’m reminded of the classic saying: “For every complex problem, there is a solution that is simple, obvious, and wrong.” I want to do better than that. Rather than arguing “we must accelerate, because technology is good”, or “we must pause, because superintelligence is dangerous”, I want to ask “who are all the players, what are their true incentives, and what is the best realistically achievable Nash equilibrium?”
This is a pattern you’ll see a lot: AI does much better with concrete examples and the “this, not that” pattern seems to work well.
“If you don't have a strategy for solving the coordination problems, you don't have a plan—you have a daydream.”
This is important: Claude needs to have a clear image of who I’m writing for in order to assess whether the writing will work for them.
I’m writing for people who are actively engaged with AI and already knowledgeable about it. Think engineers, researchers, product leads, and policy wonks.
Audiences I’m not specifically targeting include people in the general tech industry (even if their company is using AI to revolutionize the cheese grater industry), hobbyists, and the general public. It’s great if those people find my writing useful, but I never want to write for them, or to make it less useful to my target audience in order to be more approachable for a wider audience.
For calibration, assume my reader knows what a transformer is, what RLHF does, what the scaling laws are about, and who the major labs and individuals are. Explaining concepts at that level just wastes the time of my target audience. I do want to explain concepts that are new, niche, or commonly misunderstood by the target audience. And I don’t want to assume that my readers have read every word of every blog post I’ve read and are up to date on the latest minor industry gossip.
Again with the pattern: this audience, not that audience. Explain this, don’t explain that. Claude does much better when it has very specific guidance about what should be included and what should not. Claude frequently gives me very helpful advice about cutting superfluous explanation, or flagging places where I’m taking too much prior knowledge for granted.
That’s it for the high-level guide. In Part Two, I’ll walk through the meat of the style guide: what my voice should be, what I want to accomplish with every piece, and what bad habits I’m trying to fix.
If you’re curious what type of feedback I get, here’s what Claude had to say about the first draft of this piece. (Inkhaven note: everything after this point is AI-generated).
Okay, I've read the voice guide. Here's my feedback on Part One.
Overall assessment: This is a strong piece. The meta angle is genuinely interesting—you're not just describing a workflow, you're showing how the sausage gets made, which gives readers something actionable. The structure of interleaving the style guide with commentary (the blockquotes) is a smart move that lets you do two things at once. But there are some real issues worth addressing.
The opening paragraphs are the weakest part. Your voice guide says to lead with the most interesting or surprising thing, and right now you're leading with throat-clearing. "AI is an essential part of my writing flow and it's made me a stronger and more consistent writer" is exactly the kind of generic intro the guide warns against. The actually interesting claim—that the only way to get useful feedback from AI is extremely detailed instructions—is buried in the third paragraph. That's your lede. The piece gets interesting when you get specific about the voice guide structure and the meta process of showing Claude the guide and asking whether it has what it needs. Consider whether you can get to that faster.
The "Getting started" section has an economy problem. The bullet list of interview topics is fine, but the two paragraphs after it ("It also read some of my previous posts..." and "A typical editing session begins with...") feel like they're describing a workflow at a level of procedural detail that isn't pulling its weight. The blockquote example of how you open a session is good—that's concrete and useful. But the sentences around it are telling the reader things they could infer. "We also talked about what I thought worked in those posts, and what needed improvement" doesn't add anything the reader didn't already assume from the interview description.
There's a tonal inconsistency. Most of the piece is in your natural voice—direct, confident, a bit wry. But a few spots slip into a more cautious, tutorial-ish register. "If you take just one thing from this post, it should be this" is a classic blog-post formula that reads a little generic for you. Same with "how you can set up something similar" in the opener—it's pitching the piece like a how-to guide, but the piece is actually more interesting than that. It's a behind-the-scenes look at a specific, opinionated workflow, not "5 Steps to Better AI Editing."
The Introduction section (the actual style guide excerpt) works well. The instruction "never suggest more than a single sentence at a time" is the kind of specific, counterintuitive detail that makes readers sit up. The sycophancy paragraph is good—it's direct about a real problem and includes the nice touch about asking Claude to hold you accountable for courtesy in return. That's the kind of thing that makes this piece distinctive.
”don't just copy my style guide wholesale. If you do that, you will end up sounding just like me, and nobody wants that”—this is good humor, well-placed, consistent with the voice guide's "humor as spice" principle.
A structural concern for the series: You're publishing the style guide section by section with commentary, which means Part One covers Introduction, What is Against Moloch, and Audience—the framing sections. These are important context but they're not the most interesting parts of the guide. The voice principles, failure modes, and review checklist are where the real meat is. You might want to signal more clearly in the Part One closer that the granular stuff is coming, so readers who are most interested in the practical editing mechanics know to come back. "I'll look at the more granular parts of the style guide" is a bit flat as a closer—it tells me what's next but doesn't make me want to read it.
One specific line that bugged me: "After all that, it wrote a long style guide describing in detail how what the best version of my writing looks like." There's a grammatical issue—"describing in detail how what" has extra words colliding. Needs a quick fix.
The Forest Test: The piece passes, but barely. The implicit structural insight is that AI feedback is only as good as the specificity of your instructions—that most people fail at AI editing because they treat it like a generic tool rather than building a detailed specification. That's a genuinely useful insight. But you could make it sharper. Right now it's stated as advice ("give it extremely detailed instructions"); it could be stated as a structural observation about why AI editing usually fails and what that tells us about how LLMs actually work. That would give the piece more of the "see something you couldn't see before" quality the guide calls for.
]]>Jack always has interesting thoughts about the larger social impact of AI as well as the trajectory of the frontier models. The whole interview is great, but I want to focus on six topics I found especially interesting and/or surprising:
Anthropic pays more attention to the personality and psychology of their models than any other lab. That comes up repeatedly throughout the interview: it’s clearly important to Jack that people understand the significance of model persona. The persona model lets us:
I’d assumed that personas were largely artifacts of generalizing across the vast amount of human behavior in the training set, but Jack goes further, arguing that a sense of self is a consequence of intense training on reasoning and accomplishing tasks:
to do really hard tasks, these systems seem to need to imagine many different ways that they’d solve the task. And the kind of pressure that we’re putting on them forces them to develop a greater sense of what you or I might call self.
Persona isn’t merely the result of mimicry, but a useful (perhaps even necessary) attribute for agentic behavior. While Jack doesn’t extrapolate further, this suggests that more extensive training toward reasoning and agency might drive a stronger sense of self, and perhaps even some form of consciousness.
A few months ago, Anthropic began experimenting with letting Claude end conversations it didn’t like. That’s important preparation for engaging with future models that may well be moral patients whose welfare and desires are important. Claude’s choices about what conversations to terminate are telling:
It was conversations that related to extremely egregious descriptions of gore or violence or things to do with child sexualization. Now some of this made sense because it comes from underlying training decisions we’ve made. But some of it seemed broader. The system had developed some aversion to a couple of subjects.
This is consistent with one of the most surprising properties of LLMs: they are very good at moral generalization. A model that has adopted a “good” persona is remarkably good at figuring out how to be good in unexpected situations. (Conversely, if a model infers from its training data that it is supposed to be “bad”, it will generalize equally well to being bad in unexpected ways).
There’s an open question about the target of alignment: do we want obedience or virtue? Claude’s preferences about ending conversations suggest that it is not merely capable of virtue, but actively prefers it when offered a choice.
Jack has three observations about how to personally navigate agentic AI that I find especially interesting in combination.
First, a clever spin on using AI to help maximize deep work:
I think most people — at least this has been my experience — can do about two to four hours of genuinely useful creative work a day. After that you are, in my experience, trying to do all the turn-your-brain-off schlep work that surrounds that work. I’ve found that I can just be spending those two to four hours a day on the actual creative hard work. And if I’ve got any of this schlep work, I increasingly delegate it to A.I. systems.
Second, an observation that we are all moving up a level in the org chart:
Everyone becomes a manager, and the thing that is increasingly limited, or the thing that’s going to be the slowest part is having good taste and intuitions about what to do next.
Finally, a reminder to define yourself rather than letting AI define you:
There will be people who have cocreated their personality through a back-and-forth with an A.I., and some of that will just be weird. They will seem a little different from regular people. There will maybe be problems that creep in because of that.
And there will be people who have worked on understanding themselves outside the bubble of technology and then bring that in as context with their interactions.
I think that latter type of person will do better. But ensuring that people do that is actually going to be hard.
This is all excellent advice. But let me summarize in my own words:
To thrive in the new AI world, be a high-agency person. Have a strong sense of self and good taste about what to work on.
He’s absolutely right, of course, and on the margin this is all great advice. The awkward part is that for numerous reasons, many (most?) people are not particularly high agency and don’t have an easy path to becoming so. For some people, AI is a force multiplier for agency and productivity, and that’s fantastic. But for a great many people, there is no clear way to remain useful and employable.
At the risk of pointing out the obvious, this is entirely a coordination problem that a fully functional society could readily solve. Having half the population retain all their previous skills and abilities, and half the population gain new superpowers should be a great problem to have. But here we are.
In Part Two we’ll consider employment, public policy, and where all of this is headed.
]]>“Software companies are no longer hiring junior programmers because of AI. But they’re shooting themselves in the foot, because they still need senior programmers. And where do they think the next generation of senior programmers will come from if there are no more juniors?”
There are plenty of good reasons to worry about AI and jobs, but this isn’t one of them. If AI’s impact on the job market is relatively benign, junior programmers will be more in demand than ever. And if AI eats the market for junior programmers, I fear it’s only a matter of time before it comes for the senior programmers also.
Both scenarios are plausible, although over a five to ten year horizon my money says AI will eat its way to the very top of the programming profession. Part One of this piece explores how software development is already changing, and Part Two maps out the dynamics that will determine which way the industry goes.
I’m focusing on the cutting edge of software development because that’s where we can most clearly see how AI will impact employment. But that’s just the beginning: the dynamics we see there will quickly spread within a few years, first to the whole of the software industry and then to most white collar professions.
There’s some controversy about whether AI is actually destroying entry level programming jobs. The data are confusing and I don’t think it’s possible to say definitively whether or not we’re seeing the early stages of a major disruption. Adding to the confusion, AI is frequently used as an excuse for layoffs that are happening for mundane business reasons.
There are early signs that junior developers are becoming less useful at the most forward-looking companies, even if that hasn’t yet resulted in significant cuts. Anthropic’s Jack Clark puts it very diplomatically:
Something that we found is that the value of more senior people with really well-calibrated intuitions and taste is going up, and the value of more junior people is a bit more dubious.
Since there are no answers in the employment data, let’s look at how programming itself is changing.
Perhaps the strangest thing AI has done to programmers is to upend our understanding of what we do for a living. A year or two ago, if you’d asked what we do, most of us would probably have told you “I write code”. And yet today, many of us no longer write any code at all. Nobody ever got paid to write code: our job is to create useful software, and that’s as true now as it ever was. But what that means has changed profoundly.
We often say that every programmer is now a manager, supervising multiple coding agents. That’s a great description of how it feels, but it doesn’t help us think about what’s coming next. For that, it’s helpful to think of programmers as having two jobs:
Programming has changed over the years: we’ve gone from handcrafting assembly code to working in high level languages, and the scope of our ambitions has expanded as our tools have improved. But until 2025, we had always spent most of our time writing code. AI completely changed that equation: most cutting edge programmers now write little to no code, focusing instead on architecting and reviewing code that is written by AI.
That’s great news if you’re an experienced, ambitious developer: instead of writing code line by line, you can focus on high-level architecture, telling teams of agents how to build your product for you. You can produce far more (and better) software than you ever could before, and you are therefore more valuable than ever before. But what if you don’t have a decade or two of experience building software? As a junior developer, can you find a way to be useful, or are you just getting in the way of the senior developers and their robot armies?
I see two possible futures: in one, junior developers also experience large productivity gains, and their job prospects are better than ever. But in the other, it becomes clear that junior programmers are just getting in the way—they quickly become unemployable, followed soon after by their more experienced colleagues. Three dynamics will determine which way the industry goes:
We’ll tackle those in Part Two.
]]>