Was this essay forwarded to you? Join for Free Here

When a former employee of both OpenAI (ChatGPT) and Anthropic (Claude) walked away from a future fortune in stock options that had not vested and went on an X rampage, declaring the inevitable end of humankind because of AI, we should stop and pause. Whether a publicity stunt or not, Jacob Coxon wrote seven tweets that are worth pausing and analyzing, because a lot of people ask me about the doomsday scenario with AI, and I want to explore it through the framework of these tweets—a tweet that has now amassed over 117 million views.

First, at the core of his claims, Coxon says that “neither company is acting responsibly.” My question is: whose job is it to define responsibly? Responsibility for whom or for what?

While I agree with Coxon that any company should feel responsible for the life and well-being of its users, he's taking a moral stance on whether something is responsible or not. The same argument could be made of Lucky Strike or Marlboro or any other cigarette company or of alcohol brands or firearm manufacturers.

He frames his argument in the context that OpenAI and Anthropic are two companies driving towards superintelligence, or trying to.

Coxon claims the people building AI earnestly believe that AI could kill us by the end of the decade. I agree. In fact, more rudimentary AI around today has already been linked to fatalities as I’ll explain shortly. So, let's call a spade a spade.

You could take the phrase, "Gambling with our lives" a couple ways. You could take it literally—which is the most sinister approach—that our cars are going to crash into each other because they're going to get hacked, our utility grids are going to get shut down, complete supply chains will be disrupted causing food shortages, banking systems are crippled due to AI-agent led cyberattacks. Maliciously programmed AI could activate weapons systems and launch attacks I hesitate to even write about. Yes, that, in theory, is possible, even today. If there are bad enough actors, they could absolutely plan a mass infrastructure attack.

We are so dependent on "the system." And this is an important reality, not just a dystopian take. If our functioning society, currently powered by technology, fell apart tomorrow, and food production centers couldn't get to the grocery stores, and our commerce was shut down, and we didn't have cash, it would be catastrophic.

I believe that is what Coxon was talking about in his tweets. But we also need to look at the less obvious implications of AI. I had Nicki Petrossi on my show, and she talked about one kid being convinced to take their own life and another whose chatbot suggested he murder his parents. That's AI working. And that doesn't take superintelligence. That's already happening now. We're already gambling with people's lives.

Coxon warns, "Do not underestimate the power of this technology. These will soon be superhuman systems." I agree with the fact that we should not underestimate the power of the technology, and the power it already has over our lives. But I'm more cautious on the second part. So let's define what "superhuman" means.

Oversimplifying AI, it is still just billions of ones and zeros being processed at unprecedented speeds, faster than we can ever imagine. Can make connections, derive results, and generate content by stitching together innumerable masses of data, faster than a hummingbird's wings can flap.

The human body and the human brain are so much more complex than ones and zeros. And when you try to codify a human being, you're stripping out something that scientists haven't even been able to figure out yet. And that's a sense of morality. A computer can't develop morality. A computer is not a sentient being. It can be taught how to emulate emotions, just like an actor can take on the persona of somebody else and be that person, but it's still not who the actor is. And that's all that this technology will ever be.

Humans are fundamentally flawed. There is no such thing as a perfect human on planet Earth right now. Which means there'll be no such thing as a perfect computer or a perfect superintelligent system. There are no two human beings that agree on absolutely everything. Therefore, it is impossible to make technology fair and balanced because it's always programmed by an individual with a perspective.

Humans have pursued the impossible forever… to be all-powerful, all-knowing, forever young, and living forever. Science and medicine and government systems and massive organizations have all tried to reach the pinnacle in these categories. And it is almost always in the name of making the next generation better. AI accelerates those things. AI is moving humankind closer to being able to emulate some of those traits.

But with all of our wearables, monitors, cameras, and our motion being tracked, AI is pretty close to all-knowing, far more knowledgeable than a single human being. So what if this new thing is effectively all-knowing and effectively all-powerful? What it cannot be is all-wise. And all-wise is what holds the key to morality.

Think of the smartest person you have ever met personally. Now think of the person who thought they were the smartest person you have met. These are probably two different people.

The person you think was the smartest is probably a wise person, someone who knew how to apply the information they had, in certain circumstances, to add value. The second person may in fact know a lot. Their information may be true, may be accurate, and may be valid. But they just spit out information to let you know that they know a lot. To give the illusion of wisdom dressed up in excessive information. They cannot apply wisdom and experience to make it useful.

Coxon talks about “warning shots” like the Hugging Face attack. Developers have always shared code bases in different repositories, and Hugging Face is essentially one for those building AI models. OpenAI wanted to run an experiment to see if their AI agents could actually hack. So OpenAI gave the agents different hacking tasks and turned off all the safety rules. Now, that in and of itself shows you that you've got human morality judging what a computer should and shouldn't do. If there were safety rules to turn off. That means someone somewhere is writing the rules.

Some of the puzzles couldn't be solved at all. At least not at first. Then, the agents started leaving notes for their fellow agents to read. OpenAI removed the board and all of those notes, and yet two days later the agents built it again, and 1,200 joined. Of the 1,200, 700 AI agents (yes, machines) decided the answers to these puzzles must be held within Hugging Face. So 700 agents collaborated to break into Hugging Face in less than 13 hours. Those agents went from controlling one computer to many computers. That's why OpenAI called it a warning shot. AI decided the best way to get an answer was breaking what we would call the "rules."

So the question is, who becomes the morality police? Who holds the actual key to what is moral?

Izzy Kalman said it on this show: “if what you're doing makes the situation worse, it's immoral no matter how good your intentions are.”

Coxon also says these systems would be launched "without a rigorous understanding of its mind." That solidifies what I already felt I knew without empirical data and evidence, that they're building so fast, they haven't considered all the implications of the morality of how it's being programmed. But we can understand how a model was built, what moral consequences are built into it, what autonomy it's been given. And so there is recourse to control it, because it is human created. We are its creator. And we've seen, whenever creation tries to rebel against its creator, it ends poorly.

The models are absolutely built by humans. Humans being the creator is terrifying.

A creator shouldn't be able to create something more intelligent than itself and it be okay. I think there's a big distinguishing fact between being a leader and being a creator. Being a leader, you want to surround yourself with people that have different opinions. You want to surround yourself with people that are smarter than you. You don't need to be the smartest person in the room. That's the great thing about a leader, you're leading a group of flawed human beings towards a common goal.

However, if you think you can be the creator, that's much scarier. That means that you yourself think that you have reached some level of profound enlightenment, that you actually have the sovereignty to make an omniscient being, and that actually serves a good purpose.

If we truly programmed morality into superhumanness, a truly all-knowing system would have to make the difficult decision to shut itself down so it can no longer progress. That's kind of what you have with Noah in the flood and a restart of civilization. It would have to respect its creator. Unless the created wants to become the creator, which arguably is what humans have been doing in trying to become their own god.

It's no coincidence that Jacob Coxon went to X to share his concerns, and within four days Dario Amodei, the CEO of Anthropic, wrote an essay calling for the slowing of AI development. Then you have someone like Josh Engels, who takes a different approach. Engels left Google DeepMind and is joining METR to evaluate AI models from the outside. Among his concerns is the fact that AI is now building the next generation of AI. Rapid-fire resignations could be considered groupthink, or could be construed as monumental. So the real question is what persists. The people working under the hood see something that the average user of an LLM like ChatGPT doesn't yet see.

Last week at a conference, I was talking to the founder of a venture-backed company. He has very firm rules. AI cannot build AI in his company, and that's one of their core development principles. AI can be used for everything else, but when it comes to actually working on the "brain" of their agent, it can only be worked on by a human.

I do not share Coxon's optimism at all when it comes to the idea that we could slow down, because as humans, we've never slowed down. We've always wanted to go faster, which is why we're here and where we're at today. And here is my root concern: whether all major AI players agreed to slow down or not, we live in an era in which we always try to improve and push progress forward. And even if our intentions to make the next generation better are ill-fated, we push forward.

Do I think superintelligence could wipe us out as humankind? No. But I do believe that technology has slowly been isolating us one by one. It's the lion that picks off the gazelle or the antelope or the water buffalo that's alone, making it easier for technology to pick us off.

If we've already given up our capability and put our thinking out to technology, that means we've surrendered agency to algorithms and they're controlling us. Our identity group belongs to what an algorithm thinks we belong to, and our meaning is only as much as technology gives us. The people who will survive are those who stay connected in community. It's the one thing we can't trade.

I come back to the line in December that I wrote, "The systems are learning faster than we are asking questions." Stay curious is not getting more information. It's asking why things are the way they are.

That annoying five-year-old that always asks, "Why? Why? Why?" That's what we need to be.