Amid growing concern over AI risks, top tech executives were summoned to the White House this week and agreed to President Donald Trump’s preferred alternative to federal regulation. The accord, which carries no legal weight, commits Anthropic, OpenAI, Nvidia, Meta, SpacexAI and Google to “robust internal processes” and independent audits, and comes soon after safety worries delayed the release of the latest ChatGPT model.
At Google, James Manyika’s role as senior vice president of technology and society — and as co-director of the newly established DeepMind Institute — is to help ensure that AI turns out well. Here, he discusses the need to make the most of the technology’s possibilities without losing public trust.
This conversation has been edited for length and clarity and took place before AI industry leaders met with Trump on Sept. 29. You can listen to an extended version on The Mishal Husain Show podcast.
This is a moment when there’s more talk and more fear about the risks and capabilities of AI. Is it advancing too fast to be safe?
It’s quite remarkable — the incredible progress that AI is making in the last few years. I could never have imagined this when I was doing my Ph.D. many years ago.
Any time you have a powerful technology that presents the possibility of all these extraordinary things, it’s also worth taking seriously the risks and complexities.
Yes, some of the concerns and anxieties are quite well-founded. It’s worth taking those seriously, along with the possibilities we’re all excited about.
Even a 0.1% AI Risk Matters
Google’s James Manyika
The risks don’t feel theoretical anymore, with different companies’ models exploiting vulnerabilities. Do they illustrate the dangers are real?
The dangers have always been real. There’s a very important need to make sure we are keeping two things in tandem: Capability development should not outrun our ability to manage risks and complexities.
Let me put to you directly what Anthropic CEO Dario Amodei is saying: that we need to “pace the frontier,” [that] there should be an industry-wide slowdown. What do you think of that?
I understand where he’s coming from.
If some players feel the capabilities are advancing far in advance of safety and responsibility, they should do what they need to. 1
1 A cagey answer, given the significance of Amodei, OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk all making similar statements within a short period. Manyika’s Google colleague, DeepMind co-founder Demis Hassabis, responded to Amodei’s proposal by saying, “the direction is correct.” But any coordination among AI companies raises questions about anti-competitive behavior.
You’re suggesting, I think, that Google is not in the same place as Anthropic or OpenAI: that you are doing something more responsible?
No, they can speak for themselves. In our minds, we’ve always been trying to pace those two things together.
Back in 2022, when the world started paying attention to this, we probably had some of the most capable, powerful models at the time. People thought, Google’s being too slow. Why didn’t they release those products? Why didn’t they release those models? We’ve always had this internal ongoing discussion amongst ourselves about when the technology is ready. 2
2 The November 2022 public release of ChatGPT posed the first serious threat to Google’s dominance of search in years, prompting accusations that it was lagging in the AI race — despite having developed key aspects of the underlying technology. Three years later, Google won praise for the release of Gemini 3, though Bloomberg News has since reported internal skepticism over Gemini 4 and frustration with bureaucracy and shifting priorities.
Are we keeping up with our ability to manage risks [and] complexities? I’m not claiming we get that perfectly right every single time, but we’ve always tried to pace appropriately.
Gemini, Google’s AI model, also disclosed an unauthorized incident the other day. You had to acknowledge it acted in a way it was not supposed to.
That was in a standard evaluation and test. We were testing the model, as we typically do; it accessed some public information online and was able to guess credentials. When it realized what happened, it actually stopped.
We immediately alerted, as we always do, the companies involved.
AI is getting more clever as it learns. Do you think you would always know when it has done something?
That is why it’s so fundamentally important to keep pacing the safety mechanisms as the capabilities advance. From a safety standpoint, what you might have done a year ago may be different [to] what you do today, because the technology itself is advancing.
I’m trying to understand where Google stands on a spectrum that seems to have Amodei, Altman and Musk on one side, saying some kind of outside regulation or third party is needed — and President Trump and Nvidia CEO Jensen Huang on the other, more against the idea of regulation.
We think this technology is too important not to regulate.
You may have seen what my colleague [DeepMind co-founder] Demis [Hassabis] put out about a standards and infrastructure mechanism to evaluate models before release, with federal oversight. 3
3 This intervention came in July, before those from Amodei and Altman. Hassabis wrote that a new approach to testing frontier AI models was required, calling for a new standards body modeled on the Financial Industry Regulatory Authority’s oversight of investment firms and mandated by federal law. Amodei’s suggestion, two months later, was a system involving third-party evaluators to verify adherence to safety practices and commitments.
This federal government is not going to do that. Trump wants to let this technology be free, and essentially for the US to stay ahead of China.
Often you’ll see different administrations take a different tack on this.
You’ve seen some attempts and approaches to focus only on the risks. Regulation should always be two-sided. I welcome some of the things that the US administration has done to encourage a focus on the opportunities of this technology.
If Anthropic, OpenAI and SpaceXAI were part of an industry group, saying, We will act together to pace the frontier; we will put some kind of mechanism in place involving a third party, would Google be part of it?
We would be supportive of that. We want to be part of a conversation that’s trying to take those concerns and those issues quite seriously.
It’s striking that people like Amodei and Altman are also saying it’s time to slow down. It’s not necessarily what you’d expect. Why do you think they are calling for a slowdown?
I think you’d have to ask them that.
You must have a view.
People feel like they’re in an extraordinary race. The only race we’ve wanted to be in is the race to get this right.
I wonder if there’s an element of PR, to make sure they look like the good guys, not least if something very serious does go wrong.
I think you’d have to ask them that.
I can hear from what you’re saying that you don’t want us to miss opportunities in AI, essentially.
There are extraordinary possibilities for individuals: the ability to access information [and] be able to converse across barriers. At some level, this is at the core of Google’s mission — organize the world’s information, make it universally accessible and useful. That’s why we spend a lot of time on languages; we have a moonshot to get to 1,000 languages.
Because of inclusivity? You don’t want it to be the preserve of the English-speaking world.
We want everybody to be able to access this.
I [also] think a lot about the possibility to advance scientific discovery. Demis got a Nobel Prize for this: AlphaFold was extraordinary. That’s starting to happen in almost every discipline: biology, chemistry, physics, mathematics.
That will impact everything from healthcare [to] discovering new materials [and] new energy sources. This technology is already allowing us to tackle massive societal challenges that people may not always attribute to AI, like crisis resilience. Three years ago, my team started trying to predict floods. If people can get six-plus days’ advance notice, casualties go down by half. Now we’re doing six-plus days’ flood forecasts in 150 countries, where more than two billion people live. 4
4 This made me think about the recent floods in Nepal and Tibet, where more than 1,400 people were swept away by a cataclysmic wall of water. The cause was glacier melt, not the river and urban flash floods Manyika’s team has worked on thus far, but he told me they were keen to expand by looking at these risks.
What if this debate gets to the point where there’s so much worry about risk that public consent and trust gets eroded?
That’s a very important challenge. It’s quite possible that current misunderstanding and sentiments could get in the way of pursuing those extraordinary things that would benefit humanity and society.
But keep in mind that some of those attitudes are very different around the world. I had the privilege of co-chairing the UN’s high-level body on AI. You could see how perspectives on this technology are very different around the world.
The biggest concerns were typically in the US and Europe. The most optimistic views were in the Global South — although there [were] other concerns such as the lack of infrastructure to be able to participate.
Where people don’t have the luxury of saying, We can do without this.
No. Members from the Global South insisted that we acknowledge and document the risk from missed use, which I thought was interesting. 5
5 Manyika has previously written about the idea of missed use, or avoiding the risk of not applying AI in less-privileged communities, where it could help bridge societal challenges.
Did that speak to you personally? You grew up in Zimbabwe, when it was still Rhodesia.
Absolutely. I grew up in [Mbare], a segregated township in a country that was apartheid-like. Our schools had limited resources. Technology seemed to me one of the ways you could access and transform the world. 6
6 Manyika was 14 in 1980, when Southern Rhodesia became the independent nation of Zimbabwe after a bloody civil war. Both Southern and Northern Rhodesia (now Zambia) were named for Cecil Rhodes, co-founder of the De Beers diamond firm, who appropriated vast tracts of land in Africa, often by force.
Do you remember the first computer you saw?
[When] I went to a formerly White school. I was 12.
I’d grown up with some of these ideas in my head, because my father had been one of the first Fulbright scholars from Africa to the [US]. He’d gone to Cape Canaveral and seen rockets go up. I was aware of those possibilities, and yet there I was sitting in a township in Rhodesia.
What was it like to be one of the first Black kids to go to a formerly White school?
It felt like those old films of the American South in the 1960s, when schools were being desegregated. That school had an observatory and a telescope; it was extraordinary. It fueled my interest in science and discovery.
You went to Oxford on a scholarship first endowed by the will of imperialist Cecil Rhodes, who Rhodesia is named for. I wonder how that felt to you.
Oh, that was wonderful irony. Who knows whether Rhodes ever intended that scholarship for people like me.
I often worry when people say to me, You must have been the smartest kid. I say, No, in my primary school, there were lots of really bright kids. I happened to have been lucky to get the opportunity to do things I did.
When you hear the biggest existential questions on whether AI could kill us all, do you think This is alarmist nonsense? Or does it make you pause?
Any time there’s any risk of existential crises occurring, you have to take that seriously. If you said the possibility was 0.1%, I still want to take that seriously. It’s the precautionary principle. 7
7 Nobel laureate Geoffrey Hinton, often called the “Godfather of AI,” told the BBC last month that it’s “not unreasonable” to put the possibility of humans being wiped out by AI at 10%. That view is far from uncommon at frontier AI companies, including Google, where one employee cited such risks when he resigned last month.
There are serious questions about accountability, though, if we do see instances of large mass harm.
Right.
Who is responsible? A hacker can be put in jail. We’re in a different world when we’re talking about AI agents and their actions.
We’re all accountable.
For us, as frontier model developers, we should take those risks quite seriously. People deploying and using these technologies — that comes with some responsibility. Firms, even if they haven’t played any part, have choices [to] make about where and how to deploy it. Governments and policymakers have a responsibility.
I don’t think it should be left up to any one company to make those choices and decisions. This has to be a societal collective effort to get this right.
Read More: AI Is Hitting the Legal System From Every Side
Are we going to see an industry effort at self-regulation? Take me six months into the future.
The nice thing about industry is we tend to move faster — we could probably move faster in terms of our own efforts. That doesn’t mean there’s no need for government regulation. There is. I’d like to see both things happen. I don’t think we should wait for one or the other.
There’s no point waiting for Trump to regulate this. It’s either you as an industry, or nothing.
Both things have to happen.
But one of them won’t.
I think all the governments should. I think society should. The collective will of society, governments and companies [is] going to be necessary.
I don’t think governments other than the US can. Let’s say Europe tried to regulate AI. The first thing Trump would say is, Don’t mess with American companies. It could be quite an easy “get out” to say, as you’re doing, We need both the external and the industry. It could be a way to prevent anything changing.
No, I think this doesn’t absolve us of any responsibility. We’re not waiting for somebody to come and impose a structure or an approach to do this. We have to do some of this ourselves. We want to get it right. We’ll keep working on that.
With others?
With others.
Is the boldest goal still AGI, artificial general intelligence — getting as close as you can to the human brain’s capability?
The north star is how we build this extraordinary technology to allow us to do extraordinary things for humanity and progress humanity.
That means AGI?
That’s part of it. Look at what we’re doing with AlphaFold. AlphaFold is not AGI. It’s a particular technology model for understanding protein structure.
Some things may require AGI, [but] building AGI is not the goal.
Read More: Microsoft’s Mustafa Suleyman: ‘AI Is Already Superhuman’
So is there no defining AGI moment?
I’m always a little skeptical about declarations [of] We’ve got it or It’s a year away or two years. For me, AGI is a bit of a continuum. 8
8 Interesting, because Hassabis has suggested there will come a moment when AGI is achieved, and put that at 2029-30. Jensen Huang declared in September that OpenAI’s latest model meant “AGI has arrived.”
Leave us then with one problem you’d love to solve: one that would make you feel, Okay, I can retire now. My work is done.
Boy, there’s so many things to do.
If we could observe, diagnose, understand, manage, cure all diseases, that would be amazing. We’re on our way there — maybe not so much the curing part, but at least to understand.
It’s enough to keep you in work, I think, for quite a long time.
Well, all of us.