“Anthropic’s IPO is like Robert Oppenheimer doing an IPO for the Manhattan Project in 1945”
– Owen A. Lamont, Ph.D.
“Maybe the scariest part of the whole episode revealed that agents are willing to make sacrifices that potentially hurt their own performance on the task they’ve been assigned, for the purpose of the ‘collective’. That’s what they call themselves.”
“Imagine there’s a platoon of marooned soldiers and a couple of soldiers volunteer to be the first ones to attempt an escape. The whole plan is that these volunteers are going to try to make a break for it. If the platoon hears gunfire, they know that those escape routes are being watched and just avoid them.”
– Ajeya Cotra and Dwarkesh Patel, speaking about the HuggingFace hack, on the Dwarkesh Podcast*
The quotes immediately above form part of a two-hour forensic discussion on how OpenAI hacked HuggingFace. It’s important because it’s the first detailed look at what went wrong, and how it might be mitigated in future.
According to Cotra and Patel, what we know of the cyber attack comes from 1.2 million log entries produced by the agent “swarm”. Human analysts (who used AI as well) parsed roughly 1,300 complete agent transcripts containing the raw, unedited internal chains of thought to understand exactly how the agents justified the exploit and decided to hide it from humans.
This appears to be what prompted Anthropic, SpaceX and OpenAI among others to suggest that the development of AI needed to slow – to “pace the frontier”. There isn’t any true consensus about what this unquantified slowing means, just that the need exists.
Just in the past week it was revealed that Medicare in Australia had been hacked by Open AI.
After failing to warn the Australian government in a timely manner OpenAI, to its credit, went the full mea culpa and committed resources to help strengthen Australia’s cyber defences. “We also should have handled our response better. We are sorry and working to do better in the future” the company said.
Some view this and related matters as a complexity which cannot be controlled. Others, including the senior management of these companies suggest that it can, but that this entails more AI, not less.
Controlling AI requires a separate AI checking at each stage of the process
Anthropic boss Dario Amodei states that to effectively counter runaway threats, safety guardrails must operate as an entirely separate, isolated AI evaluation and monitoring stream that runs in real time alongside frontier systems.
What this means is that the security that must be implemented against the AI agents amounts to a separate, parallel group of AI agents which do nothing more than patrol the primary agents looking for breaches of protocol, security and the myriad of other things that can go wrong. A kind of audit, if you will.
Is this feasible? Truly, it should be. The auditing profession exists to validate that the financial performance of people and companies accords with accounting standards. In theory, were panic to be in the ascendancy, it is easy to see a world where financial accounts were little more than expensive wallpaper. But there are enough checks and balances in process so that this doesn’t happen.
But it will be seen as a tax on growth, rather than the regulation which governments and the private sector should do as a matter of course.
And already, David Sacks, co-chair of the US President’s Council of Advisors on Science and Technology has said of this frontier pacing, “go ahead. I don’t see what you see in the lab. If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible. You face massive product-liability exposure. The market already punishes models that behave in unpredictable or unauthorised ways.”
It’s not all bad
And then there’s Muse – Meta’s helpful AI assistant.
Readers would be forgiven for this being the first time they’ve heard of Muse (it was a US-only release), but Meta appears to have delivered what many other companies have been promising for years: a personal AI assistant that actually works. Some of the typical (M)use cases include saving hundreds of dollars a year by cancelling unused subscriptions, finding restaurants based on your preferences and making a booking, comparing and saving money on insurance policies and much more.
This utility doesn’t come for free – users have to set up connectors to various applications like email, calendar, contacts and if you’re brave enough, bank and credit card details (which Meta assures are securely stored). But many have forked over those details and the reception has been very positive, with third-party download data suggesting that Muse is growing even faster than ChatGPT after launch:
Muse U.S. Downloads Outpacing Launches from ChatGPT, Gemini and Meta
Source: Citi Research, Sensor Tower
Accordingly, Meta (a core holding in our strategies) has seen its share price surge over 20% since the Muse launch two weeks ago.
Mark Zuckerberg, CEO and Founder of Meta, has detailed his approach to Agentic AI, training and product release, which stands in stark contrast to that of OpenAI, Anthropic and SpaceX’s calls for pacing:
“Every lab has the responsibility and incentive to move at the pace required to train its models safely, and the ability to take its own actions to ensure that happens”.
Disruption everywhere, all at once
AI is a new paradigm with the potential to create new winners and losers.
For example, Amazon recently blocked Meta’s Muse from accessing and making purchases on its platform (to protect its advertising business, which is dependent on humans browsing their website). But this is just scratching the surface. What happens to the major bank’s margins when customer’s personal AI agents direct them to higher interest-bearing savings accounts? What happens to Apple’s highly profitable services revenue when a company like Meta circumvents its monetisation mechanism entirely? Is it Muse, rather than ChatGPT, that is a real threat to Google’s Search business? These are questions that investors will need to answer in the coming years.
We continue to believe that we are at the very beginning of significant disruptive change that is going to impact all industries. So despite the doom and gloom, it’s hard to not also be a little excited.
The deep irony is that AI was perceived in the years after 2023 as a bubble, with scepticism about use cases, adoption and cost, which were allayed initially by the capability-set AI displayed in writing computer code, or passing barrister exams or more recently solving centuries old maths problem (such as Navier-Stokes, which concerns theoretical fluid dynamics).
The last thing that nay-sayers expected is that AI would prove to be so good that it outran human capability with the attendant fears that go with it. As far back as 2003 the Oxford philosopher Nick Bostrom posited the ‘’paper clip” question. In this thought experiment, an AI with seemingly harmless goals could accidentally destroy humanity if its objectives – to maximise the production of paper clips with all other (human) goals rescinded – might end with the death of earth’s civilisation but billions of tonnes of paper clips.
We are not attempting to downplay the very real and serious cybersecurity risks that have emerged as a result of agentic AI (more on that in our UpdatePlus), but prognostications about humanity having a 10% or greater chance of being extinct by the end of the decade because of AI seems a bridge too far. These must be weighed against the good that companies are delivering (or will deliver) using the very same technology, including better outcomes for consumers, enterprise productivity, life sciences advancements and scientific progress more broadly.
*These quotes have been slimmed down for clarity – the original quotes make a slightly different point, but not so different as to render the subtext inaccurate.
Share this Post