I was hesitant to jump into the AI punditry around the potential consequences of, and fallout around, Jensen Huang’s resignation as CEO of NVIDIA today — in part because I’m taking a break (or trying to) while I’m on sabbatical from writing this newsletter. However, as the unfolding situation ties in directly with my work and writing around AI safety and “orphan risks,” I thought I’d post links to a couple of previous articles in case anyone’s interested.
The first is a recent arXiv preprint on orphan risks at the frontier of artificial intelligence which, while it doesn’t explicitly mention alleged affairs and Chinese spies, does explore the dangers of taking too narrow a perspective on risk in the development of AI:
The preprint’s title is Orphan risks at the frontier of artificial intelligence: What diverging safety and compliance frameworks reveal about how AI companies choose the risks they prioritize, and it’s available at: https://doi.org/10.48550/arXiv.2608.16895
It’s also available in full here — after the rather lengthy preamble:
Orphan risks at the frontier of artificial intelligence
Note: This is a post in two parts — a rather long paper on orphan risks and frontier AI models (the second part), and a pre-amble on the process that led to this, which is part of an ongoing series of experiments in using Anthropic’s Fable 5 for researching and writing academic papers. Please feel free to ignore the lengthier paper (or …
The preprint considers overlooked or “orphaned” risks associated with the potential impacts of frontier AI models, and explores how a methodology developed around the concept of “risk innovation” might be effectively applied to navigating an increasingly complex AI risk landscape. It’s intentionally outward looking as it focuses on what the consequences of AI might be to different stakeholders. But the approach taken is equally valid if turned inward and used to consider easily overlooked risks to AI’s development and — by extension — potentially beneficial impacts that may be placed in jeopardy by certain actions or assumptions.
Here, the developing situation around NVIDIA and Jensen Huang is a textbook case of how easy to overlook vulnerabilities can have an outsized impact on a company, a sector, an economy, and beyond.1
The second piece of writing is a more recent and explicit article on Huang and his approach to AI safety, which tackles the messiness of risks that aren’t easily resolvable through an engineering mindset alone — including those that arise from human actions and behaviors.
This was the third of three articles on Huang and NVIDIA’s approach to risk and safety. The second article in the series approached this from the perspective of twenty-plus years of work on a tendency to ignore or bury early warnings around tech-related risks. And the third one — and the article I reference below — considered Huang’s approach to risk and safety through the lens of my own work around risk.
As with the arXiv preprint, the article focuses on risks associated with AI that are external to the companies developing the technology. But similarly, the general points it makes are directly applicable to internal vulnerabilities that have long-reaching potential consequences:
Is AI safety just engineering, or is there more to it?
Part 3 of a three-part series. Read part 1 (the back-story) here, and part 2 (Claude’s analysis of Jensen Huang and the EEA’s Late Lessons reports) here.
A common thread between the preprint and the article is the framing of risk as a threat to value — both existing value and potential future value, and value to different stakeholders2 — the need to take orphan risks seriously, and the importance of approaching risk holistically as a complex landscape to be navigated.
Huang’s resignation, the causes that led up to it, and the consequences likely to emerge from it, fall squarely into orphan risk territory. While I suspect that the consequences will not be as disruptive as some are speculating — there’s a solid roadmap for AI development in place that has substantial momentum, and one that will be hard to disrupt substantially in the short term — the shakeup at the top of NVIDIA will nevertheless send ripples through the AI landscape. And my guess is that these are ripples that NVIDIA’s engineers didn’t anticipate within their practical (and technology-centered) solution-focused risk management models.
And yet, what we are seeing play out here is as old as time: social, political, and geopolitical risks that often feel alien to an engineering safety mindset becoming an outsized threat to business and the ecosystems surrounding it.
Of course, this won’t be news to anyone who studies social and political behavior within complex systems. But to the technologists and engineers who are at the vanguard of AI development, it’s probably a timely reminder that humans — and how we respond to the behavior of others as much as how we behave — are as big a threat to existing and potential future value in a technologically complex world, as the technologies themselves are.
The early thinking around the concept of risk innovation and orphan risks btw can be read in the Nature Perspective Why we need risk innovation.
In the case of the Risk Innovation framework, the key stakeholders are the enterprise, its investors, its customers, and the communities it impacts.