Jensen Huang is right that AI safety is an engineering problem, and that’s exactly why leaving it to market forces won’t work.
Speaking at Salesforce’s Dreamforce conference, the Nvidia CEO made his position plain: AI is “hardware and software” built by humans, safety is not a legal problem, and no new laws are needed. His advice to engineers everywhere came down to a single line. If it’s not ready, just hold it back.
I’ve spent enough years in open source to recognize that sentence. It’s the sentence you say when you’ve never had to be the person saying no.
Holding it back is the hardest thing to actually do
Every maintainer I know has held something back. A feature that mostly worked. A release that passed CI but felt wrong. An agent tool call that did the right thing nine times out of ten and something unpredictable the tenth. Holding it back is not a technical decision. It’s a social one, and it’s expensive.
Here’s what it costs in practice:
- The contributor who spent three weekends on the patch loses momentum and sometimes never comes back.
- Downstream users who were waiting file issues asking why the roadmap slipped.
- Someone forks the project, merges the thing you held back, and ships it anyway.
- The next time you want to block something, you have less credibility, not more.
That last one matters most. Saying no burns capital. In a well-funded company with a functioning safety org, that capital gets replenished. In an unfunded project, or a startup racing a competitor, it doesn’t. Which means the correct behavior is also the one the incentives punish. That’s not an engineering failure. That’s a structural one.
Market forces reward speed first, safety second
Huang’s belief is that the market will sort this out — that unsafe products fail and safe ones win. There’s a version of that which is true over a long enough horizon. Software that keeps breaking loses users eventually.
But “eventually” is doing a lot of work in that sentence. The agent tooling space right now rewards the team that ships the impressive demo this quarter. Safety work is invisible when it succeeds. Nobody stars your repo because your agent declined to run a shell command it shouldn’t have. Nobody writes a thread about the sandbox that held. The reward signal points almost entirely toward capability, and it points there loudly.
I say this as someone who agrees with the underlying engineering claim. Agent safety is mostly engineering. It’s permission scoping. It’s making destructive operations require explicit confirmation. It’s treating tool output as untrusted input instead of gospel. It’s audit logs you can actually read. None of that requires a new law to invent. All of it requires someone to be paid to care about it when the release date is Friday.
The asymmetry nobody at the keynote mentions
Huang also said he expects Nvidia to sell twice as many chips next year. Both statements were made by the same person in the same period, and they’re not in conflict from where he’s standing. Nvidia sells the substrate. The safety decisions happen several layers up, in code written by people who don’t have Nvidia’s balance sheet or Nvidia’s ability to absorb a delayed launch.
So the advice travels downhill and gets heavier on the way. A hardware company telling application developers that safety is their problem is technically accurate and practically incomplete. Yes, it’s our problem. It’s our problem with a fraction of the resources and none of the market position that makes saying no survivable.
This is where I think the framing needs adjusting rather than rejecting. The question isn’t engineering versus laws. It’s whether the people doing the engineering have the standing to stop a release. In most of the projects I contribute to, that standing comes from norms we built ourselves — review requirements, security policies, release checklists that are boring and slow and occasionally save someone from shipping something bad.
What actually helps
If safety belongs to engineers, then engineers need the tooling and the cover to exercise it. Concretely, that looks like default-deny permissions in agent frameworks rather than opt-in guardrails. It looks like shared threat models we can copy instead of each project reinventing one. It looks like making the safe path the path of least resistance, because relying on individual discipline at scale has never worked in any part of software.
Huang’s instinct isn’t wrong. Building things carefully is better than being told to build things carefully. But “just hold it back” is a description of the outcome we want, not a mechanism for getting there. The mechanism is the hard part, and it’s the part that doesn’t fit in a keynote line.
The engineers he’s addressing already know when something isn’t ready. What they often lack is the ability to act on it. That gap is where the actual work lives.
🕒 Published: