Jensen Huang gave the AI industry one genuinely useful piece of engineering advice and then wrapped it in a governance theory that falls apart the moment you look at how software actually gets shipped.
The advice: if it’s not ready, just hold it back. The theory: no new laws needed, because market forces will keep AI safe. Nvidia’s CEO said as much on Tuesday, dismissing calls for new AI security rules while pointing at how fast AI infrastructure is growing and how badly developers need to stop pushing half-finished products out the door.
I write and maintain open source agent code. I’ve held things back. I’ve also shipped things I shouldn’t have. So let me take both halves of that seriously, one at a time.
The part that’s actually right
“Hold it back” is the most underrated discipline in agent development right now. Not because of existential risk arguments, but because agents have a blast radius that traditional software doesn’t. A web app with a bug renders wrong. An agent with a bug calls a tool, and that tool touches a filesystem, a billing API, a production database, or someone’s inbox.
In practice, “not ready” for an agent means specific, boring things:
- Tool permissions default to broad instead of narrow
- No audit trail of what the agent actually did, only what it said it did
- Destructive operations that don’t require confirmation
- No kill switch that works mid-run
- Prompt injection surface you’ve never tested against
- Error handling that silently retries instead of stopping
None of that is exotic research. It’s a checklist. And every maintainer who has shipped an agent framework knows the pull to skip it, because the demo works, the stars are climbing, and someone in your issues is asking for the feature today. The pressure to ship before you’re ready is real and it does not come from a lack of regulation. It comes from attention.
Where market forces don’t reach
Huang’s argument is that competition handles safety: ship something unsafe, lose customers, correct course. That loop is real when there are customers. It is largely absent in the part of the ecosystem where most agent tooling is being invented.
I have no customers. I have users who pay nothing, owe me nothing, and can fork the repo the second I disappoint them. There is no revenue signal to punish me for a bad release, and no revenue signal to reward me for spending three weekends on sandboxing instead of a flashy new integration. The market feedback I get is stars, issue volume, and a Hacker News thread. Those things reward novelty far more than restraint.
That matters because the supply chain runs downhill. Enterprises buying AI products are, in a large number of cases, buying a wrapper around packages maintained by people like me. Market discipline applied to the vendor at the top does not automatically propagate to the unpaid dependency four layers down. It usually stops at whoever signed the contract.
So when the pitch is “leave safety to us,” my honest reaction is that the “us” is doing a lot of quiet work. Nvidia sells the compute. It does not review my tool-permission model.
New social norms, load-bearing phrase
Huang has also said society needs new social norms for the age of AI. I actually think that’s the strongest thing he’s said on this subject, and it slightly undercuts the deregulation point. Norms are how communities enforce behavior without laws, and they don’t materialize because a keynote asked for them. Somebody has to build them, write them down, and hold the line when it’s inconvenient.
Open source has done this before. Semantic versioning is a norm. Security disclosure timelines are a norm. Deprecation cycles, changelogs, signed releases — all norms, most of them unenforced by anything but peer expectation and mild public shame. They work reasonably well, and they took years of argument to settle.
We don’t have the equivalents for agents yet. There’s no shared convention for declaring what tools an agent can reach, no standard way to label a release as “demo, do not point at production,” no agreed disclosure process for a prompt injection vector in a popular framework. Those are all buildable by the community, without a single new statute. They’re also unglamorous, which is exactly why they haven’t been built.
The uncomfortable middle
Huang’s position and mine overlap more than the headlines suggest. He says the fix is developer restraint plus new norms. I agree. Where I part ways is the assumption that restraint emerges naturally from competition. In my corner of this, competition is what erodes restraint. The fastest-moving repo wins attention, and attention is the only currency on offer.
Meanwhile the infrastructure keeps compounding — Huang expects to sell twice as many chips next year, and he’s framing AI as essential infrastructure rather than a product cycle. More compute means more agents running unattended in more places. The gap between “works in a demo” and “safe in production” gets more expensive to ignore every quarter.
So take the advice. Hold it back. Just don’t mistake it for a system.
đź•’ Published: