Two Domains Built the Same Architecture and Neither Has Named Why
Sandia’s nuclear inspection deployment says three things that don’t fit together unless you read them correctly. AI outperforms manual inspection. AI running unconstrained is worse than AI with human oversight. And operators will always check the AI’s work. That’s not a transitional arrangement pending greater confidence in the technology. An AI system whose internal reasoning cannot be formally audited does not satisfy nuclear inspection standards regardless of its defect-detection rate. The human layer is there because human verification can be documented, challenged, and independently reviewed. The AI’s cannot.
The alignment research community has named the same constraint from the other direction. “We can’t look under the hood for the formal guarantees about how they work that we’d like for life-critical functions.” That’s not a vendor-specific problem. It’s a statement about the class of systems. OpenAI’s work on AI-assisted human supervision is the same architectural answer applied at scale: use AI to extend the range over which human oversight stays viable as outputs grow too complex for direct evaluation. “Shows promise” is not a solved problem. It’s a managed gap.
Two domains with no institutional relationship, not reading each other’s papers, have built the same structure. AI as mandatory first pass. Human verification as mandatory final layer. No terminal action without human confirmation. The condition producing this isn’t regulatory caution or commercial conservatism. Formal verification of neural systems isn’t available at timescales relevant to current deployment. Both domains are engineering around that fact honestly. Neither is saying so out loud, because saying so requires admitting the constraint has no current solution.
If interpretability research closes the gap, the human oversight layer becomes optional and the deployment economics of AI in regulated infrastructure change completely. If it doesn’t close at timescales relevant to the infrastructure being built right now, then this architecture isn’t a compromise position. It’s the operating condition. The regulatory implication follows directly: state technical competence in AI isn’t optional for governance design, because without an independent technical anchor, private verification markets have no external ground truth to verify against.
The governance language being deployed commercially to substitute for technical verifiability already knows this. It just hasn’t said so.