The debate over artificial intelligence has intensified again following comments by Dario Amodei, CEO of Anthropic, who warned about the potentially extreme risks associated with the technology’s development and called for greater caution across the industry. Decode39 discussed the issue with Walter Quattrociocchi, professor at Sapienza University of Rome and director of the Center for Data Science and Complexity for Society (CDCS).
Q: Professor, what should we take away from Amodei’s appeal?
A: For me, the interesting part of Amodei’s appeal is not the predictions about superintelligence. Those remain scenarios built on a number of assumptions that are difficult to quantify.
- The important part is something else: Amodei acknowledges that we cannot leave companies to certify the reliability of the systems they build themselves, and he proposes independent evaluators with extensive access to the models. That is a step in the right direction.
Q: How can we take that step?
A: The technical point is straightforward. An LLM generates outputs on the basis of learned regularities and their compatibility with the context. Within that same generative process, it does not possess a general criterion that allows it to determine whether what it is producing is true or reliable. It can be very accurate — even extraordinarily accurate — but asking it to certify its own answer still means querying the same kind of generator.
- Verification therefore has to come from outside. This is why opening models to independent researchers is useful, but it cannot simply depend on companies granting access. We need common protocols, stable access, shared criteria and reproducible results. Otherwise, those being scrutinised also get to decide how they should be scrutinised.
Q: How should we interpret the responses in support of Amodei from figures such as Elon Musk and Sam Altman, compared with that of President Trump?
A: Saying “we want to be scrutinised” is the easy part. The interesting part begins when the evaluator reaches conclusions the company does not like, is able to publish them, and continues to have access afterwards. That is when we will see how genuine this openness really is.
- Trump, meanwhile, perfectly represents the other side of the problem: the logic of the race. If the question becomes “who gets there first, the United States or China?”, any form of caution is immediately interpreted as a loss of competitive advantage. Trump has indeed played down the warnings, insisting on the need to preserve American leadership.
- The problem is that reliability is not achieved simply by accelerating. Nor is it a property that we can merely ask a model to declare. It has to be built through verification, operational limits, controlled access and external accountability.
Q: Some argue that we are witnessing a power struggle among Big Tech companies, which are using the safety debate to enhance their own status. Do you agree?
A: There is no need for a conspiracy theory. Just look at the incentives. For a company, saying “we have built an excellent statistical tool” is good publicity. Saying “we are building something so powerful that it could become uncontrollable” is even better publicity. The rhetoric of power and the rhetoric of danger can reinforce each other.
- But that does not mean the safety problems are invented. It means that, precisely because they are real, we cannot leave companies to define the risk, measure the risk and certify the solution. It would be like asking a pharmaceutical company to determine on its own a drug’s efficacy, side effects and authorisation for sale.
- The issue is always the same: if the producer is also the verifier, there is no independent verification. And there is a second risk.
Q: What is that?
A: Rules designed around the capabilities and infrastructure of the largest operators can become a barrier to entry and further consolidate their power. That is another reason why protocols need to be public, consistent and transparently applied.
Q: China has portrayed calls to slow down as an attempt to curb its development. Is that really the case?
A: The Chinese reaction did not come out of nowhere, because in the same intervention Amodei calls for cooperation on safety while also advocating tighter restrictions on chips and advanced technologies going to China. The Global Timesconsequently described the proposal as a version of Cold War logic.
- That is precisely the problem: if safety becomes indistinguishable from industrial policy, no one will trust the other side’s rules. The only sensible way forward is to build verification criteria that are as independent as possible from the flag under which a model is developed. If a system can perform a certain type of dangerous action, the test should apply to an American, Chinese or European model. If an operational limit is required, that limit must be technically verifiable.
- Otherwise, the United States and China will continue to call “safety” whatever constrains the other side and “innovation” whatever benefits themselves. That is not artificial intelligence governance. It is geopolitics.
Q: Pope Leo XIV’s Encyclical seems particularly timely in this context. How can its demands be reconciled with the dynamics of global markets?
A: On this point, the Encyclical identifies something surprisingly concrete. It says that it is not enough simply to invoke ethics: we need identifiable responsibility, rigorous verification, independent oversight and systems that can be challenged. It also explicitly notes the imbalance between the speed of technological development and the pace at which safeguards and institutions mature. That is exactly the problem.
- The market rewards those who generate faster, scale faster and bring new capabilities to market first. Verification, by contrast, costs money, slows things down and often does not produce a spectacular demo. We therefore cannot expect these two dynamics to align spontaneously.
- We need rules that separate those who build systems from those who certify them, technical limits on actions that can have real-world consequences, independent access to models, and clearly assigned human responsibility.
The bottom line: Above all, we need to avoid a very common misunderstanding: “aligning” a model by teaching it which behaviours we want does not mean that we have made its behaviour reliable. “A learned rule is still part of the system’s probabilistic functioning. An operational constraint is something else entirely,” said Prof. Quattrociocchi
- If I do not want a system to transfer €10 million, I do not simply teach it that doing so would be morally wrong. I build the infrastructure so that it cannot make the transfer without independent authorisation.
- Ethics is necessary. But when a system acts in the real world, sooner or later you also need locks.
