‘Catastrophic’: Supreme Court warns against unregulated AI use in rulings

The Supreme Court on Thursday cautioned against the use of artificial intelligence (AI) in courts, saying that “if left unregulated…(it) may infiltrate our intellectual work ethic and, before long, render us dependent on its vast capabilities”.

A bench of Justices P S Narasimha and Alok Aradhe set aside orders of the National Company Law Tribunal (NCLT) and National Company Law Appellate Tribunal (NCLAT) for relying on fake, AI-generated judgments. It noted that the August 28, 2024, NCLT order, as well as the September 11, 2025, NCLAT order, cited non-existent judgments, and where the citations were correct, the content had AI-generated paragraphs.

“Judicial process and the judgment under challenge are tainted by the usage of materials which are said to be precedents, but in reality, they are unreal, fake, and do not exist at all. A decision of a court or an adjudicating authority based on material which is fake and hallucinated is no decision at all, and it amounts to subversion of the rule of law. Such a decision is unsustainable and has to be set aside at the earliest,” the Supreme Court said.

The bench also asked the Bar Council of India “to constitute a committee and deliberate on this issue of members of the bar submitting such fake and hallucinated material before the court as if they are precedents of law.” It said that the Council “must take up this issue with utmost seriousness, deliberate earnestly, and prescribe a guiding principle to prevent such occurrences, along with the disciplinary action that will follow a violation of the norms.”

‘Zero tolerance for unverified AI-generated precedents’

Taking a strong view of the use of such fake AI material in judgments, the court said that “for those in the province of adjudication and determination of disputes, this by-product of AI, i.e., the production of fake, non-existent, and hallucinated material and its utilisation as precedents in law, is like the release of methyl isocyanate in the province of law and justice: invisible, insidious, and catastrophic by the time anyone notices. It not only contaminates but takes away the very lifeblood of judicial determination.”

Calling for a “zero-tolerance mode” by courts “for producing, citing or using AI-generated precedents without verification,” the bench said, “It is a misconduct on the part of an advocate to cite such judgments without verification.” It further said that it is equally a serious lapse if a judge relies on “such a fake or hallucinated AI-generated material as precedents in support of the determination”.

“We have no hesitation in declaring that such a decision is no decision in the eyes of the law, irrespective of whether such material had a direct or indirect bearing on the decision-making. Such decisions are to be set aside even if an iota of fake or hallucinated material enters the decision-making process, as it would violate the sanctity of adjudication. It is absolutely necessary to maintain integrity in decision- making, and we reiterate and declare zero tolerance for the Bar as well as the Bench to cite, refer to, or rely on such material. It is also clarified that our judgment shall have no bearing on the rightful use of AI, but on the presentation or reliance on fake or hallucinated material as if it were a court precedent,” the bench said.

Story continues below this ad

‘Need to be extra cautious while using AI’

The Supreme Court said that dependency on technology has never been a problem for the dispensation of justice, as courts have seamlessly absorbed technology and made it an integral part of court systems. “The story of AI, as it is unfolding, is, however, different, in fact, transformative, as it is not just an aid to assist us in our work, but is an alternative to our own thinking, reasoning and even decision making. This is where we need to be extra cautious, as unregulated use of AI will insidiously enter legal practice, the process of judicial decision-making and decision-making itself,” the bench warned.

“Wisdom and foresight compel us to recognise human vulnerability to seek comfort in delegation, but if thinking is delegated and it forms a habit, it will have serious consequences for the core of human existence, which lies in its capacity to think – to discern the distinction between what is right and what is wrong, truth and falsehood, virtue and vice, dharma and adharma. This capability is neither given nor superimposed by birth, but arises from a deliberate, disciplined, and systematic training of the mind alongside lived experiences; it is a battle of the mind against bewitchment caused by the uncertainties between fact and fiction, what is real and what is unreal, propriety and impropriety, as well as what is just and unjust,” the court further said.

The judgement added, “This intellectual exercise, coupled with experience and foresight, enables us to choose between competing values, as well as to take hard decisions with courage and conviction, and to bring about a beautiful balance between the need for order and the quest for justice. A struggle to arrive at truth, it is a Saadhana. In fact, the secret is in the Saadhana itself, for without this deliberate, conscious, and continuous practice of scientific temper, we lose the capability to discriminate between what is right and what is wrong. Lose this, and we would have lost everything.”

Need ‘absolute control’ over AI usage

The Supreme Court emphasised that it was “compelling and necessary to have absolute and total control over the application and usage of AI”. “The control lies in being two steps ahead of its application and in making deliberate choices about when and where to apply. We are aware that this is not an issue that can be resolved through judicial orders and declaratory judgments, but only through public policy and enforceable rules and regulations. We are also aware that the process has commenced, the regulations are being deliberated, and they will be notified after due process and in due course,” it said.

Story continues below this ad

It added that “the real success is, however, not in the making of the rule or regulation, but to be found in the power of the will of the Bar as well as the Bench, to harness this science and apply it with care and caution. No other facet of law and its practice has ever demanded a higher and deeper corroboration and coordination between the Bar and the Bench than the need to identify, decide, and apply AI to adjudication and the determination of disputes”.