A New York lawyer was sanctioned by a federal judge last week after submitting case law generated by ChatGPT that turned out to be entirely fabricated, a stark illustration of the challenges posed by the increasing reliance on artificial intelligence in professional fields.
Judge Kevin Castel of the Southern District of New York imposed a $500 sanction on attorney Steven Schwartz, and his firm, Levidow, Lefebvre & Bergman, for presenting six non-existent legal cases in a copyright infringement lawsuit. The judge called the submission “a unique and unsettling circumstance,” according to court documents.
The incident highlights a critical flaw in current generative AI technology: its propensity for “hallucinations,” or the creation of false, misleading, or entirely fabricated information. Experts warn that this isn’t a bug to be fixed, but a fundamental characteristic of large language models (LLMs). These models are designed to generate convincing text, not necessarily truthful text, according to a research guide from the University of Montana library.
“It’s unlikely that the problems contributing to hallucinations will be solved, since the nature and purpose of LLMs is not to tell the truth, but to produce a convincing response,” the guide states. This realization is prompting a re-evaluation of how AI-generated content is used, particularly in fields where accuracy is paramount.
The issue extends beyond the legal profession. Institutions are grappling with how to assess the trustworthiness of AI outputs across academic workflows, from research discovery to learning support. Clarivate, a company specializing in analytics and information services, notes that traditional quality assurance methods are inadequate for evaluating AI-generated content.
Evaluating AI output requires a critical approach, similar to that used when assessing information found online, but with additional considerations. Generative AI tools are trained on vast datasets of human-created content, which may contain biases or inaccuracies. These flaws can then be amplified or perpetuated by the AI itself, as noted by a guide from the library at Franklin & Marshall College.
The West Point library recommends employing fact-checking techniques and the SIFT method – Stop, Investigate the source, Find better coverage, and Trace claims, quotes, and media to their original context – to mitigate the risk of misinformation. However, even these methods are not foolproof.
The incident in New York has prompted discussion about the ethical responsibilities of lawyers and other professionals who utilize AI tools. Judge Castel’s order explicitly warned attorneys against relying on AI without independent verification. The judge stated that the use of AI requires attorneys to “confirm its accuracy” and that a lawyer’s duty of candor to the court remains paramount.
As of February 22, 2026, no formal guidelines or regulations governing the use of generative AI in legal proceedings have been issued by the New York State Bar Association. The case remains under review, and further sanctions are possible pending a hearing to determine the full extent of the misconduct.
