Study Finds AI Outperforms Law Professors on Complex Contract Law Questions
By
Alicia Park
FeedBagel synthesis
· 3 sourcesA new study led by Stanford Law Professor Julian Nyarko found that AI-generated answers outperformed responses from 16 law professors on complex contract law questions in a blind evaluation. The study, which involved researchers from Yale, NYU, and the University of Chicago, tested nearly 3,000 anonymized comparisons, according to Hacker News. Contract law was chosen because it requires nuanced reasoning and synthesis rather than rote recall, with no single correct answer, bsky reported. The findings suggest potential significant changes in how legal education is delivered.
Slightly gummy. The crust never quite set.
Summary
A blind experiment tested AI tutors against 16 law professors on contract law questions that required reasoning and synthesis rather than rote recall. The study, authored by Nyarko and researchers from Yale, NYU, University of Chicago and other institutions, found that AI outperformed the law professors by a significant margin. Contract law was specifically chosen because it resists simple answer keys and demands nuanced reasoning with no single right answer.
Key quotes
· 3 pulledContract law was chosen precisely because it resists the answer key.
The 40 questions used in the study...demanded synthesis of competing arguments and a defensible conclusion rather than rote recall.
Testing whether a model could reason where there is no single right answer.
You might also wanna read
Stanford Law Study Finds Professors Prefer AI-Generated Answers Over Peer Responses in Legal Education
A Stanford Law School study led by Professor Julian Nyarko found that law professors overwhelmingly prefer AI-generated answers to student q
law.stanford.edu·20h ago
The Ongoing Problem of AI-Generated False Citations in Court Filings
This article examines the persistent problem of lawyers submitting court filings containing AI-hallucinated case citations, despite increase
Senior Barrister Discusses AI's Transformative Impact on the Legal Profession
A senior English barrister speaks anonymously about how artificial intelligence is poised to fundamentally transform the legal profession. T
spectator.com·5mo ago
The Disconnect Between Legal Tech Vendors and Lawyer Needs
The article critiques the disconnect between legal tech vendors and the actual needs of lawyers, arguing that despite massive venture capita
theredline.versionstory.com·5mo agoThinking Fast, Slow, and Artificial: How AI Is Reshaping Human Reasoning
Oxford-led study finds AI evaluation benchmarks lack scientific rigor
A comprehensive study led by Oxford Internet Institute involving 42 researchers from leading global institutions found that many tests used
oii.ox.ac.uk·6mo ago