AI Tutors Outperform Law Professors in Blind Study on Contract Law Reasoning
By
Alicia Park
Good intentions, undercooked execution. The bake is missing.
Summary
A blind experiment tested AI tutors against 16 law professors on contract law questions that required reasoning and synthesis rather than rote recall. The study, authored by Nyarko and researchers from Yale, NYU, University of Chicago and other institutions, found that AI outperformed the law professors by a significant margin. Contract law was specifically chosen because it resists simple answer keys and demands defensible conclusions from competing arguments.
Key quotes
· 3 pulledContract law was chosen precisely because it resists the answer key.
The 40 questions used in the study demanded synthesis of competing arguments and a defensible conclusion rather than rote recall.
Testing whether a model could reason where there is no single right answer.
You might also wanna read
Stanford Law Study Finds Professors Prefer AI-Generated Answers Over Peer Responses in Legal Education
A Stanford Law School study led by Professor Julian Nyarko found that law professors overwhelmingly prefer AI-generated answers to student q
law.stanford.edu·20h ago
The Ongoing Problem of AI-Generated False Citations in Court Filings
This article examines the persistent problem of lawyers submitting court filings containing AI-hallucinated case citations, despite increase
Analysis of Copyright Law Ruling in Authors Guild v. OpenAI and Implications for AI Training Data
This article analyzes a significant copyright law ruling in Authors Guild v. OpenAI, where Judge Sidney Stein denied OpenAI's motion to dism
authorsalliance.substack.com·6mo agoThinking Fast, Slow, and Artificial: How AI Is Reshaping Human Reasoning
Senior Barrister Discusses AI's Transformative Impact on the Legal Profession
A senior English barrister speaks anonymously about how artificial intelligence is poised to fundamentally transform the legal profession. T
spectator.com·5mo ago
Study Reveals Student Perceptions of AI Coding Assistants in Programming Education
This exploratory study examines student perceptions of AI coding assistants in an introductory programming course. Researchers investigated
arxiv.org·6mo ago