The State Bar of California has revealed that some of the multiple-choice questions included in its controversial February 2025 bar exam were developed with the assistance of artificial intelligence (AI), raising concerns about the integrity and preparation of the exam. This admission comes amid widespread criticism over the exam's technical difficulties and challenging content, which affected many test-takers.
In an official news release, the legal licensing body disclosed that alongside questions provided by Kaplan and those drawn from a first-year law students’ exam, a subset of 23 scored multiple-choice questions was created by ACS Ventures, the State Bar’s psychometrician, using AI tools. The remainder of the 171 scored questions included 100 questions authored by Kaplan.
Leah Wilson, executive director of the State Bar, defended the process by stating to the Los Angeles Times, "We have confidence in the validity of the (multiple-choice questions) to accurately and fairly assess the legal competence of test-takers."
However, the involvement of AI in the exam’s creation has sparked considerable criticism among legal educators and experts. Mary Basick, assistant dean of academic skills at the University of California, Irvine, Law School, described the situation to the Los Angeles Times as “worse than we imagined” and expressed disbelief that the questions were “drafted by non-lawyers using artificial intelligence.” Basick characterised the disclosure as “unbelievable,” highlighting the unexpected nature of this development.
The February exam itself was marred by multiple complications, including repeated crashes of the online testing platform before some candidates could even begin, difficulties with completing and saving essays, and technical glitches like screen lags, error messages, and restrictions on copying and pasting text. These issues compounded the stress and challenges faced by examinees.
Katie Moran, an associate professor at the University of San Francisco School of Law who specialises in bar exam preparation, labelled the revelation a “staggering admission” in comments to the Los Angeles Times. She pointed out that “The State Bar has admitted they employed a company to have a non-lawyer use AI to draft questions that were given on the actual bar exam. They then paid that same company to assess and ultimately approve of the questions on the exam, including the questions the company authored.”
Andrew Perlman, dean of Suffolk University Law School and an advisory council member of the American Bar Association Task Force on the Law and Artificial Intelligence, explained that until now, he had not heard of AI being used to develop bar exam questions or of any standards regulating such use. He expressed a cautious but pragmatic view, noting that the rapid adoption of AI technology makes its use in exam development unsurprising. Perlman emphasised that while AI can be a useful tool for creating assessment questions, careful vetting by subject matter experts is essential. He also predicted that the use of such technology will continue to grow, and in the future, legal professionals could face scrutiny for failing to utilise AI resources. “We will be worried in the future about the competence of lawyers who don’t use these tools,” he said.
Following the controversy and technical failures, the State Bar announced it will petition the California Supreme Court to adjust the test scores of individuals who took the exam in February, an indication of the exam’s disputed fairness and accuracy.
The Seattle Times is reporting on these developments surrounding the February 2025 California bar exam, highlighting the emerging complexities introduced by incorporating AI in professional licensing assessments.
Source: Noah Wire Services