The California Supreme Court urged the California Bar Association on Thursday to explain how and why it used artificial intelligence to ask multiple choice questions for its February lawyer bar exam.
The California Supreme Court, which oversees the state bar association, revealed Tuesday that its justice was not notified before the exam, and the state attorney allowed its independent psychologists to use AI to develop a small portion of the questions.
The court on Thursday increased public pressure on the state bar association to explain how it uses AI to develop problems and what measures it has taken to ensure the reliability of the problems.
The state bar association petitions the court to adjust test scores for hundreds of potential California lawyers, these requirements Technical issues and violations In the February exam.
The controversy itself exceeds the use of artificial intelligence in state bars. It’s about how the state bar association uses AI to develop questions, and the rigor of its review process, is used for a high-stakes test that determines whether thousands of aspiring lawyers can be enacted in California every year.
It also raises questions about transparent state attorney officials trying to abandon the multi-layered attorney bar exams for lawyer reviewers nationwide (the system used by most states) and introduces a new hybrid model of in-person and remote testing to cut costs.
In a statement Thursday, the Supreme Court said it was seeking answers as to “how and why AI was used to draft, revise, or otherwise develop certain multiple-choice questions, efforts took to ensure the reliability of the AI-assisted multiple-choice questions before they were administrator, the reliability of the AI-assisted multiple-choice questions, whether any multiple-choice questions were removed from scoring because They were determined to be unreliable, and the reliability of the remaining multiple choice questions are used for ratings.”
Last year, the court approved a $8.25 million five-year contract with Kaplan to raise 200 test questions for the new exam. State bars also hired a separate company, Meazure Learning, to manage the exams.
It wasn’t until this week (nearly two months after the exam) that the National Bar revealed Press release It deviates from the plan to use the Kaplan exam service to write all the multiple choice questions.
exist Promotion Conferencethe State Bar Association revealed that 100 of the 100 multi-choice questions of 171 scores were asked by Kaplan, and 48 of them were drawn from first-year law student exams. A subset of the smaller 23 rating questions was proposed by state bar psychologist ACS Ventures and developed through artificial intelligence.
“We are [multiple-choice questions] “Evaluate the legal competence of examiners accurately and fairly,” Leah Wilson, executive director of the State Bar Association, said in a statement.
Alex Chan, an attorney who oversees the California Bar Exam, supervises Times Arec on Tuesday, supervises Times Arec, who uses AI for only a small percentage of issues, not necessarily to ask questions.
Chen also pointed out that the California Supreme Court urge The state bar association in October reviewed “the availability of any new technologies, such as artificial intelligence, that could innovate and improve the reliability and cost-effectiveness of such testing.”
“The court gave it to me guide Consider using AI, which is exactly what we are going to do. ” Chan said.
Chen later explained that the process would be reviewed and approved by the court.
Chen told the Times on Thursday that state law firm officials did not tell the lawyers’ reviewers committee that plans to use AI before the exam.
“It was never known that the committee used AI before the exam took place, so it was impossible for it to take into account its use, let alone its use,” Chan said.
Katie Moran, an associate professor at the University of San Francisco Law School who specializes in preparation for bar exams, said that this begs for a range of questions.
“Who directs ACS Ventures at the State Bar, a psychometrics company that has no background in writing exam questions, multiple choice questions that will arise in the bar exam?” she said LinkedIn. “What guidelines does the State Bar Association provide?”
Mary Basick, assistant dean of academic skills at UC Irving’s Law School, said there has been a huge change in the state lawyers’ board of lawyers or the California Supreme Court that has not approved state law firms to raise questions.
“What they approved was a multi-choice exam with Kaplan drafting questions,” she said. “Kaplan is a bar preparation company, so of course, there is knowledge about the legal concept being tested, the lawyer exam itself, how the legal concept of the problem should be structured. So, it is not considered a big change.”
She noted that any significant changes that could affect examiners’ preparation for the exam would require two years of notice based on California’s business and professional codes.
“Generally, these types of issues take years to ensure they are effective and reliable, and there are multiple review steps,” Basic said. “There is simply not enough time to do that.”
Basic and other professors have also raised concerns that hiring non-legal-trained psychologists to use AI to ask questions and determine whether the problem is effective and reliable is a conflict of interest.
National Bar objected to the idea: “The process of validating the problem and testing reliability is not subjective, and the statistical parameters used by psychologists regardless of the source of the problem,” it said in a statement.
The State Bar Association told the Times Tuesday that before the exam, the content verification team and subject matter experts reviewed all issues, including legal accuracy, minimum ability and potential bias.
The state bar association says that when based on reliability, the combined scores come from multiple choice questions from all sources, including AI, “above the psychometric target of 0.80.”
The State Bar Association has not answered questions about why it deviates from Kaplan to select multiple choice questions for all exams. It also does not elaborate on how ACS Ventures uses AI to develop its problems.
