A study conducted by researchers at Cornell University, Consumer Reports, and the Data & Society Research Institute has found that New York City Local Law 144 (LL144), enacted in 2021 to address bias in AI hiring algorithms, is largely ineffective. LL144 mandates employers using automated employment decision tools (AEDTs) to conduct annual audits for race and gender bias, publish the results on their websites, and disclose the use of such software in job postings.
The study, yet to be published but shared with The Register, sampled 391 employers. It revealed that only 18 had published the required audit reports, and merely 13, 11 of whom also published audit reports, included necessary transparency notices. LL144, according to the study, grants significant discretion to employers to determine if their system falls within the law’s scope, allowing multiple ways to escape its provisions.
Jacob Metcalf, a researcher at Data & Society and one of the study’s authors, highlighted that LL144 lacks a requirement for companies to take action if audits reveal discriminatory outcomes. Despite this, companies found using biased AEDTs may face legal consequences, as audits can be grounds for civil lawsuits on employment discrimination, which can be costly.
The study also revealed that auditors reviewing AEDTs for NYC companies have identified cases of discrimination. Metcalf and his colleagues are working on a second paper focusing on the experience of auditors under LL144, which is currently under peer review.
Laws similar to LL144 in other jurisdictions have faced challenges, with proposals stalling as lawmakers recognize the New York City law’s limited effectiveness. Metcalf noted that sponsors of similar bills are rethinking their structures. The European Union’s AI Act, provisionally agreed upon in December 2023, categorizes AI used in recruiting as “high risk,” requiring reviews before market entry and throughout their lifecycle.
While LL144 has been deemed largely ineffective, the researchers view it as a crucial first step toward better regulation. Metcalf emphasized the importance of broadening the scope of laws addressing AEDT discrimination, suggesting that future legislation should consider any system rendering a score as within scope, eliminating employer discretion.
In conclusion, the study suggests that LL144’s shortcomings offer valuable insights for future enforcement efforts to enhance accountability in AI hiring algorithm regulations.