How AI bias turns into a discrimination claim.

All right. Hello everyone and welcome to another Testudo webinar where this month we'll be focusing on discrimination and bias lawsuits from our AI lawsuit database. If you're a broker attending, we really appreciate your time as discrimination and bias is one of the most asked about topics in the AI risk space at the moment. We hope that with the information shared today, we can enable you to navigate these conversations with clients and address their concerns about third party harm arising from their use of AI tools. This webinar will be recorded and available for distribution following the session.

While Testudo does not cover pure discrimination claims, our regulatory violations insuring agreement contains a carve-back of our discrimination exclusion to address regulatory investigations alleging non-compliance with AI laws. Our lawsuit and incident database has ample cases involving discrimination. So, we decided to focus this month's session on what is becoming a very hot topic in the AI world given the increase in third party facing deployments. In addition to offering AI regulatory violation cover, we hope that as a market, we can serve as a valuable resource for brokers assisting their clients with discussions around how AI claims can manifest from deployment cases.

For a background on how and why discrimination and bias emerge in AI models, we can look at the basics of how these models are built and trained. At its most basic level, AI models are programs that have been trained on data sets to recognize certain patterns or make certain decisions. This reference comes from IBM and should be kept in mind when we start thinking about how harms can arise from models trained on vast data sets. When AI models are trained with data which contains a bias, that bias pattern often ends up surfacing in outputs generated by that model or in the decisions it makes.

Methods of mitigating this bias can include feedback loops where human review or the ability to flag biased responses and thus retrain the model supplies the model with examples of bad behavior from which it learns and hopefully does not reproduce biased outputs. Regarding early examples of this effect and why it matters, one of the easiest examples to draw on is the Amazon hiring bias case, which cut female candidates from eligibility based on the company's hiring history consisting mostly of men.

Now, because the roles listed were largely for technical staff and Amazon had previously hired majority male applicants for these roles, Amazon's hiring algorithm tracked gender as a pattern to be matched in its hiring decisions and basically did its best to arrive at a decision which would suit past hiring decisions. Now, the key issue here is twofold. Both at the surface of the harm that we see, but also beneath the surface as the harms that we don't see right away.

First, there's the evident gender discrimination we would notice at first glance, which may include women with the same technical qualifications being rejected in an early stage résumé analysis. But when an accumulation of hiring results surfaces a bias, what we don't see at first is the failure to hire cases on an individual basis. If an individual candidate understands that their application was rejected, but has reason to believe that it was not reviewed by a human, they may raise a case over algorithmic discrimination.

But how many cases were never raised? What I mean by this is that there may be a substantial number of applicants who are rejected, but their lack of vision into the hiring process or the transparency of Amazon's multi-stage review process involving algorithmic sorting of résumés prevented them from understanding the stage at which their application was denied. This figure of failure to hire cases cannot be determined based on public complaints or lawsuits. In other words, as Rachel Goodman of the ACLU Racial Justice Program puts it, it can be difficult to sue over disparate impact, particularly in failure to hire cases.

Such lawsuits are very rare because it's so hard for someone who never got an interview to identify the policy or practice that led to a rejection. Thus, this type of lawsuit may be rare but material as the failure to hire cases may accumulate without the employer noticing the evident bias. Now what causes this is a lack of transparency around where AI systems are being relied on to make consequential hiring decisions.

So this comprises a key issue around how bias and discrimination arises via AI deployment in businesses. We cannot measure what we don't see. And if candidates are unable to opt out of this automated decision-making and therefore unable to raise complaint over potential algorithmic discrimination, this muddles our view of how much harm is really being done. These answers can become increasingly opaque when we look to generative and agentic AI systems where the decision-making nexus is less visible even to those human reviewers. Thus, we arrive at the issue for commercial entities deploying these systems who tend to absorb liability for the harms they cause.

Why this is relevant to our discussion today is because regulation and case law are beginning to hold companies liable when the AI systems they deploy cause these harms. And these businesses must now implement accountability and transparency principles such that this harm can be measured and mitigated. At Testudo, we insure companies who have deployed generative AI systems in their business operations as they are concerned about downstream liability risk when the systems cause adverse third party effects. Discrimination and bias are one example of a downstream impact that a company's AI deployment can cause. We often underwrite other downstream third party harms such as bodily injury, property damage, personal injury, and IP infringement, and data disclosure losses arising from the AI systems that these companies deploy.

We do not underwrite discrimination risk per se apart from our AI regulatory violation coverage which would address regulatory investigations alleging that the insured has violated an AI state regulation. But since we have ample lawsuit data on those discrimination and bias cases, we're exploring their causes and outcomes today to further educate and inform our brokers. Now, as I mentioned briefly before, the core issue around detecting and correcting bias in training data such that models do not output biased or discriminatory decisions is that it requires a feedback loop to verify or falsify outputs. If a model is not being instructed when it gets something wrong, it becomes very difficult for it to improve its decisions. When we look at large commercial enterprises deploying these models, often their reason for deployment is to compete in their respective industries. In this way, a model that can automate high volume

simple tasks is a massive benefit. And some even say it's better than the risk of human error or professional negligence. However, skeptics of AI enablement would argue that increasingly outsourcing critical decisions to a system operating on pre-trained data can lead to disastrous consequences. Given the additional generative and agentic capabilities of newer systems, there's a degree of unpredictability and potential for even more harm of outputs beyond the bias patterns they may have been trained on. So today we'll cover some of the emerging litigation and incidents involving the commercial deployment of these tools to look at what workforce strategies and governance procedures can assist in this natural risk and ultimately improve our reliance on AI systems for high impact decisions.

As I said, many companies now deploy generative and agentic AI systems to accommodate vast volumes of customer inquiries or administrative burdens. Examples here would include hotels using chatbots to manage guest questions or the healthcare industry using AI-assisted tooling to manage patient administrative logs and checks. Now the point of these systems is to take on a volume of tasks which humans cannot and therefore they are designed to be deployed without that critical human review feedback loop which would verify outputs and avoid harm.

The definition of a consequential decision is emerging in US state regulation to define where these systems can cause adverse effects when deployed in third party facing environments without human review. This largely centers around eligibility and accessibility to opportunities which will be touched on later. Eligibility and accessibility is what also emerges in our lawsuit database often based on the Americans with Disabilities Act and various personal injury legal theories.

Certain industries have also been identified by US Attorneys General as requiring regulation in order to avoid the scenario where an unsupervised AI or generative AI system is given third party data and makes a decision on that third party's access to an opportunity without any human review. Emerging regulation often requires disclosures to third parties when AI is being used or relied on in a consequential decision and privacy notices regarding the use of third party data by a company if they are using an AI tool to process it.

Now, our legal database at Testudo flags public court records when a lawsuit is filed and involves AI or generative AI as a component of the harm caused. When we break this down by sector, we can see where arguments around discrimination and bias are emerging and what types of AI tools tend to cause legal action. To begin with the education sector, typical lawsuits and AI incidents we see here involve students suing educational institutions over their accessibility and autonomy.

Some insights we can pull from these suits involve an evident leaning towards a new baseline of equal access both in terms of being able to interact with the human form admissions processes as well as student accessibility to AI tools in educational settings. In other words, students may allege that AI proctors are discriminatory for identifying plagiarism without a teacher confirming that decision, but simultaneously demand that they themselves be allowed the use of AI tools.

Now, the example we have is Kato versus Palo Alto Unified School District, where a teacher allegedly used an AI plagiarism detection tool called Turnitin to falsely accuse a multilingual Asian student of cheating, forcing him to retake an essay exam, which dropped his grade from an A/B to a D. The school allegedly ignored extensive evidence proving that he wrote the work himself. But the family which sought $150 million from the district says that the school discriminated against the student based on his national origin and background, retaliated against the family for complaining, and never followed proper due process rules before punishing him.

Now it's the final point here which is most important. Most complaints and incidents involving AI plagiarism detectors allege that these tools are negligently relied on to make what we would call consequential decisions. In this case, the teacher allegedly decided to alter the student's grade and punish the student based on the decision of this AI tool. Thus, the family has reason to accuse the school of outsourcing an important decision which restricted the student from academic opportunities to an AI tool without conducting proper human review and disciplinary procedures. Their blind faith in the tool's decision contributed to student harm.

As for some takeaways of the impact of AI tools on accessibility and discrimination in education, we highlight here AI tools relied on by the educational institution to make decisions around plagiarism combined with the growing amount of litigation around students being entitled to use AI in their schoolwork. New standards around what constitutes work actually done by the student and what would constitute cheating is further complicated by the fact that many schools use this plagiarism detecting AI software to flag cases of academic disintegrity.

Thus, we have two sides of an argument using AI as a means to their own end and arguing different standards. If a student argues that they are entitled to use an AI tool in their schoolwork, can they claim that their school is discriminating against them by negligently relying on the decision of an AI tool, thereby outsourcing a decision on that student's academic integrity to technology that the student personally advocates for access to? And for the school's risk management, if they are liable for the outputs of these systems, does the non-zero chance of these systems making an error sign these schools away to years of future litigation from disgruntled students?

In the medical domain, we see applications of AI and generative AI systems in both nonclinical and clinical settings. But there arise discrimination bias issues in both spaces. This is because we're naturally dealing with a vulnerable third party here, the patients. Looking at the administrative side, we see discrimination claims regarding the accessibility of healthcare services. An example here would be restricting patients' access to human administrators and outsourcing healthcare billing to AI systems whose decisions cannot be easily overridden by patients.

This is similar to concerns raised in the education sector as the critical decision of a bill being marked as paid or overdue becomes an accessibility issue when there's not a human escalation procedure available. Discrimination and bias arguments arise here when models trained on past data make, as we've discussed, consequential decisions without human review, and that lack of a human escalation procedure limits vulnerable third parties from being able to rectify this harm.

Wimberly versus Atlantic Dialysis Management Services evidences exactly this. A dialysis patient was wrongly billed for about $1,400 per month as a private pay patient for treatments, even though dialysis staff had confirmed in writing that Medicaid coverage was available and active for three years. The staff allegedly did not communicate with him in writing despite his documented disability-related requests, only calling him minutes after dialysis treatment for billing.

Now the output of an unspecified large language model was provided to plaintiff as a response to his request to discuss hospice instead of human response. So what's evidenced here is an outsourcing of critical decisions to a third party AI tool and an inability of the affected patient to raise the discriminatory decision errors with a human, leading the plaintiff to allege disability discrimination and retaliation under the Americans with Disabilities Act.

In another case, Carlin versus UnitedHealthcare Insurance Company of New York, UnitedHealth used an AI system called nH Predict, which automatically denied Medicare claims for elderly patients, erroneously overriding doctors' decisions with a high error rate. The case alleges that UnitedHealth was aware of this error trend as they allegedly knew the system had a high error rate and that they relied on patients being too sick or underresourced to appeal.

Thus, this case raises consumer protection concerns to illuminate the critical nature of these decisions and why outsourcing these decisions to an AI system would adversely affect a vulnerable population. Rather than raising this under the ADA, the suit includes allegations under the Racketeer Influenced and Corrupt Organizations Act, breach of contract, fraud, and negligence in state consumer laws. However, clearly the argument here relies on the vulnerable characteristics of third parties, raising important questions about AI deployment in healthcare settings where third parties interacting with these systems' decisions are typically ill, underresourced or elderly.

Just reflecting on two sector examples being education and healthcare, our underwriting has undergone a tremendous amount of consideration when we look at what types of AI we find insurable. When generative AI is deployed by a regulated entity in such a manner that it could materially influence a consequential decision, we require a human review process to keep in line with emerging regulatory developments prohibiting the outsourcing of these decisions to ADMT.

Now, what is ADMT and a consequential decision? We touched on this in our last webinar, but in the United States, consequential decisions refers to public accessibility or eligibility for employment, housing, education, admissions, lending, and financial services, insurance, healthcare, legal, and government services. Colorado's Automated Decision-Making Technology Act or the ADMTA is intended to be enforceable as the 1st of January 2027 and is one example of state focus on reducing the risk to consumers regarding their access to, eligibility for, or selection for those outlined industries when decisions are outsourced to AI systems.

In the education and healthcare examples, we can see clearly how student accessibility to human review procedure in a plagiarism or disciplinary review process or patient accessibility to human review in the event that their access to insurance is restricted by an AI system both hit this regulatory inflection point and thus emerging state AI regulation seeks to reduce consumer harm primarily in these sectors. We've discussed a few cases in depth and now we'll get into some more cases from the financial services industry, employment cases, housing and government services.

As attendees today, you'll now hopefully begin to recognize patterns where these AI systems have been deployed in critical decision-making contexts within financial services. Earnest Operations, which is a lender for student loans, faced allegations of algorithmic lending bias. A state alleging that Earnest trained their AI underwriting models on risk variables, including a school's cohort default rate, immigration status, and other factors which disproportionately harmed minority applicants in securing loans.

The Massachusetts Attorney General settled on $2.5 million from Earnest Operations to resolve allegations that their use of AI led to disparate harm for Black, Hispanic, and non-citizen applicants of student loans. The attorney general noted that Earnest's failure to comply with consumer protection and fair lending laws, including through its AI models, unfairly put historically marginalized student borrowers at risk of being denied loans or receiving unfavorable loan terms, impeding their chances of economic growth and opportunity.

Earnest is now required to perform regular risk assessments, maintain model inventories, and complete testing for disparate impacts arising from their AI systems. They're also required to submit regular compliance reports to the attorney general's office, which is a requirement now seen in certain AI state regulations or specific industries. Employment cases frequently involve discrimination claims orbiting a theme we touched on earlier for the education sector. Third parties alleging that their prospective employer's use of AI systems is discriminatory itself.

In Mobley versus Workday, Workday's technology product, an AI-powered hiring algorithm, allegedly automatically screened out job applicants who are Black, over 40, or disabled, rejecting plaintiff Derek Mobley over 100 times despite him being qualified for the positions he applied for. The lawsuit claims that Workday's AI tools were trained on biased data and lack safeguards to prevent discrimination, blocking these groups from even being considered for jobs at thousands of companies.

As AI enabled employment discrimination can be applied to nearly every sector, it's important to flag this theme again. In the Workday case, we see a non-employee discrimination claim. Mobley specifically tested the hiring process by applying to hundreds of different companies which use Workday software, even applying in the middle of the night and received rejections within minutes after applying. This suggested to him that the automated algorithms were filtering him out rather than human hiring managers. And knowing that he held the relevant qualifications, he suspected that Workday's AI used clues from résumés, including graduation year and employment gaps to infer age, race, and disability, and use these points to reject candidates.

As we saw with the failure to hire issue at the beginning of this webinar, we only see the harm in full light because Mobley has raised a complaint. But how many other applicants were denied before the résumé was sent to a human reviewer? Now, interestingly here, because Mobley noticed the common denominator in near instantaneous rejections was the Workday portal, he decided to sue the software company rather than the individual employers. What we see in AI deployer liability is that often if a business deploys an AI system, they're liable for its outputs if they cause harm. Because Mobley knew the name of this specific software, he is able to sue them directly. But when businesses deploy similar systems which do not advertise the name of the vendor behind them, third parties can only see the brand associated with those outputs, which would be the deployer.

To touch on the housing sector, Vargas versus Facebook brings up interesting questions around algorithmic discrimination. Facebook used its ad targeting platform, including tools which allowed advertisers to exclude people, include people, and multicultural affinity, thereby letting housing advertisers block certain users from seeing rental and home sale ads based on race, sex, disability, familial status, and national origin, steering people of color, women, single parents, and people with disabilities away from housing opportunities.

Facebook's own delivery algorithms also automatically skewed which users received housing ads based on protected characteristics even when advertisers did not intentionally discriminate. Clearly this example would manifest as an AI-related claim tied to discrimination. But an even more interesting point here around algorithmic discrimination ties to the recent Meta ruling on addictive algorithms. When we consider this housing case, we can see that the algorithm may have automatically skewed the distribution of potential home buyers to discriminate based on certain characteristics.

But the recent Meta ruling regarding their addictive social media algorithms saw Meta attempt to revive its liability insurers' coverage for this class action. The judge ruled that Chubb and Hartford, among other liability insurers, did not have to pay defense costs as a social media platform deliberately engineered to maximize screen time through specific algorithms cannot credibly claim the resulting engagement or harm was an accident. The court has found so far that the alleged injuries, including addiction, depression, and self-harm, were not the results of unforeseen intervening events or third party actions, but rather were foreseeable consequences of Meta's alleged platform design and business practices.

Now this is limited to the insurer's duty to defend and may change when it comes to indemnification based on facts discovered. But this is a very important point to keep in mind when we look at AI liability. If a model or system is deliberately designed to do something, it becomes increasingly difficult to ascertain whether harms resulting from that design count as an accidental occurrence under commercial general liability policies. So when we look at Facebook's algorithmic design excluding certain groups based on the characteristics, would you say that this algorithmic design causing third party harm would count as an accident? We don't see many liability insurers cover discrimination claims. But when looking at traditional liability harms now arising from the use of AI tools, there will need to be careful consideration and attention to whether the intention of the insured in deploying these systems could frame the resulting harms as reasonably foreseeable.

Finally, to touch on these trends in the government services space, in Abrams versus the Division of Unemployment Insurance, Colorado's unemployment agency wrongly calculated Joshua Abrams' benefits by leaving out his out-of-state wages, paying him only $119 a week instead of what he was owed, but blocking him from fixing the error through a broken phone system and a rigged appeals process. This forced him to visit crowded offices during COVID-19, where he got sick and was hospitalized, while also failing to accommodate his autism under the ADA.

Now, this lawsuit addresses a variety of systemic failures, but the suit's mention of automation and algorithmic fraud play a key role in leading to Abrams' harms. This highlights what he alleges are major flaws in the modern public infrastructure relying on AI systems. The suit was raised under the Americans with Disabilities Act because the state's dependence on automated channels allegedly discriminates against people with disabilities who cannot navigate them. The lawsuit asks the court to order the state to provide a solution to this issue, alternative communication methods, a human escalation procedure, or portal messaging services. And Abrams uses the ADA here to highlight that state services cannot hide behind an algorithm, but must accommodate human interactions with human escalation procedures.

Our work at Testudo is based on identifying insurable deployments of AI, which can lead to accidental errors adversely impacting third parties where the deployer in this case is liable for that impact. When we examine our lawsuit database, we continuously monitor which sectors are seeing an increase in AI-related claims and how the deployment style and human elements of the narrative are used by plaintiffs to allege negligence.

Our mission as an insurer is to assist businesses who want to expand by leveraging the capabilities of AI or who want to reduce pressure on their staff by capitalizing on the time-saving capabilities of AI chatbots. Our job is to ensure that these deployment cases are well governed, used for understandable and useful reasons and intelligently engaged such that third party risk can be reduced. Our role is to be there when accidents happen so that businesses can feel free to expand the capabilities and evolve as corporate entities without holding on to that risk unnecessarily.

A resource we offer to assist in identifying these exposures is running intelligence reports on companies to identify how they're using AI operationally. Among systems identified as currently deployed, these reports frequently source LinkedIn job postings by the insured for responsible AI governance, indicating that businesses increasingly hire for these capabilities and of concern for responsible AI deployment and minimizing errors. Another signal that identifies the insured may be considering their AI risk is frequent evidence we find showing a corporate or public entity's recent AI projects.

These may be test cases where they produced an AI generated marketing campaign or tried a third party vendor tool. If the insured is interested in deploying these systems fully, AI affirmative liability insurance is likely to be key to that enablement. Now, this signals to us that this insured would benefit tremendously from that affirmative AI liability insurance. If their business is undergoing transformation with new AI capabilities, their trust in their broker to secure insurance for the event that deployed tools make errors is crucial. If you're a broker just getting into AI liability risk insurance, please reach out. We'd love to work with you.

Thank you very much for attending our webinar today. As always, feel free to reach out with any burning questions and our team is happy to help.