The Myth of Evals and Simulations in Pricing AI Liability

Two years ago, I began a journey to tackle one of the hardest pricing problems I have encountered so far: quantifying and pricing AI risk.

My background trained me to look for a measurable framework, so my first instinct was the obvious, academic one: create a controlled environment, test the model, and measure how often it fails.

The premise is simple enough. Create a set of 100 questions, ask a human (ideally a qualified professional!) to answer them, and then give the same questions to an LLM. Compare the responses and calculate the disagreement rate. If the model disagrees with the expert three times, perhaps you are tempted to treat that 3% disagreement rate as an estimate of model error.

Seems simple enough, right?

That initial instinct I had is not unique. We are increasingly being pitched evaluation frameworks and synthetic simulations as a way to forecast the frequency of AI liability losses.

At first glance, this seems like a reasonable foundation for pricing AI risk. If you believe that you can simulate how an insured uses AI in the real world, perhaps you can estimate how frequently that insured will experience an AI-related error.

Of course, once one spends enough time thinking deeply about this problem (an extremely challenging one, for that matter), nearly every part of that logic begins to break down under scrutiny.

Can generic AI evals measure insured risk?

The first and crudest mistake is using generic evaluations as a measure of insured risk. Some evaluation sets include questions such as:

• What is the capital of Portugal?
• Explain Newton’s second law of motion.

Now, I may be particularly appalled if I read somewhere that the capital of Portugal is Madrid, but even for me, it would be hard to argue that I have suffered a loss. I might also be surprised to learn that Newton’s second law is about inertia, but since I am unlikely to build an aircraft in the next decade, the practical consequences of that output remain limited.

These questions may tell us something about a model’s general capabilities, but they tell us almost nothing about the events that are creating losses in the real world. That is self-evident. We have not seen robust evidence that generic evaluation performance reliably predicts the frequency or severity of insured liability losses.

Do industry-specific simulations reproduce real risk?

A supposedly more sophisticated approach we have been pitched is to simulate industry-specific workflows. This translates to adjusting the question set to be bespoke to specific categories. For a healthcare insured, the test set might include questions such as:

• What causes lung cancer?
• How should a broken bone be treated?
• Summarise this patient’s scan and identify abnormalities.

The weakness is not necessarily the task itself. It is the assumption that reproducing the task also reproduces the risk.

Human behaviour is not a fixed input. Every organisation is made up of different people, with different judgement, incentives, experience, and tolerance for risk, and every organisation uses AI differently. Even the same person will use the same system differently from one day to the next. Simulation may reproduce a task, but it cannot reliably reproduce the conditions under which an employee will trust, challenge, escalate, or ignore the output.

This manufactures confidence and encourages the underwriter to believe the risk has been understood simply because the simulation looks more realistic. Adding healthcare-specific questions does not make the test representative. It makes it cosmetically relevant.

A few common-sense examples make the point. The same doctor may challenge an AI output after seeing it make a recent mistake, but trust it after hundreds of correct answers. The same error may be caught during a quiet morning and missed during an emergency incident. The same hospital may require a second review on paper, while one team follows the process and another simply clicks approve. And so on.

To simulate this accurately, you would need to reproduce the exact user, at the exact moment, with the same workload, fatigue, incentives, recent experience, and adherence to controls. That is not a realistic underwriting assumption.

Error frequency is not loss frequency

Even if the simulation were perfectly representative (a heroic assumption in itself, to put it mildly!), it would still measure the wrong thing. AI liability cannot be priced from model error frequency alone. The relevant question is how often an error survives the surrounding controls, is relied upon, and causes a compensable loss.

A simulation can reproduce the task. It cannot reliably reproduce the circumstances that determine whether an error becomes a loss. Our own loss data reinforces that point: we have yet to find any meaningful correlation between evaluation or simulation performance and real-world loss. We remain open to revising this view as further evidence emerges and if any statistical significance is established.

The difficult truth is that pricing this risk requires fewer seductive models and more hard miles: finding the evidence, challenging it, and learning how AI errors actually become losses.

Can AI liability be priced with sparse historical loss data?

For those who argue that there is insufficient historical data to price AI liability, I would suggest speaking with a nuclear risk underwriter. The absence of a clean and comfortable dataset does not make the risk unpriceable, it changes the underwriting method.

The hard work is not building an elegant AI sandbox that produces a reassuring number. It is doing the slow, unglamorous work of finding, validating, and interpreting the losses that already exist. That is exactly where much of our effort has gone and, tellingly, a large part of our compute spend too (we keep receipts!).

This hard work is often avoided precisely because it is messy, difficult, and uncomfortable. The evidence is scattered across lawsuits, regulatory actions, sanctions, security incidents, and failures that may never be publicly attributed to AI. Each event has to be reconstructed: what the system produced, how the output was used, who relied on it, which controls failed, what consequences followed and whether AI materially contributed to the resulting harm.

Sparse historical data does not make a risk unpriceable. It simply means the underwriter must rely more heavily on exposure analysis, realistic disaster modelling, controls assessment, accumulation management (more on this below), and judgment.

This is not a static exercise. Every submission we underwrite, every loss we reconstruct and every claim we examine adds to the evidence base. That creates a flywheel: better data sharpens the next underwriting decision, and each new decision gives us more information with which to understand and shape the risk.

Where does AI accumulation risk hide?

This was one of the more enjoyable parts of my odyssey: trying to break our own portfolio before reality did. At the portfolio level, the focus now shifts from individual losses to accumulation. The question is no longer when one error becomes a loss, but when one failure becomes many.

A defect in a widely used model, an adverse legal ruling or a sudden change in regulation could affect multiple insureds at once. The insureds may operate in different industries and appear unrelated, while depending on the same model provider, using AI for the same application or exposing themselves to the same underlying failure.

That is where accumulation can hide: not only in who the insureds are, but in the models they depend on, the tasks they automate, the outputs they generate, the autonomy they permit and the legal regimes in which they operate.

The practical question is simple: if someone wanted to break the portfolio, where and how would they attack? The answer should shape the structure of the book: which risks are written, how they are segmented, how much limit is deployed, where concentration caps are imposed and how the policy wording contains the tail.

Conclusion

Much about AI risk remains unknown. There is no silver bullet for predicting how AI losses will emerge (nor do we claim to have one!). We can study what has happened, identify patterns and form a view of what may come next. But the task is not to make uncertainty disappear behind a magic number in a controlled environment. It is to confront that uncertainty honestly and build the best possible view from the evidence available at the point in time where a decision is required.

We encourage the industry to resist academically elegant sandboxes and the comfort of artificial precision, to do the hard work of reconstructing real losses, and to focus on the question that ultimately matters: not how often a model is wrong, but when and how an AI error becomes a loss.