The Optum Algorithm: Racial Bias, Proxy Variables, and Justice in Medical AI

It was measuring spending, not sickness — a bioethics case study

• Who: Ziad Obermeyer, an emergency physician turned researcher, and his colleagues — plus the roughly 200 million Americans scored each year by tools like the one they audited.
• What: A widely used health algorithm ranked patients for extra care using predicted cost as its measure of need. At the same risk score, Black patients were far sicker than white patients. The tool routed help away from the people who needed it most.
• Where / When: A large U.S. academic hospital; published in Science, October 2019; the repair and the argument continue today.
• Why it matters: The case is the clearest example on record of algorithmic bias — a system that harms one group without anyone intending it to. It shows how a proxy variable (a stand-in measure for the thing you really want) can carry decades of inequality into a brand-new machine. And it forces a hard question: can a tool be accurate and unjust at the same time?
• Pair with: AI in Medicine

The Case

Start with a number in a chart. At hospitals and insurers across the United States, every patient in certain health plans gets a risk score — a number from 1 to 100, recalculated as claims roll in. The score predicts how much trouble a patient’s health will bring next year. Patients above the 97th percentile are flagged for automatic enrollment in a care-management program. Patients above the 55th get a closer look.

Nobody in the building picks the number. Nobody in the building can quite say how it is picked. The score arrives with the software.

What care management is. Some patients are far sicker than others, and a small share of patients account for most health spending. Care-management programs give those patients extra support: a nurse who calls to check in, help juggling many prescriptions, fast access when something goes wrong. The programs work — they keep fragile patients out of the emergency room. But the slots are scarce and expensive. So the whole system turns on one question: which patients should get them?

That question — who gets the extra help — is what the algorithm was built to answer. Tools of this kind are applied to roughly 200 million Americans every year (Obermeyer et al. 2019). The one in this story was among the most widely used in the country.

The people who finally checked the number came at it from two directions. Ziad Obermeyer was an emergency physician before he was a researcher. Years of ER shifts had left him with a working obsession: medicine’s judgments — human or machine — are tested against messy reality, and they fail more often than anyone records. By the late 2010s he was a professor at Berkeley, studying what algorithms in medicine actually do. Sendhil Mullainathan, his longtime collaborator, was an economist — a MacArthur “genius” award winner who had spent a career studying human bias, and who saw algorithms as a new kind of evidence about it. With them were the physicians Brian Powers and Christine Vogeli.

What the team got, at one large academic hospital, was a dataset that almost never exists. Companies guard their algorithms’ scores as trade secrets. Hospitals hold the medical records. The two are almost never in the same room — which is why claims about algorithmic bias usually end in a shrug, with no way to check. Here, for once, both halves sat side by side: the machine’s ranking, and the truth it was supposed to track.

So they asked a simple question. Take a Black patient and a white patient with the same risk score. The algorithm says they have the same level of need. Are they, in fact, equally sick?

They were not. At the same score, Black patients were carrying about 26 percent more chronic conditions than white patients (Obermeyer et al. 2019). And the gap was not just a count of diagnoses. It showed up in the lab values themselves: at the same score, Black patients had higher blood sugar, higher blood pressure, worse kidney function, more severe anemia. These were not borderline cases sorted slightly differently. These were sicker bodies, scored as if they were healthier ones. The algorithm was telling care managers the two groups were equally needy. The blood work said otherwise.

The team then ran the repair as a thought experiment. Keep the program the same size. Rank patients by actual sickness instead of by the algorithm’s score. The share of Black patients selected for extra help would jump from 17.7 percent to 46.5 percent. Nearly two-thirds of the Black patients who should have been getting help were not getting it.

How does a machine do that? Not the way most people first guess. The developers had not fed the algorithm race at all. They had excluded it on purpose, in the belief that leaving race out made the tool fair. The problem was hiding somewhere else — in a single design choice that had seemed purely technical.

The algorithm needed a target: a number, in the data, meaning “how much will this patient’s health need next year?” No such number exists in any database. Sickness is not a field you can sort by. So the designers chose something close by that is in the data, down to the penny: next year’s health-care costs. Predict the spending, the logic ran, and you have found the need.

What a proxy variable is. A proxy variable is a stand-in: something you can measure, used in place of something you actually care about but cannot measure directly. Proxies are everywhere and often harmless — a thermometer reading is a proxy for how feverish you feel. The danger comes when the proxy and the real target drift apart for some people more than others. Then a system built on the proxy will be quietly wrong in a patterned way.

Cost is a proxy for sickness — and it is a warped one. On average, sicker patients do cost more. But cost does not measure sickness alone. It measures sickness times everything that stands between a patient and care. Less money is spent on Black patients who are exactly as sick as white patients — the study measured the gap directly. The reasons are layered. Some are practical: distance from clinics, jobs without sick leave, thinner insurance. Some live inside the exam room, where research has long shown that Black patients’ pain and symptoms are taken less seriously, and fewer tests and referrals follow. And some of it is earned distrust. A community that remembers Tuskegee comes to the doctor later, and less often.

Every one of those forces leaves the same fingerprint in the data: lower spending. The algorithm read that fingerprint exactly as designed. Less money spent meant a lower predicted cost. A lower predicted cost meant a lower score. A lower score meant no care manager, no extra support — and another year in which less was spent. The bias did not just persist. It looped.

The proxy feedback loop. Each step follows from the last — and the loop closes on itself. No step requires anyone to intend harm.

The study appeared in Science on October 25, 2019 (Obermeyer et al. 2019). The response was unusually fast for a research paper. Within days, New York’s financial-services and health regulators sent a joint letter to the algorithm’s corporate parent, demanding proof that the tool did not discriminate — “These discriminatory results, whether intentional or not, are unacceptable and are unlawful in New York,” it warned. Newspapers around the world covered the finding.

A note on the name. The Science paper never names the product — that was part of the deal that got the researchers access. Reporters quickly identified it as Impact Pro, sold by Optum, a subsidiary of UnitedHealth Group. The company disputed parts of the coverage but engaged with the researchers on a fix. Because the underlying design choice — cost as the label — was industry-standard, the authors stressed that the problem was bigger than any one product.

Then came the part that makes this case unusual among scandals: a third act. The researchers did not stop at exposure. Working with the manufacturer, they rebuilt the target — predicting a measure closer to actual health instead of cost alone. That one change removed about 84 percent of the measured bias (Obermeyer et al. 2019). The fix was that close at hand, once someone could see the problem.

And that is what makes the case hard, not easy. The algorithm had done exactly what it was built to do, and done it well. The result was unjust anyway. So where, exactly, did the wrong live — in the math, in the label, or in the world the data described? The argument over that question outlived the algorithm, and it is not finished.

The Argument the Audit Started

Algorithmic fairness already had a central puzzle before Optum: a model can use no forbidden variable — not race, nor anything close to it — and still reproduce the bias of the world it learned from. The Optum audit turned that abstraction into a measured number, on roughly 200 million patients. What the people who studied it then disagreed about was where the wrong actually lived: in the data, the proxy, the way the score was used, or the assumption that a clean fix was available at all.

“Accurate and colorblind”: the designers’ defense

Begin with the position the developers — and much of the industry — actually held. It deserves its strongest form, because most of us walk in believing some version of it. The algorithm used no race data. And at its assigned task, it was accurate: it predicted future costs about equally well for Black and white patients alike. By the standard tests the field used, the tool passed.

The Colorblind Accuracy Argument

1. A prediction tool is fair if it ignores race and predicts its target equally well for every group.
2. This algorithm excluded race and predicted costs equally well for Black and white patients.
3. Therefore, the algorithm was fair.

Premise 2 is true — the study itself confirmed it. The whole dispute lives in premise 1, and specifically in two words: its target. The premise assumes that hitting the target is what fairness means. It never asks whether the target was the right one. That is exactly where the study’s authors aimed their reply.

“The label is where the values hide”: the auditors’ reply

Obermeyer’s team answered that fairness is not a property of the math. It is a property of the match between the math and the job. The job was to route scarce help to the sickest patients. The tool was never asked to do that job. It was asked to predict spending — and spending, in America, tracks race for reasons that have nothing to do with health.

The Warped Proxy Argument

1. A tool that allocates care is fair only if it serves the allocation’s real goal: helping the sickest.
2. The tool predicted cost, and cost understates sickness for Black patients, because less is spent on them at every level of illness.
3. A tool that accurately predicts a warped proxy reproduces the warp in every decision built on it.
4. Therefore, the algorithm was unfair — because it was accurate, not in spite of it.

On this view, choosing the label was never a technical decision. It was an ethical decision wearing technical clothes, made in a room where no one thought ethics was on the agenda. The most chilling detail is how the bias survived inspection: every internal quality check compared predictions to cost. The yardstick and the flaw were the same object. You cannot audit your way out of a proxy problem with the proxy.

But the reply has a soft spot, and the defenders of algorithms pressed on it at once. The warp in the proxy came from the world, not from the code. If the data record an unequal system, is the algorithm really the thing to blame — or is it just the messenger that finally got caught carrying the message?

“Biased algorithms are easier to fix than biased people”: the optimists’ counter

The most forceful version of that counter came from inside the study itself. Two months after the paper, Sendhil Mullainathan — one of its authors — published an essay whose title carried the whole argument: “Biased Algorithms Are Easier to Fix Than Biased People” (Mullainathan 2019). Yes, the algorithm absorbed the world’s bias. But notice what happened next. The bias was found, because an algorithm’s behavior can be tested on thousands of cases. It was measured, down to the percentage point. And it was mostly repaired, by changing one label — within months.

The Fixable Machine Argument

1. Bias in human judgment is diffuse, deniable, and very hard to change.
2. Bias in an algorithm lives in specific, inspectable choices — above all, the label.
3. This algorithm’s bias was found, measured, and 84 percent repaired by changing the label.
4. Therefore, audited and regulated algorithms can reduce injustice, compared with the human status quo.

Consider the human baseline honestly. Decades of studies show clinicians undertreating Black patients’ pain. No one has ever cut that bias by 84 percent with any intervention, let alone in one quarter. The machine, on this view, did medicine a favor: it gathered a thousand scattered human biases into one place where they could finally be seen and operated on.

The soft spot here is the word audited. The fix happened because outside researchers got something the industry almost never grants: access to the scores and the truth, side by side. Premise 3 quietly assumes someone gets to look. That assumption is where the next speaker took the argument.

“The bug-fix is not the lesson”: Benjamin’s reframing

Ruha Benjamin, a Princeton sociologist, wrote the commentary that ran beside the study in the same issue of Science (Benjamin 2019). She did not dispute the finding or oppose the fix. She challenged the frame around them. Treating the case as a bug that engineers patched, she argued, hides the questions that produced the bug in the first place. Who decided that cost was the objective? Who was in the room? Who was not? Colorblind design was not a neutral default that happened to fail — it was a choice to not look, and the harm it concealed fell exactly where American health care’s harms have always fallen.

Benjamin’s reframe moves the problem from engineering to governance. The lesson of the case is not “pick better labels.” It is that label-picking is an exercise of power, and power needs structure: mandatory audits, impact assessments before deployment, regulators with authority, and seats at the design table for the communities the tool will score. On her view, Mullainathan’s optimism is true but conditional — machines are easier to fix if anyone with power is obliged to look for the flaw. The Optum case had volunteer auditors and a cooperative company. Justice should not have to depend on that kind of luck.

So the exchange narrowed to a real question: is the fixable machine the normal case, or was it a lucky one? The years since 2019 have been running that experiment.

The move that hasn’t ended: what the years since have shown

The case did not close. It split into three live threads, and they pull in different directions.

The lesson spread — into medicine’s oldest tools. The audit taught a generation to ask what a number actually measures, and the question escaped AI entirely. In 2020, physicians writing in the New England Journal of Medicine catalogued the “race corrections” built into everyday clinical formulas — kidney function, lung capacity, birth risk — adjustments that quietly moved Black patients away from treatment (Vyas et al. 2020). The kidney equation was rebuilt without race in 2021. Federal rules followed: in 2024, health regulators made hospitals responsible for discrimination risk in the decision-support tools they use, effective mid-2025. Score one for the optimists: visibility produced repair, at scale, beyond anything the authors predicted.

But the technology changed underneath the debate. The audit method that caught Optum needs two things: one label to inspect, and ground truth to compare it against. The generative AI now entering medicine has neither. When Stanford researchers tested large language models in 2023, the chatbots repeated debunked race-based medicine — false claims about Black patients’ kidneys, skin, and pain thresholds — woven into fluent clinical answers (Omiye et al. 2023). There is no label to relabel. The bias is not in a column of the spreadsheet; it is dissolved into the model’s whole way of talking.

The Vanishing Label Argument

1. The Optum bias was findable and fixable because one label could be checked against the truth and replaced.
2. Generative models have no single label, and often no ground truth to check their answers against.
3. An injustice that cannot be located cannot be repaired by relabeling.
4. Therefore, the “easier to fix” defense weakens just as medicine adopts the technology it fits least.

And the politics moved — backwards. The governance Benjamin called for advanced, then collapsed. Colorado passed the country’s most ambitious algorithmic-discrimination law in 2024, covering health care by name. It never took effect. Lawmakers delayed it from February to June 2026, and then, in May 2026, repealed and replaced it outright (SB 26-189) (Skadden, Arps, Slate, Meagher and Flom 2026). The replacement drops the three provisions that gave the original its teeth: risk-management programs, impact assessments, and — most pointedly for this case — the duty to use reasonable care to prevent algorithmic discrimination. What remains is a transparency law: tell people an automated system was used, disclose adverse outcomes within thirty days, correct bad data, offer human review. Compliance begins January 1, 2027.

Read that against the Optum facts. A pure disclosure regime would have required UnitedHealth to say an algorithm helped decide this. It would not have required anyone to notice that the algorithm’s target variable was cost rather than illness — the very thing Obermeyer’s team had to obtain proprietary data to discover. Meanwhile federal oversight of clinical AI loosened through 2025–26 in the name of innovation, and the original sore spot remains untouched: no law yet guarantees the next Obermeyer the data access that made the first audit possible. The fix was real. The system that produced the flaw — and the luck the fix required — look much the same.

So the question the case leaves open has widened rather than settled. The Optum story had an address for its injustice: one label, one fix, 84 percent. If the same injustice now lives everywhere and nowhere in a model no one can relabel — was the 2019 case the model for governing medical AI, or merely the last one easy enough to solve?

Discussion Questions

1. Reconstruct the Colorblind Accuracy Argument and the Warped Proxy Argument. The two sides agree on the facts. Identify the single premise they actually dispute, and say which side you find more convincing — and why.
2. The developers excluded race from the algorithm in order to be fair — and that choice helped hide the bias for years. Is colorblindness in a decision system a moral virtue, a neutral default, or a refusal to look? Defend your answer using the specifics of this case.
3. You are a nurse care-coordinator. Your daily call list comes from a risk score, and you notice it rarely includes the patients you know are struggling most. What do you do that week — and what would your hospital need to provide for your concern to actually be checkable?
4. Mullainathan says biased algorithms are easier to fix than biased people. Benjamin says the fix depended on power and luck that the system does not guarantee. Whose position is better supported by what has happened since 2019? Use at least one documented development from the final section.
5. A hospital’s new chatbot gives clinically biased advice, but there is no risk score, no label, and no ground-truth file to audit. Design the 2026 version of the Obermeyer audit — what would you compare against what? — or argue that no such audit is possible and say what should replace it.

Further Reading

• Benjamin, Ruha. Race After Technology: Abolitionist Tools for the New Jim Code. Polity, 2019. (The book-length version of the reframing move.)
• Mullainathan, Sendhil. “Biased Algorithms Are Easier to Fix Than Biased People.” The New York Times, December 6, 2019.
• Obermeyer, Ziad, et al. Algorithmic Bias Playbook. Chicago Booth Center for Applied Artificial Intelligence, 2021. (The audit, turned into a how-to guide for health systems.)

References

Benjamin, Ruha. 2019. “Assessing Risk, Automating Racism.” Science 366 (6464): 421–22. https://doi.org/10.1126/science.aaz3873.

Mullainathan, Sendhil. 2019. “Biased Algorithms Are Easier to Fix Than Biased People.” The New York Times, December. https://www.nytimes.com/2019/12/06/business/algorithm-bias-fix.html.

Obermeyer, Ziad, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. “Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations.” Science 366 (6464): 447–53. https://doi.org/10.1126/science.aax2342.

Omiye, Jesutofunmi A., Jenna C. Lester, Simon Spichak, Veronica Rotemberg, and Roxana Daneshjou. 2023. “Large Language Models Propagate Race-Based Medicine.” Npj Digital Medicine 6: 195. https://doi.org/10.1038/s41746-023-00939-z.

Skadden, Arps, Slate, Meagher and Flom. 2026. Colorado Repeals and Replaces Its AI Act. Client alert, June 2026. https://www.skadden.com/insights/publications/2026/06/colorado-repeals-and-replaces-its-ai-act.

Vyas, Darshali A., Leo G. Eisenstein, and David S. Jones. 2020. “Hidden in Plain Sight — Reconsidering the Use of Race Correction in Clinical Algorithms.” New England Journal of Medicine 383 (9): 874–82. https://doi.org/10.1056/NEJMms2004740.