What Do We Mean by Bias in Health Data Research?

Bias is a word that gets thrown around a lot in research and increasingly in the world of LLMs and AI. But what does it actually mean? In this blog post, I am going to set out what bias means within the world of health data research - focusing on the type of research that I tend to do which is electronic health record (EHR) research. This is intended as a reference post, so that later posts and courses can point back here rather than redefining the term each time. I use these definitions in my codelist work and across the synthetic data series. Finally this is my personal take on the topic delivered in plain English, via a blog post. If you want a more formal and technical treatment of bias in health data research, I recommend this paper by my good friend Professor Rohini Mathur and colleagues: Promises and pitfalls of electronic health record analysis that works through an example of bias in EHR research in a very clear way.

Bias is a systematic error, not a random one

We know that all data has some degree of error in it. Data records events that happen in the real world, and the recording process is imperfect. Mistakes get made by humans or computers. Sometimes data is just not collected. But there is a difference between random error and systematic error, and that difference is what we mean by bias.

Let’s start with random error. This can be thought of as noise. It is scattered around your dataset with no particular pattern. If you take a sample of the population, some people will get included, and others will not. It ends up scattering your measurements around the true value. The prevailing thought here is that as your sample size grows, the noise averages out and you get closer to the truth. This is good news for those of us who work with EHR data since the datasets tend to be enormous.

Bias is different. It is a systematic error that pushes your estimate in a particular direction. It is not random. It is some systematic and inherent property of the data collection process or analytical method that skews your results in one direction. This means that as your sample size grows, the bias does not average out. In fact, it could make it a whole lot worse. A bigger sample of the wrong people gives you a more precise wrong answer, not a right one.

Every dataset is biased in some direction

My personal view is that every dataset contains some sort of inherent bias. We can start very simply. If we are using a dataset drawn from people who have attended their GP, we are missing the people who never attend their GP or who are not registered with a GP. We can imagine people for whom this is true, are they are likely to be different in some way from the people who do attend their GP. They might be healthier, younger, or have the means to afford private healthcare. Or they might have a chronic and serious health condition meaning that their place of care is usually a hospital rather than a GP. We can also imagine people who are not registered with a GP are likely to be different in some way from those who are registered. They might be more transient, homeless, be international students or people who have recently moved to the country.

The data we have is a shadow or echo of the real world. It is the map, not the territory. It is a reflection of the people who have come into contact with the healthcare system, and it is shaped by the way that system works. It is not a census of the population.

For me, the key question is not whether the data is biased, but how it is biased and what the implications are for my specific research question. The useful questions are to ask are:

• Biased in which sense?
• Against whom?
• Does that particular skew matter for what you are going to do with the data? I.e. are we expecting it to impact the results of our analysis, and if so, how?
• How can we mitigate the bias or at least try to quantify it so that we can be transparent about the limitations of our work?

Answering those requires knowing which type of bias you are dealing with, because they enter the data at different points and behave differently.

A field guide to the types of bias in health data

The table below sets out the types of bias you are most likely to meet in health data, what each one means, and an example of how it might arise in practice.

There are other types of bias that are less common in health data research, but the 7 above are the ones I see most often. I will go through each of them in turn below.

Selection bias

Selection bias is about who is in the dataset in the first place. The people in the data are not a fair sample of the people the study is meant to be about. Hospital records only contain patients who reached hospital, so any pattern you find is a pattern about hospitalised patients, not the general population. The people missing from the data are not absent by chance. They are absent for reasons that are often tied to the very thing being studied, whether that is their health, their access to care, or their circumstances.

The same logic operates outside health too. In credit scoring, the training data only contains people who were previously granted credit, so a model built on it knows nothing about the people who were declined or did even apply because they thought they would be turned down. The gap is shaped by the outcome you care about.

Non-response bias

Non-response bias is a specific relative of selection bias that shows up in surveys and studies people opt into. The people who agree to take part differ systematically from those who decline. A Finnish research programme built an entire validation methodology around the fact that survey non-participants differ from participants in hospitalisation rates, education, and mortality.

Imagine you are using data from a biobank where the original data was collected via an appointment that is long, unpaid, held during the working day, and only available in a major city. Every one of those features filters the sample in the same direction. Shift workers, carers, people who cannot take a day off, working parents and anyone far from the site are not responding to invitations to join and you end up with a cohort skewed towards healthy, urban, comfortably-off retirees. Nobody was excluded on purpose, but the practical details of how the data was collected did the excluding for you.

This really matters because if you treat your participants as if they were the whole population, you might end up with findings that inform policy or design an intervention for the whole population, when in fact they are only valid for a very specific and unrepresentative subset of it.

I’m sure you can see that non-response bias is a specific type of selection bias. The reason we care about it and give it its own name is that the mechanism is a bit different. With selection bias, the people who are missing never enter the dataset at all, whereas with non-response bias, they were invited or eligible, but did not take part.

Measurement bias

Measurement bias is different. The right people were in the dataset but the value or “measurement” recorded in their records (and hence our study) is wrong in some systematic way. Self-reported alcohol consumption is the classic example. People who are worried about their drinking or have problematic drinking patterns tend to under-report how much they drink. The same is true for smoking. The people who are most likely to be heavy smokers are also the people most likely to under-report their smoking status. This is a systematic error that skews the data in a particular direction. It is part of a wider group of “information bias” that includes recall bias, reporting bias, and others. The common thread is that the recorded value differs systematically from the true value, and this is a property of the data collection process rather than a property of the people in the dataset. In my experience measurement bias tends to be the case for things in society that we have deemed as “bad” or “unhealthy” and that carry a social stigma.

Measurement bias can also arise from the way the data is collected, and be really hard to track down. This is where computer design and human factors come into play. For example, a poorly designed form or interface can lead to systematic mis-recording of values. If a dropdown menu defaults to “No” for a health condition, clinicians may be more likely to leave it at that default rather than actively selecting “Yes” when appropriate. Since we know that computer systems tend to be bought in at regional levels, you can end up with a situation where one region’s data is systematically different from another’s, not because the patients are different. This is a form of measurement bias that is not about the patient, but about the system they are in.

Informative missingness

Informative missingness is about the gaps, and specifically about gaps that are not random. Unfortunately humans have their own biases and this affects the way they record data. You can imagine that there might be clinicians out there who are more likely to record a patient’s blood pressure if they are concerned about it or if the patient is older. This means that the missing values are not missing at random.

Representation bias

Representation bias is about whether subgroups are adequately represented in the data. A dataset can be biased against a group even if that group is present in the data, sampled fairly, and measured correctly. The problem is that the group is too small for its patterns to be characterised reliably. Rare diseases and minority groups are the usual cases.

Collider bias

Collider bias is the odd one out here. The biases above are all about the sample being the wrong sample. Collider bias is different. The sample can be perfect and you can still manufacture a relationship that does not exist in the real world, purely through what you “condition” on in the analysis.

A collider is a variable that is caused by two other variables. If you select or stratify on that shared effect, you open up a false association between its two causes, even when they are unrelated in the population. It is easier to think about a made up example to get your head around this, at least for me!

The classic example is studying only hospitalised patients. Say two conditions, a broken hand and gallstones, are unrelated in the general population, but each independently raises your chance of being admitted. Now if we only look only inside the hospital as our population, we end up with some weird results. A patient admitted without a broken hand is more likely to have been admitted because of the gallstones, and the other way round. The two now look negatively correlated in your data - that having a broken hand makes it less likely you will have gallstones - which you can reason is for sure not true! Admission is the collider, and by restricting to admitted patients you conditioned on it, and ended up linking two unrelated conditions.

Survivorship bias

Often as EHR researchers, we only get access to data from a particular type of care, for example, national hospital records or GP surgeries. Often we don’t have access to the data from people who have died - which tends to go to the Office for National Statistics (ONS) datasets. This creates a survivorship bias in the data.

Say we are studying late-onset lymphoma in adults who received radiotherapy as children. Everyone in our data survived their childhood illness, survived the radiotherapy, and survived long enough to reach adulthood and show up in an adult record. We never see the children who did not, and they are not a random subset. They were plausibly the more severe cases. So any conclusion we draw is a conclusion about the survivors, and it may not hold for the group as a whole.

These biases exist before any model touches the data

This post is designed to be an introduction to some of the common types of bias that you will meet in health data research. This is not an exhaustive list, but it is a good starting point. If I could get you to take one takeaway, it would be that pretty much all of these biases are baked into the data before you start any analysis. They are properties of the data collection process, the study question (and hence design and population chosen) and the way the data is recorded. They are not artefacts of a particular method or model. In fact these biases affect a conventional statistical study exactly the same way as a fancy machine learning model. The biases are in the data, not the model.

Further reading

I have written a few other posts that are relevant to this topic, and I have linked to them below:

• What is an EHR? - A primer on what electronic health records are, how they are structured, and how they are used in research.
• What is synthetic data and why does it matter? - A primer on what synthetic data is, how it is generated, and why it is important in health data research. My next post in the synthetic data series will cover bias in synthetic data, and this post is a good foundation for that.
• What is a codelist? - A primer on what codelists are, how they are used in health data research, and why they are important. Codelists are a place where additional bias can creep in so its worth understanding them in the context of this post.