How Do We Handle Rare Events in Synthetic Data?

In my article on utility, I described a model trained on synthetic hospital records that performed well overall but the patients with the longest and least typical stays were not modelled well. I mention this because I see the same issue fairly often, where the bulk of the distribution is reproduced well, but the edge cases are not. I touch on this issue in my blog on model collapse as well, and it is a common problem in synthetic data generation. The edge cases are often rare events, and they are difficult for a generator to learn from.

In the paper described above, Bietsch and colleagues used New York inpatient discharge records as the raw data, and found that diagnoses and procedures were the hardest variables for the generator to reproduce. Across more than 230 different diagnoses and 190 different procedures, the categories connected to only a small number of patients were not learned at all. In other words, they were rare events.

In this blog, I want to work through the methods that people use to generate rare events in the data output, and I’ll also go through what each method does as well as some of the limitations of applying these methods.

So what kind of rare events are we dealing with?

We often use the term rare events to describe two quite different situations, so it’s worth separating them before we go into the methods. This distinction has been termed ‘absolute rarity’, where there are few examples (scarce), and ‘relative rarity’, where a group is not rare in itself but is rare compared to the data around it (a data imbalance).

These two are caused by different underlying problems. With scarcity there are too few examples for the generator to learn from, so it either copies the ones it has too closely, or invents relationships that weren’t there in the original data. With imbalance the generator learns the majority but smooths the rare cases away. Which of the two you are dealing with is what determines which method will help.

When the data is scarce

With scarce data, the problem is that the generator has very little information to learn from. Where there is enough data to train on, the first option is to use a generator that can work with a small cohort, however if there isn’t enough data to train a generator reliably, resampling methods can be used.

1. Generating from a small cohort

Let’s look at a simple worked example for generating from a small cohort. Imagine we start with 200 expert-validated uveitis records, including six patients with a rare cause. If we generated a dataset of 10,000 records while preserving the 3% frequency, we would end up with around 300 rare records. The important limitation, however, is that all 300 are still being generated from information contained in those original six patients, so increasing the row count does not necessarily improve how well the rare group is represented.

Resampling as a fallback

If the cohort is too small to train a generator reliably, you might look to use a resampling method such as SMOTE or random oversampling. These can create more examples from the rare cases you already have, but they cannot add information that was never present in the original cohort. There is a practical floor here too, because with very few examples you are repeatedly drawing from the same limited information. I come back to how SMOTE works in the next section.

When the data is imbalanced

With imbalance the problem is different, as the generator has plenty of data overall, but learns the majority class much better than the rare one. A common first step is to rebalance the training data using methods such as SMOTE or random oversampling. Another option is to condition the generator on the rare class, using models such as CTGAN or MTGAN. A comparison of these approaches is included in the table below.

Resampling with SMOTE

The most common approach people take for resampling is the Synthetic Minority Over-sampling Technique (SMOTE), which takes one of your rare patients and one of its nearest neighbours, and builds a new record somewhere between the two. Let’s look at a worked example.

As you can see, Patient C is generated as a hybrid of A and B, which is also the main limitation of this method. For each variable, the new value sits somewhere between the two patients used to create it. In this example Patient C can therefore never be younger than 40 or older than 60. SMOTE gives you more rows, but it is still working with the information already present in the rare cases, so any imbalance can be carried straight across into the newly generated data.

Conditional generation

Conditional generation takes a different approach by making the rare class an explicit input to the generator. This means you can deliberately ask the model to generate more records with the outcome or characteristic you care about, rather than relying on it to appear at its original frequency. The example below shows how this differs from normal generation, and from a more targeted approach such as BayesBoost.

• In column A, we have 10,000 real patients and 50 of them (0.5%) have the rare outcome we care about, e.g. a serious adverse reaction to a drug (ADR).
• In column B, a normal generator copies that frequency, so the rate of ADRs is reproduced at 0.5%.
• In column C, conditioning is used to deliberately generate more records with the rare outcome such as ADRs across the whole dataset.

BayesBoost

A method known as BayesBoost takes a more targeted approach by first identifying which groups are poorly represented, then generating additional records for those groups. This means it can target imbalance within a subgroup, rather than simply increasing the rare class across the dataset as a whole.

• In column D in the table above, the number of ADR cases is still increased to 2,000, but the proportion from the underrepresented ethnic group rises from 6% to 30%.

What these examples show is that simply generating more rare cases is not always enough. You also need to look at which patients those new records represent, and whether the imbalance you were trying to fix is actually any better.

Conclusion

Rare events are difficult for synthetic data for two different reasons. Sometimes there simply are not enough examples for a generator to learn from, and sometimes the rare cases are overwhelmed by the majority of the dataset. Those problems look similar in the output, but they need different solutions.

As we’ve discussed, the practical starting point is to work out which kind of rarity you are dealing with, and the simple way to think about it is

• If the data is too scarce, you need a method that can learn from a small cohort,
• If the data is imbalanced, then resampling, conditioning or a more targeted approach can help.

Whichever method you choose, the rare group needs to be checked directly against the real data, because an overall utility score can still look good while the subgroups or events you care about most are poorly represented.

Further reading

If you want to read more about the ideas in this article, I recommend starting with my article on measuring utility, which looks at why synthetic data can perform well overall while failing on smaller groups. My articles on representativeness and fairness and bias amplification are also useful background for understanding why rare groups can be lost during generation.

This blog is part of my series on synthetic data, which is worth checking out if you want to understand the different methods and applications of synthetic data generation.