Oh, we don't need that data anymore.
Not More Data, But Better Data: Active Learning and Data Curation
We’re inundated with data.
In 2020, OpenAI published a technical report titled “Scaling Laws for Neural Language Models,” which demonstrated that model performance improves as the model size, training data, and computational power increase. Since then, actual advancements in AI have also begun to demonstrate significantly better performance than before by training on vast amounts of internet-scale data.
However, recently, it has become difficult to simply say, “The more data, the better.” The reason is simple: there is simply too much data now. While the amount of data that can be collected is growing rapidly, the number of labels that humans can assign and the computational resources available for training models are limited.
So, when not all data is available, which data should we choose?
Deciding which data to spend human effort annotating—and which data, out of the vast amount available, to actually use for training—has become a critical issue. In fact, this question has long been a subject of research in machine learning. Technologies that have been researched in various areas—from active learning to data curation—are now converging into a single industrial trend known as the “Data Engine”.
To reduce labeling costs: Active Learning
One of the long-standing challenges in AI training has been the cost of labeling. While it is possible to collect millions of images, manually annotating each one with the correct labels is no easy task. The cost is even higher for data that requires expert judgment, such as medical images.
So, among the data points that don’t yet have answers, which ones should we use to create answer keys for and train the model on first?
This is how Active Learning came about. Just as people actively learn things they don’t know well, this approach proposes that we identify what data the model needs and label those samples accordingly. So, instead of continuing to label samples the model already knows well, this approach involves selecting samples that confuse the model or that best represent the overall data distribution.
So how can we tell which samples are difficult for the model?
The simplest method is to look at the model’s predictions. If it predicts a specific class with 99% probability, we can consider it to be relatively confident. Conversely, if the probabilities for multiple classes are similar, the sample is considered a case where the model is uncertain, so we label it and use it to train the model.
However, if only difficult data points are selected, similar samples may be chosen repeatedly. Therefore, we also use methods to preserve the diversity of the entire dataset. Core-set selects samples that represent different regions in the feature space, while Herding adds data in such a way that the average characteristics of the selected samples resemble those of the entire dataset.
These methods are put to good use in real-world industries. When data continues to accumulate but budget constraints limit the amount that can be labeled, these methods are used to prioritize which samples to train on first and label them accordingly. Active Learning is employed in this way to achieve the greatest possible performance improvement within limited labeling resources.
Should we train on all the data? Data Curation
Simply reducing labeling costs does not solve all data problems. In this case, the problem is the opposite: there is simply too much data.
We can collect billions of images and text documents from the internet. However, training a model on all this data requires massive amounts of GPU power and time. Furthermore, not all of this data contains new or useful information. There may be duplicates of the same images or documents, as well as low-quality data or data where images and descriptions are incorrectly matched.
So, what kind of data should we use for training? Data Curation addresses this question. It involves removing duplicate data, filtering out low-quality samples, and selecting data from the entire dataset that will be useful for training. Ultimately, it means focusing limited training resources on high-value data.
First, SemDeDup, released in 2023, focused on reducing duplication. Rather than simply identifying and removing identical files, it uses the model’s embeddings to identify data with similar content and features as semantic duplicates. This approach made it possible to significantly reduce training time while retaining most of the performance, even after removing about half of the data.
Once duplicates are removed, the question arises: “Which data actually helps with training?” DataComp prepared 12.8 billion image-text pairs and, while keeping the model and computational costs the same, varied only the selection of data used for training.
They used the CLIP score to evaluate how well images and descriptions matched, filtering out data with irrelevant pairings, and clustered image features to construct a training set consisting of data in the same cluster as the target domain. The model trained on this curated dataset—comprising approximately 10% of the total data—demonstrated significantly higher performance than a model trained on the entire dataset.
In the end, what matters isn’t doing more, but doing it better.
Active Learning and Data Curation originated from different problems. One considers where to allocate limited labeling costs, while the other considers which data to use given limited training costs. Ultimately, however, the questions posed by both areas of research are similar.
“What is the most valuable data for models right now?”
The reason this question has become important is that the scale of data handled by AI has grown. While collecting as much data as possible was crucial when data was scarce, the ability to decide what to keep and what to discard from that data is becoming increasingly important.
Ultimately, the competition for AI data boils down to who can extract more training signals from limited data and resources.
Tags
- #artificial-intelligence
- #ai-papers
- #active-learning
- #data-curation
- #scaling-laws
- #training-data
















