Creating High-Quality Synthetic Data Even When Data Is Scarce
Creating High-Quality Synthetic Data Even When Data Is Scarce
“If there isn’t enough data to train an AI, couldn’t we just use a generative AI to create the data?”
“Synthetic data”—artificially generated images or text used for training—stems from this idea. For example, it involves creating photos of defective products that rarely occur in factories or medical images that are difficult to obtain. This approach is widely considered because it reduces the cost of collecting the necessary data directly.
However, this raises another question. What if there isn’t enough data to train the model that will generate that data? For example, suppose we want to create a composite image because we don’t have enough defective photos, but we actually lack the data needed to train the model to generate those defective photos in the first place.
To solve this problem, we need to take a closer look at the phrase “insufficient data.” Even if there are few samples available for immediate training, other forms of information may still be available—such as the knowledge a model has acquired from training on normal images, the model’s scores for the limited data, or the grammatical rules identified by experts.
Three studies—on manufacturing processes, brain MRI scans, and low-resource languages—demonstrate how information other than the data (samples) can be transformed into information that aids learning. As we explore these studies, the initial question evolves further: where should we obtain the missing information, and how can we verify that the data created in this way is actually helpful?
1. Identifying Clues to Defects in Good Products
Let’s imagine we’re building an AI to detect defective products. We need to collect photos of good products and defective products and train a model to distinguish between the two. If the factory is running smoothly, it’s easy to collect photos of good products.
On the other hand, defects occur rarely, and they come in all shapes and sizes. Since there aren’t enough photos of defects, it’s difficult to train the model effectively; and even if we try to fine-tune the model to generate defect images using a generative model, we run into the same problem: a lack of those very images.
DeltaDeno, presented at ECCV 2026, drew inspiration from pre-trained image generation models. It uses two types of sentences—one describing the normal state and one describing the defective state—alongside images of normal products. For example, “a photo of a coconut” and “a photo of a cracked coconut,” where only the description of the defect differs.
The key is to track the differences in the image generation process under these two conditions. By identifying which parts change and to what extent at each stage of generation, we can pinpoint the areas that respond to the defect description.
We created a “mask” to indicate where to modify this area, and generated defects in that specific region while preserving the surrounding areas. As a result, we obtained both the defective image and the mask indicating the defect locations, and by using these to train the model, it demonstrated superior performance compared to existing models.
In this case, the resource that fills in the missing information is the visual knowledge the generative model acquired during pre-training. By linking images of normal products with sentences describing defects, the model can generate new defect data without the need for additional training on actual defect examples. The research team evaluated not only the quality of the generated results but also the actual defect detection performance of the model trained using them.
This approach demonstrates the potential to extract necessary information from models trained on different data. Now, let’s take it a step further. ==If there are even a few real-world examples, couldn’t we use them to adjust the synthetic data so that it’s generated in a more useful way?==
2. If we use the limited amount of actual data to guide the synthesis process,
When it comes to brain MRIs, simply acquiring the images isn’t enough to complete the preparation. To train a segmentation model—which identifies which parts of the image correspond to specific brain tissues—you also need labels that color-code each region.
Creating these with precision requires the expertise and time of specialists who are well-versed in the brain. There is also the challenge that the same tissue can appear differently depending on the imaging conditions. Therefore, researchers have studied ways to make the most of the few anatomically labeled maps they have—even though they couldn’t produce many due to the high cost.
Learn2Synth, presented at ICCV 2025, takes as its starting point a method of generating synthetic images by modifying anatomical label maps. It enhances these maps—which show the various regions of the brain—by assigning different brightness levels and image characteristics to them. Since the process begins with well-defined ground-truth labels, these can be reused as the ground truth for the newly augmented images.
The key question here is how to modify the synthetic images to aid the learning process. Learn2Synth derives the answer from a small amount of real labeled data.
First, we train the segmentation model using synthetic images. Next, we check how much the model’s predictions differ from the ground truth in actual MRI images, and use that error to adjust the augmentation method for the synthetic images.
Rather than rule-based augmentation, we use learnable augmentation to teach the model how to adjust its output to produce high-quality synthetic images. In this architecture, the segmentation model is trained on synthetic data, while real-world data serves as feedback to improve the training data.
To use an analogy, it’s similar to having students solve practice problems (synthetic videos), then identifying the weaknesses revealed in real-world problems (actual videos), and adjusting the structure of the next set of practice problems accordingly. In a sense, the few real-world examples help guide the direction of the many synthetic examples.
3. How helpful are sentences written according to the rules of grammar?
A study that directly addressed this question is “Is linguistically-motivated data augmentation worth it?” presented at ACL 2025. The researchers compared the effectiveness of data augmentation in translation and in the task of attaching morphological semantic and grammatical information to two Native American languages: Uspanteko and Arapaho.
The methods compared were a simple approach—involving randomly inserting, deleting, or repeating words—and an augmented method that reflected the grammar of each language. Grammar-based methods included changing the perfect and imperfect forms of Uspanteko verbs, inserting conjunctions and interjections at the beginning of sentences, and altering the permissible word order in Arapaho.
The results varied depending on the method and task. Inserting conjunctions and interjections proved more effective than randomly inserting words in the translation. However, improvements achieved by altering verb forms (perfect vs. imperfect) were smaller than those achieved by simply augmenting the text with repeated words, and changing the word order in Arapaho actually reduced performance.
Even though changing the word order is grammatically possible, why did performance decline? The researchers cited the discrepancy with actual usage patterns as the reason. Even in languages with flexible word order, people may have preferred arrangements; the interpretation is that even if many grammatically correct sentences are generated, they may not aid learning if they differ significantly from expressions actually used in practice.
This difference becomes even more significant when you consider the effort required to generate the data. In this study, the first author—who has a background in linguistics—spent over a year, or approximately 200 hours, studying grammar books and dictionaries for both languages. The authors noted that, given the modest improvement in performance, it might have been more productive to devote this effort to actual data collection and annotation instead.
Expert knowledge is a vital resource for filling in data gaps. However, we must separately verify whether the results derived from that knowledge are actually useful for training. This is because there is a gap between knowing the rules for constructing sentences and determining which sentences are helpful for training the model.
Before creating data, ask what’s missing
The three studies drew on scarce information from different sources. DeltaDeno utilized the visual knowledge of pre-trained models, Learn2Synth relied on anatomical structures and feedback from real-world data, and the low-resource language study leveraged knowledge from grammars and dictionaries. What they have in common is that they transformed resources that are difficult to incorporate directly into training into new learning examples.
So, if you encounter a project with insufficient data, you first need to pinpoint exactly what kind of data is missing. Is it simply that there are no rare cases? Is it that the data exists but there are no ground truth labels? Or is it that specific expressions or situations aren’t sufficiently represented? You need to know what’s missing in order to decide where to seek help—whether from other models, existing data, or expert knowledge.
Next, we need to verify the effectiveness of the data created in this way in actual projects. Part of this assessment involves determining whether complex synthesis methods are superior to simple augmentation, and how they compare to spending the same amount of time and effort on acquiring real-world data.
Let’s return to the original question. When there isn’t even enough data to generate synthetic data, the starting point is to reexamine the location and use of existing information.
What is missing, what information can fill that gap, and how can we verify its effectiveness? When we ask these three questions together, synthetic data can bring us one step closer to solving the problem at hand.
Tags
- #artificial-intelligence
- #ai-papers

















