When data is scarce, where does the source material for synthetic data come from?
Sent Nov 5, 2026
Ways to solve problems by making full use of other information in environments where data is scarce
“If there isn’t enough data to train an AI, couldn’t we just use a generative AI to create the data?”
“Synthetic data”—artificially generated images or text used for training—stems from this idea. For example, it involves creating photos of defective products that rarely occur in factories or medical images that are difficult to obtain. This method is widely considered because it reduces the cost of collecting the necessary data directly.
But this raises another question. What if we need data to train the model that will generate that data? We want to synthesize defective photos because we don’t have enough of them, but we lack the data needed to train the model to generate them in the first place.
To solve this problem, we need to take a closer look at the phrase “insufficient data.” Even if there are few samples available for immediate training, other forms of information may still exist—such as the knowledge of a model already trained on normal images, the model’s scores for the limited data, or the grammatical rules identified by experts.
Three studies—on manufacturing processes, brain MRI scans, and low-resource languages—demonstrate how information other than the data (samples) can be transformed into information that aids learning. As we explore these studies, the initial question evolves further: where should we obtain the missing information, and how can we verify that the data created in this way is actually helpful?
1. Identifying Clues to Defects in Good Products
Let’s imagine we’re building an AI to detect defective products. We need to collect photos of both good and defective products and train a model to distinguish between them. If the factory is running smoothly, it’s easy to collect photos of good products. Defects, on the other hand, occur rarely and can take many different forms. With a shortage of defective product images, it becomes difficult to train the model effectively. Even if we try to further train the model using a generative model to create defective images, we run into the same problem: a lack of those very images.
DeltaDeno, presented at ECCV 2026, drew inspiration from pre-trained image generation models. It uses two types of sentences—one describing the normal state and one describing the defective state—alongside images of normal products. For example, “a photo of a coconut” and “a photo of a cracked coconut”—the only difference is the description of the defect.

The key lies in tracking the differences in the image generation process under these two conditions. By identifying which parts change and to what extent at each stage of generation, we can locate the areas that respond to the defect description. We turned these regions into a “mask” that indicates where to make modifications, and then introduced defects in those specific areas while preserving the surrounding features. As a result, we obtained both defective images and masks indicating the defect locations; by using these to train the model, we achieved performance superior to that of existing models.
In this case, the resource that fills in the missing information is the visual knowledge the generative model acquired during pre-training. By linking images of normal products with sentences describing defects, the model can generate new defect data without the need for additional training on actual defect examples. The research team evaluated not only the quality of the generated results but also the actual defect detection performance of the model trained using them.
This approach demonstrates the potential to extract useful information from models trained on different data. Now, let’s take it a step further. If we have even a few real-world examples, couldn’t we use them to fine-tune the synthetic data so that it’s generated in a more useful way?
2. If we use the limited amount of real data to guide the synthesis process,
When it comes to brain MRIs, simply acquiring the images isn’t enough to complete the preparation. To train a segmentation model—which identifies which parts of the image correspond to specific brain tissues—you also need labels that color-code each region. Creating these labels with precision requires the expertise and time of specialists who are well-versed in the brain. There is also the challenge that the same tissue can appear differently depending on the imaging conditions. Therefore, researchers have been studying ways to make the most of the limited anatomical label maps they have—even though they are expensive to produce and thus not widely available.

Learn2Synth, presented at ICCV 2025, takes as its starting point a method of generating synthetic images by modifying anatomical label maps. It enhances these maps—which show the various regions of the brain—by assigning them different brightness levels and image characteristics. Since the process begins with well-defined ground-truth labels, these can be reused as the ground truth for the newly augmented images.
The key question here is how to modify the synthetic images to aid the learning process. Learn2Synth derives the answer from a small amount of real labeled data.
First, we train the segmentation model using synthetic images. Next, we check how much the model’s predictions differ from the ground truth in real MRI data, and use that error to adjust the augmentation method for the synthetic images. Rather than using rule-based augmentation, we employ learnable augmentation to teach the model how to modify the images to produce high-quality synthetic images. The structure is such that the segmentation model is trained on synthetic data, while the real data serves as feedback to improve the training data.

To use an analogy, this is similar to having a model solve practice problems (synthetic images), identifying weaknesses revealed in real-world problems (actual images), and then adjusting the structure of the next set of practice problems accordingly. In a sense, the few real-world examples help guide the direction of the many synthetic examples. In the brain MRI experiment described in the paper, segmentation performance improved compared to basic synthetic learning, and the method demonstrated strong generalization performance, consistently performing well even under conditions where the contrast of the images differed from that during training.
At this point, the goal of synthetic data becomes a little clearer. When deciding what to create and how to create it, performance on real-world tasks should serve as the benchmark. So, what about the rules that experts know? In language research, if we create new sentences that strictly adhere to grammar rules, will they serve as good training data for language models?
3. How helpful are sentences written according to the rules of grammar?
A study that directly addressed this question is “Is linguistically-motivated data augmentation worth it?” presented at ACL 2025. The researchers compared the effectiveness of data augmentation in translation and in the task of attaching morphological semantic and grammatical information to words, using two Native American languages: Uspanteko and Arapaho.
The methods compared were a simple approach—involving randomly inserting, deleting, or repeating words—and an augmented method that reflected the grammar of each language. Grammar-based methods included changing the perfect and imperfect forms of Uspanteko verbs, inserting conjunctions and interjections at the beginning of sentences, and altering the permissible word order in Arapaho.

The results varied depending on the method and task. Inserting conjunctions and interjections proved more effective than randomly inserting words in the translation. However, improvements achieved by altering verb forms (perfect vs. imperfect) were smaller than those achieved by simply augmenting the text with repeated words, and changing the word order in Arapaho actually reduced performance.
Even though changing the word order is grammatically possible, why did performance decline? The researchers cited the discrepancy with actual usage patterns as the reason. Even in languages with flexible word order, people may have preferred arrangements; the interpretation is that even if many grammatically correct sentences are generated, they may not aid learning if they differ significantly from expressions actually used in practice.
This difference becomes even more significant when you consider the effort required to generate the data. In this study, the first author—who has a background in linguistics—spent over a year, or approximately 200 hours, studying grammar books and dictionaries for both languages. The authors noted that, given the modest improvement in performance, it might have been more productive to devote this effort to actual data collection and annotation instead.
Expert knowledge is a vital resource for filling in data gaps. However, we must separately verify whether the results derived from that knowledge are actually useful for training. This is because there is a gap between knowing the rules for constructing sentences and determining which sentences are helpful for training the model.
Before creating data, ask what’s missing
The three studies drew on scarce information from different sources. DeltaDeno utilized the visual knowledge of pre-trained models, Learn2Synth relied on anatomical structures and feedback from real-world data, and the low-resource language study leveraged knowledge from grammars and dictionaries. What they have in common is that they transformed resources that are difficult to incorporate directly into training into new learning examples.
At the same time, the last example clarifies the criteria for data synthesis. It is necessary to verify, separately, that the information is available and that the data generated from it is useful. Key criteria include whether defects appear natural, whether the images and labels correspond well, and whether the sentences are grammatically correct. In addition, we must ask whether the system is better at detecting actual defects, distinguishing brain tissue more accurately, and improving the quality of translations and annotations.
So, if you encounter a project with insufficient data, you first need to pinpoint exactly what’s missing. Is it that there are no rare cases? Is there data, but no clear answers? Or are specific expressions or situations not sufficiently represented? You need to know what’s missing in order to decide where to seek help—whether from other models, existing data, or expert knowledge. Next, you must verify the effectiveness of the data you’ve created in real-world tasks. Part of this assessment involves determining whether complex synthesis methods are better than simple augmentation, and how they compare to spending the same amount of time and effort on acquiring actual data.
Let’s return to the original question. When there isn’t even enough data to create synthetic data, the starting point is to reexamine the sources and uses of existing information. What is missing? What information can fill that gap? And how will we verify its effectiveness? By asking these three questions together, synthetic data can bring us one step closer to solving the problem at hand.