Test First with Fake Data Whose Answers You Know: Measuring Whether Fakes Resemble the Real Thing (2)

Even when you test an AI with synthetic data that has planted answers, scores get inflated if the fakes do not resemble the real thing. Part 2 draws on the inverse crime, domain randomization, the ISO image analysis standard and the Korean privacy regulator's synthetic data guide to show how to compare fakes and real data with the same yardstick, how to vary one condition at a time to find error sources, and an execution sequence that ends with real-world validation.

AXAI transformationValidationSynthetic data

If the test is easy, everyone is a top student. To trust scores on synthetic data, first measure whether the test resembles the real thing.

Can you trust the test?

Part 1 covered building samples with known answers and using them as a test. Once you have a test, scores come out right away. But a high score is no reason to relax yet. If the person who made the fakes and the person who built the AI were working from the same assumptions, the score may only mean the test was easy.

This part covers three principles for checking the test itself, the mistakes first-time teams often make, and an execution sequence that runs all the way to real-world validation.

One. First measure whether the fakes resemble the real thing

This is where the most common trap in synthetic data validation lies. If the assumptions used to create the fakes match the assumptions the AI expects, the AI passes the test easily.

The mathematics of inverse problems has a name for this. Using the same theoretical ingredients both to synthesize data and to solve it is called the inverse crime, and it is taught as something to avoid because the results look unjustifiably good. In business terms: if you test an inquiry classification AI with fake inquiries written only in clean sentences, an AI that expects clean sentences scores high. The typos, abbreviations and two-requests-in-one-sentence cases that fill real inquiries are missing from the test.

So before using fakes, measure real and fake data with the same yardstick and compare. For inquiries, count sentence length, typo rate and the share of messages mixing two or more requests, in exactly the same way for real and fake. For images, measure brightness distribution, blur and the share of overlapping objects the same way. If the numbers differ, fix the fakes before scoring.

Diagram showing that when the assumptions used to create fakes match the assumptions the AI expects, scores look unjustifiably good, and that to avoid this you should count sentence length, typo rate and the share of mixed requests in exactly the same way for real and fake inquiries
Concept diagram of the inverse crime and its remedy, using an inquiry classification AI as an example. Inverse crime concept: Wirgin, 2004

There is also a remedy from the opposite direction. Domain randomization (Tobin et al., 2017) from robot learning, rather than matching fakes to one version of reality, varies textures and lighting very widely so that reality looks like just one of many variations. In that study, an object localization model trained only on synthetic images worked in the real environment with accuracy of around 1.5 cm. Whichever approach you take, the question is the same: does this test resemble the real problem?

Two. Vary one condition at a time to find what causes error

The real value of data with known answers is not a single score but that it tells you what drags the score down. In field data, conditions change all at once, so causes cannot be separated, but with synthetic data you can change conditions one at a time.

The international standard for measuring particle size from static images, ISO 13322-1, directly recommends this kind of experiment. Measuring touching particles as one creates error, and image analysis alone often cannot detect it, so the standard says to check the effect by increasing and decreasing the number of particles per image. In effect, the standard writes down "change the conditions and measure the effect."

The same approach works in business. For document recognition, build separate sets that change only tilt, only resolution, or only stamp smudging, and score each. Once you can see which condition makes the score drop sharply, you can tell whether the fix belongs in the model, the capture method or the input format.

Table-style diagram of three synthetic sets that change only tilt, only resolution or only stamp smudging, each changing one condition and holding the other two fixed
An example of designing synthetic sets for testing a document recognition AI by changing one condition at a time. Measuring the effect by varying conditions is also recommended by ISO 13322-1.

The same applies when measuring improvements. Every time you change a prompt or fix a rule, re-score on the same test. If you pick values by eye, you tend to end up with settings that only look good on a handful of examples. That is the only way to know whether a change really made things better.

Three. A synthetic score is not a real score

Synthetic data validation does not replace real-world validation. A 15% error on fakes means "15% on fakes that resemble the real thing," not 15% in the field. Reports should state the two numbers separately.

Korean guidance on synthetic data stresses the same order. In December 2024, the Personal Information Protection Commission (PIPC), Korea's data privacy regulator, issued a guide to generating and using synthetic data (in Korean) that sets out five stages: preparation, generation, safety and utility verification, evaluation by a review committee, and use and management. Its main purpose is privacy protection, but its requirement to verify separately how well the generated data keeps the usefulness of the original applies equally to synthetic data for AI testing.

The realistic order is this. Use synthetic data to find and fix failure types and causes first, then do a final check with a small number of real samples whose answers you know. Making 100 real samples is expensive, but if synthetic data has already filtered out the problems, you can get much more out of those 100.

Three common mistakes

First, making fakes that are too clean. Fakes that are easy to make, such as inquiries with no typos, neatly captured documents and parts placed apart from each other, are usually easy for the AI too. That score is only an upper bound. Pick a few of the most troublesome cases from the field and deliberately build their characteristics into the fakes.

Second, looking at a single score. An average accuracy of 90% does not tell you what was missed. There may be no misses but many cases where two items were merged into one, or many cases of finding things that are not there. Count hits, misses, wrong merges and false finds separately.

Count separatelyMeaning
HitsResults that match the answer sheet
MissesItems on the answer sheet that were not found
Wrong mergesTwo items merged into one
False findsItems found that are not on the answer sheet

Third, fixing and evaluating on the same test. If you adjust rules while looking at the test, you end up with rules that fit only that test. Rules that filter on the very features used to create the fakes, in particular, will naturally look best on those fakes. Keep the test you use for fixing separate from the test you use for evaluation.

Execution sequence

If this is your first time, we recommend the following order.

  1. Define the answers. Write down what must be got right and in what format the answer data will be kept
  2. Make the fakes. Measure conditions in real data, such as noise, lighting and typo rates, and reflect them
  3. Measure the resemblance. Measure real and fake with the same yardstick, compare the statistics, and fix the fakes if they differ
  4. Score. Count hits, misses and wrong merges separately
  5. Vary the conditions. Change one at a time to find the conditions that inflate error
  6. Fix and re-score. Keep a separate evaluation test and compare before and after improvement
  7. Confirm with real samples. Do a final check with a small number of real samples whose answers you know, and record the gap between the synthetic and real scores

When you move from pilot to production, this test becomes your regression suite as is. Running the same test every time you change the model or prompt stops performance from quietly slipping.

The test stays useful in operation too. Just as the same wine was secretly mixed into the judging, regularly mix a few samples with known answers into the real workflow. For example, put five test quotes in with the other documents each week and check the results. When input formats change or camera lighting ages and performance gradually drops, these samples will tell you first. Human review can be checked the same way. Which metrics to use for operational performance is covered further in the measuring AI results post.

In summary

First check whether the test resembles the real problem, vary conditions one at a time to find causes, and finish by confirming with real samples. The same AI can be a top student on an easy test and fail a hard one. To trust the score, you first have to be able to trust the test.

I wrote up a hands-on experiment on this topic in Fake Coffee with a Known Answer.


Series: "Test First with Fake Data Whose Answers You Know"

  1. Building a test with planted answers
  2. Measuring whether fakes resemble the real thing (this post)

References