A Digital Twin Sized for Small Businesses — What to Check Before Trusting a Simulator AI Built (2)
A simulator written by AI usually runs, and its graphs usually look plausible. But running and being right are not the same. Drawing on Google researchers' FEABench, the sinking of Norway's Sleipner A platform, the Columbia accident investigation and FDA guidance on simulation credibility, this article sets out four principles and a step-by-step plan for checking simulation results before using them in decisions: field yardsticks, sanity checks, labeling calibrated values and listing what the model cannot see.
A simulator written by AI usually runs. The graphs usually look plausible. What you need to check before using it in a decision is whether it is right.
In Part 1, we covered which questions that are expensive to test for real should go to a simulator and which should be avoided.
A number coming out is not an answer coming out
Ask AI to "write code that calculates how much damage increases under these conditions," and you get results within minutes, complete with tables and graphs. The problem is that you cannot tell from the outside whether the output is right or wrong.
The history of simulation is full of cases where numbers came out all the way to the end and were still wrong. That is why you need to check four things before using results in a decision.
Principle 1. Get a field yardstick in hand first
Before writing a simulator, the first thing to do is know what range the results should fall in. Internal records, published papers, manufacturer specifications and past incident records serve as yardsticks. With a yardstick, you can tell when the first result is wrong. Without one, a simulator will make anything sound plausible.
And even with a yardstick, dragging a model outside the range in which it was validated gets dangerous. In 2003, the board investigating the Columbia accident in the United States sharply criticized the analysis tool NASA used to assess wing damage. The tool had originally been built to predict, before launch, the damage that small pieces of ice would do to the external fuel tank, but the piece of foam actually analyzed was 400 times larger than the tool's database standard. The board judged that using the tool in this new and entirely different situation compromised its ability to accurately predict damage (Nextgov).

Regulators use the same principle. In November 2023, the U.S. Food and Drug Administration (FDA) issued final guidance on assessing the credibility of simulations used in medical device submissions. It covers models based on laws of physics and requires credibility to be shown through verification and validation procedures. Machine learning or standalone AI models were excluded from its scope.
Principle 2. Check separately that it runs and that it is right
This is the biggest trap of AI-built simulators. The code usually runs. Whether it is right is a separate question.
FEABench, released by Google researchers in 2025, had AI agents operate commercial finite element analysis software directly. The best approach produced executable commands 88% of the time. Yet only one problem had a calculated value within 10% of the correct answer, and even that one was right by chance: the program printed its default value of 20°C instead of the correct 18.3°C. The researchers also treated this as a sign that the problem had not really been solved.

Analyses built by people fall into the same trap. In August 1991, the concrete substructure of Norway's Sleipner A offshore platform sank during a submergence test. The loss was about 700 million dollars. The investigation found that the finite element analysis had underestimated the shear stress in a critical area by 47%, and that the reinforcement anchorage at that spot was also insufficient (excerpt from the SINTEF report). The analysis ran to the end, and numbers came out. They were just wrong.
So attach at least two of the following checks to every simulator.
- Conservation check: What goes in, what goes out and what remains must add up. For an operations simulator, check that receipts, shipments and inventory changes match.
- Hand-calculation comparison: Compare key results with a rough calculation on paper. If the order of magnitude differs from a simple calculation like "daily throughput divided by headcount," something is wrong.
- Analytical-solution comparison: First run a simple problem with a known answer and confirm you get the same value as the textbook formula.
- Common-sense limits: Make sure results don't cross lines that cannot be crossed, such as how much one person can handle in a day or a machine's rated capacity.
Why these checks are needed shares the same root as When AI Is Confidently Wrong. How to design a verification process is covered separately in How Do You Verify What AI Built?
Principle 3. Record calibrated values separately from calculated ones
When fitting a simulator to field measurements, you end up adjusting some values by hand. If the first result comes out far below the measurements, you add a missing term or change a coefficient to pull it into range. Up to this point, it is a normal process.
The problem is when results fitted this way are then used as evidence that the model has been "validated." Values fitted to measurements are not evidence that the model is right. They were put in to make it match, so of course it matches. This distinction is also why the FDA guidance requires verification and validation procedures separately.
So pull calibrated values out and label them. Once they are separated as adjustable values, you can also see how much the conclusion shifts when you change them. If the conclusion swings a lot, it is leaning on the calibrated values.
In reports, separate the two kinds of values like this.
| Item | Calculated values | Calibrated values |
|---|---|---|
| How they were obtained | Calculated from the model's formulas and assumptions | Adjusted by hand to fit the range of field measurements |
| Evidence the model is right? | Something to compare against measurements | No. They were put in to match, so of course they match |
| How to report them | Show as results | Label separately, and note whether the conclusion shifts when they change |
This is also the question executives receiving the report should ask: "Which of these numbers are calculation results, and which were fitted in?" A simulation report without this distinction cannot be told apart, from the outside, from one that fixed the conclusion first and fitted the numbers to it.
Principle 4. Write down first what the model cannot see
Every model throws something away. A model that flattens space into a single line cannot see the vortices inside a cross-section. A model that calculates only static forces cannot see fatigue cracks from repeated loading. A volume model that calculates with averages cannot see bottlenecks that pile up at particular times of day. In the end, Columbia's analysis tool could not see outside the range it was built for, either.
Write this list down before the results, and the way you read the numbers changes. You trust the trend that "option B is better," but not exactly how many percent better it is. If you use a simulator for company decisions, put five lines on "what this model cannot see" on the first page of the results report. Those five lines are the cheapest device against overconfidence.
Steps to start small
- Write one decision as a single comparative question. "Which of A and B reduces X more?"
- Collect the range of field measurements first. Internal records, published papers, manufacturer specifications and past incident records serve as yardsticks.
- Start with the simplest model. Begin with a single lump or a one-line, one-dimensional model. Hand it to AI, but have it first explain the formulas and assumptions it uses in sentences.
- Attach at least two checks. A conservation check and a hand-calculation comparison are the basics.
- Compare with field measurements, and record calibrated values separately.
- Write the list of what the model cannot see.
- Use it in a decision, then compare again with actual results. As this step accumulates, the simulator gets closer to a real digital twin.
In summary
To use an AI-built simulator in a decision, get a field yardstick in hand first, check separately that it runs and that it is right, and write down what was calibrated and what the model cannot see. Then even a small simulator can save you a few rounds of trial and error that cost tens of millions of won.
A simulator does not make the decision for you. It just adds a line of evidence to a decision that used to be made on gut feel. That one line prevents expensive mistakes more often than you would think.
I kept records of experimenting with this topic firsthand in Was the Bus Driver Who Told Us to Close the Windows in Tunnels Right? and Where Does a Tent Give Way First in a Windy Campsite?
Series: A Digital Twin Sized for Small Businesses
- Ask a simulator first when a real test is expensive
- What to check before trusting a simulator AI built (this article)
References
- Nextgov, Columbia board blasts use of analysis tool (2003)
- HPN, FDA Issues Final Guidance: Assessing the Credibility of Computational Modeling and Simulation in Medical Device Submissions (2023)
- Mudur et al., FEABench: Evaluating Language Models on Multiphysics Reasoning Ability (2025)
- Excerpt from the SINTEF report, The sinking of the Sleipner A offshore platform (1991)