Handing a Veteran's Intuition to AI — Exposing the Weights and Making Experts the Reviewers (2)
Once intuition has been broken into conditions, you still have to decide how to combine them, how far intuition can be trusted and where to place the experts. Drawing on Grove and colleagues' meta-analysis of clinical versus mechanical prediction, Kahneman and Klein's paper on the conditions for intuition, the dispute over expert compensation in a Korean government tacit-knowledge program and the WHO surgical safety checklist study, this article sets out the principles for finishing the criteria and an eight-step plan.
Breaking down the conditions is only the start. How to combine them is for the company to decide. And the final reviewer of those criteria should still be the veteran.
In Part 1, we covered how to break an expert's intuition into a few named conditions and write down how each will be measured.
Three questions left after the breakdown
Once the conditions and measurement methods are set, projects usually go like this: collect data, train a model, report accuracy. But three questions are often skipped along the way.
- With what weights should the conditions be combined?
- How far should the expert's intuition be trusted as the right answer?
- What role does the expert play in this process?
All three are closer to judgment calls than to technical questions. That is why executives need to answer them directly.
Principle 1. Expose the weights instead of hiding them
At the combining stage, many projects move on with "let the AI learn it." Finding the weights through training is fine in itself. The problem is when the result is buried inside the model where no one can see it.
Weights are a judgment, not a technology. Whether quality or delivery matters more, and how much safety margin to keep, are for the company to decide. Lay out the weight of each condition in a table or on screen so that experts and managers can push back: "Should this condition really carry this much weight?" When the weights are visible, mistakes get caught faster. Errors such as a newly added condition entering with a weight of 0 are more common than you would think, and when hidden, nobody notices for months.
There is evidence for putting criteria into a formula in the first place. A 2000 meta-analysis by Grove and colleagues, covering 136 studies in medicine, finance and criminology, reported that combining the same information with a set formula was on average about 10% more accurate than combining it in a person's head. The gap appeared consistently regardless of the judges' experience. This is not because experts are wrong, but because even the same person judges slightly differently from day to day.
Principle 2. Separate where intuition can be trusted from where it cannot
Not every veteran's intuition is of the same quality. Daniel Kahneman and Gary Klein long stood on opposite sides of the debate over intuition, but after nearly six years of joint work they agreed on one thing in a 2009 joint paper: whether intuition can be trusted depends on two things. Does the environment in which the judgment is made have regularities, and has the person had enough feedback to learn them? The paper states plainly that "subjective confidence is not a reliable indicator of the accuracy of judgment."
Apply this to company work and the line becomes clear.
- Judgments with fast, clear outcomes (whether a part is defective, machining conditions, equipment faults): the master's intuition is a good answer key. These come first for codification.
- Judgments with slow, blurry outcomes (new product success, the market five years out, hiring success): using intuition as the answer key locks in that person's biases as well.

Once you build the criteria, you also see the areas the criteria cannot separate in the first place. Among cases with nearly identical conditions, score differences are bound to be small. Inflating those differences with colors or rankings on screen pretends to distinguish what cannot be distinguished. Writing down "within this range, the criteria cannot tell which is better" is also part of the criteria.
That is why it is safer to use the first version of the criteria as a sieve that filters out "this is definitely not it" rather than as a judge that picks "this is the best."
Principle 3. Bring experts in as reviewers, not as data providers
Tacit knowledge projects most often collapse over people, not technology.
The press-line project (in Korean) introduced in Part 1 set its deliverable as "at least 1,500 golden-dataset records reviewed by the master." The expert is built into the structure not as someone who hands over data but as the person who confirms what the right answer is.
There is a counterexample, too. In the same government program, the recommended research allowance for skilled workers was 20% of labor costs, but according to Korea Industrial Daily (in Korean), 14 of 30 teams set it lower, and one consortium set it at about 0.97%. There was also a case where a team of 49 participants was allocated a total of 5.94 million won. If this is how people are treated when asked to hand over a lifetime of know-how, what comes out will most likely be only what is already in the manual.

There is also evidence that outcomes improve when experts hold the criteria. In a 2009 study that introduced the World Health Organization's 19-item surgical safety checklist at eight hospitals in eight countries, the death rate fell from 1.5% to 0.8% and inpatient complications fell from 11.0% to 7.0%. The checklist did not replace surgeons. It took the checks experienced staff had always done, put them into words and had everyone do them in the same order.

How people's roles change when AI is used to build criteria is covered further in AX: How Do Human Roles Change?.
One more thing: don't build the criteria once and lock them in place. When raw materials or equipment change, intuition changes too. Experts follow those changes with their bodies, but criteria frozen in a document do not. At least once a quarter, gather the cases where the criteria and floor judgments diverged and realign them. The person who chairs that session should also be an expert.
The plan: eight steps to turn one intuition into criteria
- Pick one judgment. A good candidate is frequent, has outcomes you can confirm quickly and stops when that person is absent.
- Observe and ask beside them. Ask "Why did you stop just now?" and "What did you see?" on the spot, and record the answers. The floor beats a meeting-room interview.
- Break it into 5 to 8 named conditions. Using the expert's own words as names reduces resistance.
- Write down the measurement method and data source for each condition. Mark the data you don't have separately.
- Draft the weights and adjust them with the expert. The weights must be visible on screen or in a document.
- Check against past cases. Feed in cases with known outcomes and see where the criteria and the expert's judgment diverge. The method is laid out in How Do You Verify What AI Built?
- Document the areas the criteria cannot separate. A sentence like "under these conditions, a person has to look" is also a criterion.
- Attach the expert's name and compensation to the criteria. You need a record of who built the criteria so there is someone to ask when they need fixing.
In summary
When combining the conditions you have broken out, expose the weights, separate where intuition can be trusted and make experts the reviewers. Then the criteria become something AI can use, and they stay with the company after that person retires.
Building criteria is not about replacing experts. It is about writing down what experts know in the company's language. That is why the final reviewer of the criteria should still be them.
I kept a record of experimenting with this topic firsthand in Finding a Feng Shui Site with AI.
Series: Handing a Veteran's Intuition to AI
- Breaking intuition into measurable conditions
- Exposing the weights and making experts the reviewers (this article)
References
- Grove et al., Clinical versus mechanical prediction: a meta-analysis (2000)
- Kahneman, Klein, Conditions for intuitive expertise: a failure to disagree (2009)
- 세계일보, 숙련공의 손길 AI로 데이터화… 명장의 감 다 살렸어 (2026) (in Korean; Segye Ilbo, "A skilled worker's touch turned into data with AI, capturing the master's feel")
- 산업일보, 제조 암묵지 AI 데이터화 사업, 숙련 기술자 보상 기준은 미흡 (2026) (in Korean; Korea Industrial Daily, "Tacit manufacturing knowledge AI data program falls short on compensation standards for skilled workers")
- Haynes et al., A surgical safety checklist to reduce morbidity and mortality in a global population, NEJM (2009)