When AI Says "It Can't Be Done as Is": Breaking Down Why and Finding Something Else to Measure (1)

When an AI or data specialist says something can't be done as is, the project usually stops right there. But that answer can be the starting point for redefining the problem rather than the end. Drawing on OpenAI's discontinued AI text classifier, Seoul's late-night bus routes, welfare checks based on electricity data and satellite-based poverty maps, this piece shows how to split the reasons into data, rules and accuracy, and how to use proxy metrics to measure what you can't measure directly.

AXAI transformationProblem definitionProxy metrics

"It can't be done as is" is often not a rejection but a sign that the question was set up wrong. What you ask after that answer decides the fate of the project.

The moment the meeting room goes quiet

There is one scene where AI adoption projects most often stall. It's when the specialist says, "We can't do this with the data we have," or "We can't reach that accuracy." These days AI itself sometimes says so. Before even attempting the task, it may say first, "This approach won't produce the result you want."

At that moment the room splits two ways. One side ends with "Then let's drop it," and the other pushes ahead with "Then make it work." Both are poor choices. The first throws away a problem that could have been solved, and the second produces output that only pretends to work.

What we recommend is a third path. When you hear "it can't be done," break down what exactly can't be done, and put a solvable question in its place. This part covers the first two steps: how to break down the reasons and how to find something else to measure.

Principle 1. Split "it can't be done" into three

"It can't be done" sounds like one block, but the reason is usually one of three.

  • The data doesn't hold that level of precision. Resolution, frequency or volume falls short. A request to find human-sized objects in free satellite imagery where each pixel covers 10 meters is an example.
  • The question is whether it should be done. Law, ethics or security mean it could be built but should not be released.
  • It works, but falls short of business standards. The accuracy is not at a level people can trust and use.

A typical example of the third type is OpenAI's classifier for AI-written text. When OpenAI released the classifier in January 2023 to distinguish human-written from AI-written text, it disclosed that the tool correctly identified only 26% of AI-written text as "likely AI-written," while wrongly labeling 9% of human-written text as AI-written. Then on July 20 of the same year, it discontinued the service, citing its low accuracy, adding that it would research more effective techniques for verifying where text comes from. The question shifted from "judge finished text" to "record provenance from the moment of creation."

Bar chart of the performance of OpenAI's AI-written text classifier. It correctly labeled 26% of AI-written text as likely AI-written, and wrongly labeled 9% of human-written text as AI-written.
Classifier performance as disclosed by OpenAI at release. Source: OpenAI, 2023
A diagram splitting the answer that something can't be done as is into three reasons: data, rules and accuracy. If the data falls short, find something else to measure; if it's a rules problem, narrow the scope; if accuracy falls short of business standards, add a human review step or change the question.
Three kinds of reasons it can't be done, and the response to each

Once the reasons are separated, the responses diverge too. For the first type, find something else to measure. For the second, narrow the scope. For the third, add a human review step or change the question. Some things remain impossible to the end. We set out the criteria for that judgment separately in When AI Is Not the Answer.

Principle 2. If you can't measure it directly, measure what moves with it

Many "we don't have the data" answers really mean "there's no data that records that value directly." In that case, look for a value that moves together with it and is already being accumulated somewhere. This is called a proxy metric.

Seoul's late-night buses. When the Seoul Metropolitan Government planned late-night bus routes in 2013, none of the data it had directly recorded "who wants to go from where to where after midnight." Instead, the city analyzed 3 billion mobile phone calls made between midnight and 5 a.m., provided by KT (in Korean), to find where people gathered late at night and set the routes accordingly. Rather than measuring travel demand directly, it measured the traces people leave where they are at that hour.

Welfare checks using electricity data. Checking on elderly people who live alone usually means fitting them with a wristband or sensors. A welfare check service for single-person households developed jointly by SK Telecom and Korea Electric Power Corporation (KEPCO) uses AI to analyze a household's electricity usage patterns and combines them with whether calls, texts and data are being used (in Korean), alerting welfare workers when it detects signs that differ from normal. Because no separate device is needed, operating costs are low, and users don't have to change how they live. Instead of measuring daily life itself, it looks at the traces of electricity use that daily life leaves behind.

Poverty maps drawn from satellite images. The same approach exists outside Korea. A 2016 study published in Science by Stanford researchers started from the problem that poverty surveys themselves are rare in Africa. According to the paper, 39 of 59 African countries did not conduct even two nationally representative poverty surveys between 2000 and 2010. The researchers first trained a neural network to predict nighttime light intensity from daytime satellite images, so that it learned features like roads, roofs and farmland, then used those features to estimate village-level consumption and assets. Using only public data, they explained up to 75% of the variation in local economic indicators across five countries, including Nigeria and Tanzania.

All three cases asked the same question: "When the thing we want to know happens, what other value is already being recorded somewhere?"

CaseWhat they wanted to knowWhat they measured instead
Seoul late-night buses (2013)Who wants to go from where to where after midnight3 billion mobile phone calls between midnight and 5 a.m.
Welfare checks for single-person householdsWhether something is wrong in the life of an elderly person living aloneHousehold electricity usage patterns plus whether calls, texts and data are being used
Satellite poverty maps (2016)Village-level consumption and assetsFeatures from daytime satellite images learned by predicting nighttime light intensity

Finding proxy metrics in your own company

You can ask the same question inside your company. If you can't measure customer satisfaction directly, look at reorder intervals and the content of inquiries. If you can't see equipment condition directly, look at power consumption and noise. Values originally collected for other purposes, such as access logs, shipment tracking records and quote request emails, often turn out to hold unexpected answers.

There is one test for a good proxy. You must be able to explain in one sentence how it connects to what you originally wanted to know. If the path is clear, like "where there are many late-night calls, there are people at that hour," it is usable. If all you can say is "they seem to move together somehow," it is risky. To find where such values are stored in your company, use the checklist in Is Your Company's Data Ready for AI?

In short

"It can't be done as is" is usually an honest answer. Whether you take that honesty as the end of the project or as a starting point for reframing the problem is up to the listener.

First, split the reasons it can't be done into data, rules and accuracy, and measure what you can't measure directly through values that move with it. These two steps alone get many stalled projects moving again.

In the next part, we go one step further: how to change the question itself, and how to measure how far the new question has drifted from the original goal.


Series: "When AI Says It Can't Be Done as Is"

  1. Breaking down why and finding something else to measure (this article)
  2. Changing the question and measuring the drift

References