How to tell a real AI case study from a fake one
Most AI projects fail and most AI case studies report a win. Both cannot be true of the same pages. Seven checks sort the records from the marketing, and the last one is the tell: a real case study says what its evidence does not support.
A real AI case study names or describes the client, names the system and what it touches, states the constraint that made the work hard, gives a number with its unit and baseline or says plainly that it has none, carries a date and a status, lists what the system refuses to do, and says what the record does not support. A fake one has a multiplier, no client, no system, no date, and nothing it would not claim.
Why the fakes outnumber the records
Start with the base rate. RAND found that more than 80 percent of AI projects fail, twice the rate of IT projects without AI. MIT's NANDA initiative found that about 5 percent of generative AI pilots produced rapid revenue gains and the rest stalled. Now read any ten AI vendor pages. Every one of them has a case study, and every case study is a win. The arithmetic does not work. Most of those pages are describing something other than a finished, measured system.
The regulator noticed. In September 2024 the FTC announced five enforcement actions against companies making AI claims they could not back, from a "robot lawyer" that was never tested against a human one to storefront schemes that took at least $25 million from people on the promise of AI-generated income. The chair's line was plain: using AI tools to trick, mislead, or defraud people is illegal. A month earlier the FTC finalized a rule banning fake reviews and testimonials, AI-generated ones included, with civil penalties for knowing violators. Two years on, Holland and Knight counted more than a dozen new AI-washing cases in a single year and summarized the standard in one sentence: present-tense claims need present-tense substantiation.
None of that stops a consultant from publishing a "10x faster" line with no client behind it. It does mean the burden is on the page, and the seven checks below are how a buyer applies it.
The seven checks
- The client. A real record names the client or describes it closely enough to picture: a 30-person orthodontic practice, a semi-private airline. When the name is withheld, the page says so and says why. A fake has "a leading company" or no client at all.
- The system. What it reads, what it writes, and what software it touches. Dentrix. ACH. A PDF intake form. A fake has "an AI solution" and a screenshot of a dashboard.
- The constraint. What made the work hard: a legacy system with no API, protected health information that could not leave the clinic, a process that changed shape. A fake has no constraint, because nothing was built.
- The number, or its absence. A real number has a unit, a baseline, and a window: hours a week before and after, over a normal month. If the number does not exist, the page says that. A fake has a multiplier. "Ten times faster" and "90 percent cheaper" with no before, no after, and no period are the signature of a page written before the work.
- The date and the status. When it shipped and whether it is still running. "In production since" is a claim someone can check. A fake is timeless.
- What it refuses to do. Every real system has a stop rule: the uncertain match goes to a person, the retry does not post twice, the judgment call stays with the owner. A page with no stop rule is describing a demo.
- What the record does not support. This is the tell. A real case study has at least one sentence that limits its own claim: we do not publish a savings figure because the client has not shared one we can stand behind; identifying details are withheld at the client's request. A fake never limits itself, because it was written to sell.
Six of the seven can be faked with effort. The seventh cannot, because a page written to sell does not volunteer what it cannot prove.
Read three records held to this standard.
Our case studies state only what the available record supports, and each one says where the evidence stops. Judge them by the seven checks above.
Three tests you can run in a minute
- Search the number. Paste the multiplier into a search engine with the vendor's name. A real figure has a source, a date, and often a client quote. A recycled one appears on three other vendor pages.
- Ask for the call. Ask to speak with the client for ten minutes. A real record has a client who will take the call, or a stated reason they will not. A fake produces a delay.
- Ask what went wrong. Every shipped system had a week where it did not work. A consultant who can name that week built the thing. One who cannot is reading the page to you.
Our own standard, applied to us
The standard behind our case studies is the seven checks above, written down before the pages were. Applied to our own records, it produces sentences that a marketing page would cut.
The dental payment integration moved recurring payments to ACH without replacing Dentrix, keeps patient matching and ledger posting inside the clinic, stops every uncertain match for a person, and is running in production. We do not publish a savings figure for it, because the practice has not shared one we can stand behind. The patient presentation automation assembles presentations from clinical inputs and keeps missing inputs visible to the operator instead of filling them in. The airline engagement withholds identifying details, and the page says that is at the client's request.
Those limits cost us the "ten times" line, and they are the reason a buyer can read the pages at all. A case study is evidence or it is advertising, and the page tells you which by what it is willing to leave out.
What to hold on to
- The base rate is the first check. Most AI projects fail, and every vendor page reports a win. Most of those pages are not describing a finished system.
- Seven checks. Client, system, constraint, number with baseline, date and status, stop rule, and what the record does not support.
- The seventh is the tell. A real case study limits its own claim. A page written to sell never does.
- Multipliers without a baseline are the signature of a fake. Search the number, ask for the client call, ask what went wrong.
- The FTC treats unsubstantiated AI claims as deception. Five actions in September 2024, a fake-reviews rule with civil penalties, and more than a dozen cases in the last year.
Questions owners ask us
How can I tell if an AI case study is real?
Check for a named or closely described client, the actual system and software it touches, the constraint that made the work hard, a number with a unit and baseline or a plain statement that there is none, a date and production status, what the system refuses to do, and at least one sentence saying what the evidence does not support. A page with a multiplier and none of the rest is marketing.
Why do AI case studies not name the client?
Sometimes for a real reason: the client asked, the work touches patient or financial data, or the system is a competitive advantage. A real record says which reason applies and describes the client closely enough to picture anyway. A page that withholds the name and gives no reason is usually withholding the client because there is not one.
What should a real AI case study include?
The situation, what was built, and what happened, with specifics at each step: which software, which data stayed where, what the system stops on, when it shipped, and whether it is still running. If there is a result figure, it needs a before, an after, and a period. If there is not one, the page should say so.
Is it legal to publish fake AI case studies or testimonials?
No. The FTC finalized a rule in August 2024 banning fake or false reviews and testimonials, including AI-generated ones, with civil penalties for knowing violators, and in September 2024 it brought five enforcement actions against companies making AI claims they could not substantiate. The standard is that present-tense claims need present-tense proof.
Why does Prometheus not publish savings numbers?
Because the practice in our dental case study has not shared a figure we can stand behind, and a number we cannot stand behind is the exact thing this article tells you to distrust. The page says the system is running in production and what it does. When a client shares a measured result, the page will carry it with its baseline and period.
What should I ask about a case study before hiring the consultant?
Ask to speak with the client for ten minutes. Ask what went wrong during the build and how it was fixed. Ask what the system refuses to do. Ask when it shipped and whether it is still running. A consultant who built the thing answers all four in a sentence each.
Sources
The enforcement facts are quoted from the FTC and from counsel commentary; the failure rates from the research; our own limits from our case pages. Checked September 5, 2026.
- Federal Trade Commission, "FTC Announces Crackdown on Deceptive AI Claims and Schemes". September 25, 2024. Five actions, including DoNotPay, Ascend Ecom, and Rytr.
- Federal Trade Commission, "Federal Trade Commission Announces Final Rule Banning Fake Reviews and Testimonials". August 14, 2024.
- Holland and Knight, "Operation AI Comply 2 Years Later: Continued Enforcement Against Misleading Claims". August 18, 2026. The case count and the present-tense substantiation standard.
- RAND Corporation, "The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed". The 80 percent failure rate.
- Fortune on MIT NANDA, "The GenAI Divide: State of AI in Business 2025". August 18, 2025. The 5 percent pilot success rate.
- Our case studies and the evidence standard stated on that page.