How to tell a real AI case study from a fake one

Why the fakes outnumber the records

Start with the base rate. RAND found that more than 80 percent of AI projects fail, twice the rate of IT projects without AI. MIT's NANDA initiative found that about 5 percent of generative AI pilots produced rapid revenue gains and the rest stalled. Now read any ten AI vendor pages. Every one of them has a case study, and every case study is a win. The arithmetic does not work. Most of those pages are describing something other than a finished, measured system.

The regulator noticed. In September 2024 the FTC announced five enforcement actions against companies making AI claims they could not back, from a "robot lawyer" that was never tested against a human one to storefront schemes that took at least $25 million from people on the promise of AI-generated income. The chair's line was plain: using AI tools to trick, mislead, or defraud people is illegal. A month earlier the FTC finalized a rule banning fake reviews and testimonials, AI-generated ones included, with civil penalties for knowing violators. Two years on, Holland and Knight counted more than a dozen new AI-washing cases in a single year and summarized the standard in one sentence: present-tense claims need present-tense substantiation.

None of that stops a consultant from publishing a "10x faster" line with no client behind it. It does mean the burden is on the page, and the seven checks below are how a buyer applies it.

The seven checks

1. The client. A real record names the client or describes it closely enough to picture: a 30-person orthodontic practice, a semi-private airline. When the name is withheld, the page says so and says why. A fake has "a leading company" or no client at all.
2. The system. What it reads, what it writes, and what software it touches. Dentrix. ACH. A PDF intake form. A fake has "an AI solution" and a screenshot of a dashboard.
3. The constraint. What made the work hard: a legacy system with no API, protected health information that could not leave the clinic, a process that changed shape. A fake has no constraint, because nothing was built.
4. The number, or its absence. A real number has a unit, a baseline, and a window: hours a week before and after, over a normal month. If the number does not exist, the page says that. A fake has a multiplier. "Ten times faster" and "90 percent cheaper" with no before, no after, and no period are the signature of a page written before the work.
5. The date and the status. When it shipped and whether it is still running. "In production since" is a claim someone can check. A fake is timeless.
6. What it refuses to do. Every real system has a stop rule: the uncertain match goes to a person, the retry does not post twice, the judgment call stays with the owner. A page with no stop rule is describing a demo.
7. A measurement you can inspect. The result needs a baseline, unit, and scope. “34 minutes to 6 minutes per quote” can be checked. “Ten times more productive” cannot.

Six of the seven can be faked with effort. The seventh cannot, because a page written to sell does not volunteer what it cannot prove.

Three tests you can run in a minute

• Search the number. Paste the multiplier into a search engine with the vendor's name. A real figure has a source, a date, and often a client quote. A recycled one appears on three other vendor pages.
• Ask for the call. Ask to speak with the client for ten minutes. A real record has a client who will take the call, or a stated reason they will not. A fake produces a delay.
• Ask what went wrong. Every shipped system had a week where it did not work. A consultant who can name that week built the thing. One who cannot is reading the page to you.

Our own standard, applied to us

The standard behind our case studies is the seven checks above, written down before the pages were. Applied to our own records, it produces sentences that a marketing page would cut.

The dental payment integration moved payment fees from ≈2.9% to ≈0.7% without replacing Dentrix. The wedding proposal portal reduced hands-on preparation from 34 minutes to 6 minutes per quote. The airline engagement delivered separate masterclasses, exercises, and weekly guides to Commercial and Marketing.

Those limits cost us the "ten times" line, and they are the reason a buyer can read the pages at all. A case study is evidence or it is advertising, and the page tells you which by what it is willing to leave out.

Frequently asked questions

Questions owners ask us

How can I tell if an AI case study is real?

Check for a named or closely described client, the actual system and software it touches, the constraint that made the work hard, a number with a unit and baseline or a plain statement that there is none, a date and production status, what the system refuses to do, and at least one sentence saying what the evidence does not support. A page with a multiplier and none of the rest is marketing.

Why do AI case studies not name the client?

Sometimes for a real reason: the client asked, the work touches patient or financial data, or the system is a competitive advantage. A real record says which reason applies and describes the client closely enough to picture anyway. A page that withholds the name and gives no reason is usually withholding the client because there is not one.

What should a real AI case study include?

The situation, what was built, and what happened, with specifics at each step: which software, which data stayed where, what the system stops on, when it shipped, and whether it is still running. A result figure needs a before, an after, and a defined unit or period.

Is it legal to publish fake AI case studies or testimonials?

No. The FTC finalized a rule in August 2024 banning fake or false reviews and testimonials, including AI-generated ones, with civil penalties for knowing violators, and in September 2024 it brought five enforcement actions against companies making AI claims they could not substantiate. The standard is that present-tense claims need present-tense proof.

What result did the dental payment integration produce?

The production integration moved payment fees from ≈2.9% for traditional card processing to ≈0.7% on the ACH path. Patient matching, exception handling, posting controls, and ledger read-back remain inside the clinic.

What should I ask about a case study before hiring the consultant?

Ask to speak with the client for ten minutes. Ask what went wrong during the build and how it was fixed. Ask what the system refuses to do. Ask when it shipped and whether it is still running. A consultant who built the thing answers all four in a sentence each.

What this is based on

Sources

The enforcement facts are quoted from the FTC and from counsel commentary; the failure rates from the research; our own limits from our case pages. Checked September 5, 2026.

1. Federal Trade Commission, "FTC Announces Crackdown on Deceptive AI Claims and Schemes". September 25, 2024. Five actions, including DoNotPay, Ascend Ecom, and Rytr.
2. Federal Trade Commission, "Federal Trade Commission Announces Final Rule Banning Fake Reviews and Testimonials". August 14, 2024.
3. Holland and Knight, "Operation AI Comply 2 Years Later: Continued Enforcement Against Misleading Claims". August 18, 2026. The case count and the present-tense substantiation standard.
4. RAND Corporation, "The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed". The 80 percent failure rate.
5. Fortune on MIT NANDA, "The GenAI Divide: State of AI in Business 2025". August 18, 2025. The 5 percent pilot success rate.
6. Our current client case studies.