AI proof of concept (AI PoC)

What an AI proof of concept should prove

A good AI PoC answers one question. Can a model pull line items from our supplier invoices accurately enough that a clerk only checks the hard ones? That's testable. "Can AI help finance?" isn't.

Write the pass mark down before you start. Pick a measure, such as the share of fields extracted correctly, and the level that would make the next stage worth funding.

How an AI proof of concept runs

Gather a sample of real inputs, including the messy ones. Label what the right answer looks like for each. Then try the simplest approach first, often a well-prompted model with a few examples, and score it against your labels.

Google's Rules of Machine Learning open with a reminder that you don't need machine learning if a simple heuristic does the job. A PoC that shows a plain rule beats the model has still saved you money.

OpenAI's evaluation guide makes a related point. Generative models can give different output for the same input. So you test them with evals over many cases, not one good demo.

Keep the sample honest. Take it from real work over an ordinary period, not from the examples someone remembers. If one in ten real inputs is a scan, a forwarded email or a document in another language, one in ten of the test cases should be too.

Which pass marks an AI PoC should use

Accuracy is the obvious measure, and it is rarely enough on its own. A model that gets the right answer but takes half a minute, or costs more per run than the person it helps, has not passed.

Write down a pass mark for each measure that could stop the project:

• Quality. The share of cases handled correctly on your labelled test set, and which kinds of case it gets wrong.
• Cost per run. What one request costs at the volume you expect, from the provider's published prices.
• Latency. How long a person or a system waits for the answer.
• Failure behaviour. What the system does when it is unsure: guess, or hand the case to a person.

The numbers are yours to set. What matters is that they are agreed before the first result comes in, so the result is a decision and not a debate.

AI PoC vs prototype vs MVP

Each one tests a different risk. A PoC asks whether the technology can do the job. A prototype asks whether people understand the interaction and want it. An MVP asks whether real users will adopt it, or pay for it, in their real work.

The usual mistake is letting a PoC drift into an MVP. Someone adds a login, then a dashboard, and weeks later nobody can say whether the core extraction works. Keep the PoC rough and the question narrow.

You won't always need all three. If the AI part is well understood, like summarising short tickets, you can skip the PoC and prototype the workflow instead.

If the PoC passes, the next step is usually a minimum viable product: the smallest version real users can do one job with. The PoC told you the AI can do it. The MVP tells you whether people will use it.

What you should hold at the end of an AI PoC

A finished PoC is a small pile of evidence, not a demo. You should be able to hand it to someone who was not in the room:

• A score against a labelled test set. The same set can be re-run when the prompt, the model or the approach changes.
• The failure cases. The inputs it got wrong, grouped by why, so you can see whether they are fixable.
• A running-cost and latency note. What it would cost and how fast it would be at real volume.
• A recommendation. Build, change approach, or stop, with the reasons.

The test set is the part people undervalue. It is what turns later arguments about quality into a number, which is why AI evaluation starts here. Our longer post on what an AI proof of concept is covers when the PoC code is worth keeping and when to throw it away.

Why AI proofs of concept stall

Many stall for reasons outside the model. The test data was cleaner than real inputs, so production breaks it. Nobody agreed a pass mark, so the result is a demo instead of a decision.

Others pass and then sit on a shelf because no one planned the next step. Before you start, decide who owns the build if the PoC succeeds and what budget a pass would release.

How Gr8r runs an AI proof of concept

We scope a PoC around a single task and a written pass mark, using a sample of your real data. We build an evaluation set alongside the code, so the result is a score you can compare across approaches.

At the end you get a recommendation: build, change approach or stop. We'd rather tell you to stop early than build a product on a shaky model. The code, prompts and evaluation set are yours.

A PoC sits inside our product engineering work. Projects start at $8,000, and the final price is agreed in writing before work starts. If the PoC is the first step towards a product you plan to sell, see how we build an AI product with founders.