What a production AI product actually contains

A model, a prompt and a chat window can demonstrate an idea. A paid product also needs the surrounding data, permissions, billing and operational controls. This article looks at that surrounding work.

This note uses an August 2026 engineering snapshot of our own OsceSetGo: OSCE exam practice for UK medical and pharmacy students, where a student opens a station, holds a real spoken consultation with an AI patient, and gets marked against the kind of scheme an examiner would use. It runs to 342 stations, 9,959 mark-scheme items, 33 database tables, 442 migrations and around thirty-four thousand lines of TypeScript. None of those numbers is the AI. That is the point.

The product is not the model

OsceSetGo exists because practice partners don't scale. To prepare for an OSCE you need someone to play the patient — usually a coursemate, which caps practice at whenever two people are free and whoever is willing to be difficult on purpose.

A real patient withholds things. They have their own theory about what's wrong. They answer the question you asked rather than the one you meant. So the product is not a question bank with a chat window — it is a patient who behaves like one, available at 1am, plus the mark scheme that says what you missed. Everything else in this note is what it takes to make that sentence true in production.

Data design is where the product lives

A station — one practice scenario — is not a document. It is six related rows in six tables, split by who is allowed to see them: what the student sees before opening, the brief, the resources, the mark scheme the marker uses, the behaviour script only the AI ever reads, and the examiner questions afterwards.

The interesting one is the behaviour script. It carries the patient's personality, their wrong idea about their own illness, and the list of things they will not volunteer. A patient with chest pain who has been quietly taking their partner's medication doesn't mention it unless asked. That single field is what makes a station examinable rather than a quiz.

And one rule runs through all of it: nothing student-facing may give away what the station is testing. A reviewer once caught a resources table that quietly handed over four of a section's twelve marks by printing a value the student was supposed to derive. That is an easy mistake to make 342 times — so it is now an automated check, not a reviewer's sharp eye.

What happens when a student presses start

The patient's persona is assembled server-side from the knowledge base, the behaviour script and the demographics. An API key is leased from a small pool by live session count, with a heartbeat so a browser that dies mid-consultation doesn't hold one forever.

The server brokers the WebRTC connection, then audio flows directly between the student's browser and the model. This avoids relaying the conversation through our application server. Model usage and other application activity still have costs; this architecture removes one source of server work rather than making conversations free to run.

Afterwards, the transcript is marked item by item against the real scheme — not given a holistic score. Items carry points and a critical flag, so missing a red flag is distinguishable from missing a nicety, and every attempt is filed against twelve skill domains. The analytics can say "your safety-netting is weak", which a student can act on, rather than "you scored 61%", which they cannot.

Content is code

Every station ships as a version-controlled database migration, validated before it lands by a script that checks point totals, category rules, formatting — and answer leaks. Content review is code review. There is no CMS to drift out of sync with what is live.

The same discipline covers the awkward corners. Mark schemes written by different people over months use inconsistent section headings; an ordered rule set maps 646 of them onto the twelve skill domains, first match wins — so the order of the rules is the design, not an implementation detail. Access control resolves through one function used by exactly four call sites, because access logic that lives in five places drifts.

Security is mostly saying no in the right places

Row-level security is on for all 33 tables. Four of them have no access policies at all — which sounds like an oversight and is the opposite: enabled-with-no-policy means deny-all, and the API-key pool, billing logs and payment webhook events have no business being readable by a browser.

The rest is the unglamorous list that separates production from demo. Routes that bypass the database's security derive who you are from the session, never from the request. The admin surface answers 404 to a non-admin rather than 403, so it isn't discoverable. Uploaded screenshots live in a private bucket behind expiring links, and their file type is taken from the bytes themselves rather than the name — a browser will happily call a PHP file an image.

Four bugs that cost real time

A redirect disguised itself as success for months: unauthenticated background jobs were being bounced to the sign-in page, which returns a cheerful 200, so scheduled work was quietly doing nothing — not failing, not alerting. API routes now answer plain 401s and never redirect.

The patient's persona was built in two places — once for voice, once for text — and the two drifted twice. The symptom was students saying the patient "acted weird", which is not a bug report you can act on. An automated check now fails the build when the two files stop matching.

The free tier is the twenty oldest stations, but each station's migration re-asserts its own pricing on replay — and 107 files disagreed with the live database, 96 of which would have re-opened the entire pharmacy track for free. Nothing was broken. It was one replay away from being broken, which is worse, because nothing tells you.

And a change that passed the type-checker failed the deploy, because different tools check different things. The lesson underneath all four is the same: a production system is the sum of the checks that run without anyone remembering to run them.

Documentation that checks itself

Every figure in this note — the station counts, the table count, the invariants above — is re-derived by a script rather than written down and trusted. It runs in seconds and exits with the number of failing checks, so it can gate a build.

That inverts the usual relationship. Documentation goes stale because being wrong costs nothing; here, when the doc and the script disagree, the script is right by definition and the doc gets fixed. It is the difference between a description of a system and a test of it.

What to take from this if you're buying a build

When you price an AI product, include the data design, access controls, automated checks and operating costs alongside the model integration. Voice cost depends on the model, duration and usage pattern. Ask for an estimate with those assumptions stated, and for a way to inspect actual usage after launch.

Ask a prospective builder how they test the system and investigate failures. Useful examples include model output, permissions, billing, integrations and deployment. Our software brief checklist helps turn those discussions into a defined scope.