What a production AI product actually contains

OsceSetGo exists because practice partners don't scale. To prepare for an OSCE you need someone to play the patient — usually a coursemate, which caps practice at whenever two people are free and whoever is willing to be difficult on purpose.

A real patient withholds things. They have their own theory about what's wrong. They answer the question you asked rather than the one you meant. So the product is not a question bank with a chat window — it is a patient who behaves like one, available at 1am, plus the mark scheme that says what you missed. Everything else in this note is what it takes to make that sentence true in production.

A station — one practice scenario — is not a document. It is six related rows in six tables, split by who is allowed to see them: what the student sees before opening, the brief, the resources, the mark scheme the marker uses, the behaviour script only the AI ever reads, and the examiner questions afterwards.

The interesting one is the behaviour script. It carries the patient's personality, their wrong idea about their own illness, and the list of things they will not volunteer. A patient with chest pain who has been quietly taking their partner's medication doesn't mention it unless asked. That single field is what makes a station examinable rather than a quiz.

And one rule runs through all of it: nothing student-facing may give away what the station is testing. A reviewer once caught a resources table that quietly handed over four of a section's twelve marks by printing a value the student was supposed to derive. That is an easy mistake to make 342 times — so it is now an automated check, not a reviewer's sharp eye.

The patient's persona is assembled server-side from the knowledge base, the behaviour script and the demographics. An API key is leased from a small pool by live session count, with a heartbeat so a browser that dies mid-consultation doesn't hold one forever.

Then the server does something worth noticing: it brokers one WebRTC handshake and steps aside. Audio flows directly between the student's browser and the model — no socket held open, no audio proxied through our infrastructure, no per-user server cost while somebody talks for eight minutes. That one architectural decision is the difference between a voice product that scales and one whose hosting bill grows with every conversation.

Afterwards, the transcript is marked item by item against the real scheme — not given a holistic score. Items carry points and a critical flag, so missing a red flag is distinguishable from missing a nicety, and every attempt is filed against twelve skill domains. The analytics can say "your safety-netting is weak", which a student can act on, rather than "you scored 61%", which they cannot.

Every station ships as a version-controlled database migration, validated before it lands by a script that checks point totals, category rules, formatting — and answer leaks. Content review is code review. There is no CMS to drift out of sync with what is live.

The same discipline covers the awkward corners. Mark schemes written by different people over months use inconsistent section headings; an ordered rule set maps 646 of them onto the twelve skill domains, first match wins — so the order of the rules is the design, not an implementation detail. Access control resolves through one function used by exactly four call sites, because access logic that lives in five places drifts.

Row-level security is on for all 33 tables. Four of them have no access policies at all — which sounds like an oversight and is the opposite: enabled-with-no-policy means deny-all, and the API-key pool, billing logs and payment webhook events have no business being readable by a browser.

The rest is the unglamorous list that separates production from demo. Routes that bypass the database's security derive who you are from the session, never from the request. The admin surface answers 404 to a non-admin rather than 403, so it isn't discoverable. Uploaded screenshots live in a private bucket behind expiring links, and their file type is taken from the bytes themselves rather than the name — a browser will happily call a PHP file an image.

A redirect disguised itself as success for months: unauthenticated background jobs were being bounced to the sign-in page, which returns a cheerful 200, so scheduled work was quietly doing nothing — not failing, not alerting. API routes now answer plain 401s and never redirect.

The patient's persona was built in two places — once for voice, once for text — and the two drifted twice. The symptom was students saying the patient "acted weird", which is not a bug report you can act on. An automated check now fails the build when the two files stop matching.

The free tier is the twenty oldest stations, but each station's migration re-asserts its own pricing on replay — and 107 files disagreed with the live database, 96 of which would have re-opened the entire pharmacy track for free. Nothing was broken. It was one replay away from being broken, which is worse, because nothing tells you.

And a change that passed the type-checker failed the deploy, because different tools check different things. The lesson underneath all four is the same: a production system is the sum of the checks that run without anyone remembering to run them.

Every figure in this note — the station counts, the table count, the invariants above — is re-derived by a script rather than written down and trusted. It runs in seconds and exits with the number of failing checks, so it can gate a build.

That inverts the usual relationship. Documentation goes stale because being wrong costs nothing; here, when the doc and the script disagree, the script is right by definition and the doc gets fixed. It is the difference between a description of a system and a test of it.

When you price up an AI product, the model conversation is the cheap bit — ours is a few pence per consultation. What you are actually buying is the data design that keeps the answers away from the students, the architecture that keeps the hosting bill flat, the checks that catch the leak before the 343rd station ships, and the boring security list done properly.

Ask any prospective builder how their system fails. If the answer is about the model, they have built the demo. If the answer is about redirects that look like success, migrations that disagree with production, and documentation that tests itself — they have built the other ninety-five per cent before.