Building an AI platform: architecture, boundaries, and trade-offs

This is the extended version of the code-factorio/platform case study. The same confidentiality rule applies: patterns, decisions, and trade-offs are publishable; the client, their data, and their code are not. Where I cannot give numbers, I say so instead of inventing them.

Where these platforms come from

Internal AI platforms rarely start as platforms. They start as a dozen teams independently wiring provider SDKs into services, each solving credentials, retries, usage tracking, and prompt versioning alone — until the organization notices it is paying for the same infrastructure twelve times and cannot answer basic questions: what did we spend last week, which feature regressed, who owns the prompt that just paged someone.

The instinctive response is centralization: build a platform team, mandate a stack. The organizations that succeed do something subtler — they centralize the seams and leave the product surface alone.

What the platform owns, and what it must not

The boundary question decides everything downstream. The boundary that worked:

• The platform owns the seams: provider access, credentials, routing, budgets, evaluation infrastructure, telemetry contracts, and the agent runtime's policy layer.
• The teams own the product: prompts-as-product-code, feature behavior, release cadence, and their own eval cases.

The failure mode of platform projects is owning too much. When the platform team approves prompt changes, it becomes a queue. When it owns feature logic, every product bug is a platform bug. The platform's job is to make the teams' choices safe and observable — not to make the choices.

A platform is successful when teams stop noticing it exists and can still explain everything it did.

The four planes

Model gateway

One chokepoint for provider access: authentication, routing, failover, rate shaping, and budget enforcement. Product code gets a stable interface and never sees a provider credential. The trade-off is a new dependency in the hot path — accepted deliberately, because the alternative (per-service integrations) is why the platform exists. Provider outages become routing changes in one place instead of emergency patches across the fleet.

Agent runtime

Agents get sessions, tool allowlists, and policy evaluation before any tool executes. This is the plane where "agents with production credentials" becomes "agents with scoped, auditable access." The rejected alternative — agents as privileged scripts — is faster to prototype and unauditable forever.

Evaluation harness

Eval suites run in CI against changes to prompts, tools, and models, and can block merges. The expensive part is not the harness; it is the discipline of maintaining datasets. Owning that work centrally — as platform work — is what keeps teams honest about it.

Observability

Structured events with stable IDs across every agent step: traces, cost attribution, drift alerts. The rule we set: if the platform cannot explain an agent action after the fact, the action should not have been possible.

Trade-offs, named honestly

• Latency budget. Gateway hops and policy checks cost milliseconds on every call. We bounded it by batching telemetry and keeping the policy path in-process. Zero overhead was never the goal; bounded and known was.
• Flexibility ceiling. Teams needing something the contract does not support can step outside it — through an explicit, reviewed exception path. Silent escape hatches defeat the platform; loud ones improve it.
• Maintenance asymmetry. Eval datasets and tool contracts are ongoing platform costs that produce no feature value directly. Budget for them or the platform rots from the edges.

What got rebuilt

The first agent runtime tried to be an SDK with strong opinions about application structure. Teams routed around it, correctly. The rebuild was smaller: a policy layer and a session contract that fit existing codebases instead of demanding new ones. The lesson generalizes: platforms should be adoptable in an afternoon, or they will be adopted never.

The first evaluation harness produced beautiful offline reports nobody read. The rebuild made evals a merge gate with a one-click failure diff. The difference was not technical quality; it was making the output occur where decisions happen.

Adoption mechanics

Nothing about adoption required a mandate. Teams migrated when the platform solved a problem they had personally hit: a rotated credential that took a service down, a spend spike with no attribution, a regression the eval gate caught before release. Each migration was a decision record, a small diff, and a rollback path.

The qualitative outcomes I can state: incident response went from cross-repository log archaeology to reading a single trace, and provider changes went from fleet-wide emergencies to configuration edits. Quantitative outcomes belong to the client.

If you are starting one

1. Write the boundary document before the code. What the platform owns, what it never will, and the exception process.
2. Gateway first. It pays for itself the fastest and builds trust for the harder planes.
3. Make evaluation a gate early, while the eval surface is still small.
4. Instrument everything from day one; retrofitting observability is rebuilding it.
5. Prefer adoption pull over mandate. A platform nobody opted into is a liability everyone maintains.