Building with AI is not a prompt demo. Production platforms need data pipelines, retrieval quality, guardrails, evaluation, and the same engineering discipline as any other business-critical system.
The teams that succeed treat models as replaceable components. What lasts is the surrounding system: how documents are ingested, how permissions are enforced, how answers are grounded, how failures are observed, and how quality is measured after launch.
Private inference matters when data cannot leave the perimeter. Managed models matter when speed-to-value dominates. The architecture decision is ownership and risk — not fashion.
Agents amplify both capability and blast radius. Without tool allowlists, audit trails, and human escalation paths, autonomy becomes an incident generator.
Evaluation is not a slide. Offline golden sets, regression suites for prompts and retrieval, and production traces are how you know the system still works after the next model upgrade.
Start from the workflow, not the model
Map the decisions humans already make. Then decide where language models reduce latency, cost, or inconsistency — and where they must never act alone.
If you cannot name the system of record, the retention policy, and the escalation owner, you are not ready to put a model in the path.
Retrieval and grounding beat clever prompting
Most enterprise AI failures are retrieval failures wearing a model costume. Chunking, access control, freshness, and citation discipline matter more than temperature tweaks.
Design answers to show provenance. Users trust systems that can point to sources more than systems that sound confident.
Private vs managed is a risk decision
Fine-tuned private runtimes (including LoRA → GGUF/Ollama-style serving) are appropriate when assessment content, case files, or regulated text cannot touch public APIs.
Managed APIs remain rational when the data classification allows it and iteration speed is the bottleneck. Hybrid is normal: private for sensitive corpora, managed for generic drafting.
Operate it like software
Version prompts, indexes, tools, and models together. Ship behind feature flags. Monitor refusal rates, empty retrievals, latency, and ticket escalations.
Schedule evaluation the way you schedule load tests. Model vendors change under you; your suite must catch drift.
Takeaways
- AI platform work is systems engineering with probabilistic components.
- Grounding, permissions, and evaluation outrank demo polish.
- Private serving and managed APIs are both valid — pick by data risk.
- Agents need tool boundaries and human handoff by default.