95% cut in AI operating cost for a venture-backed AI startup

The challenge

The company was growing fast and leaning on frontier large language models for nearly every step of its GTM data work: enriching records, classifying inbound, drafting outbound, monitoring competitors, and more. Every one of those calls hit a premium paid API. The result was a monthly AI bill near $10,500 that scaled linearly with usage, an attribution picture with gaps, and infrastructure that no single person could fully see end to end.

The brief was not "make it cheaper." The brief, once I had diagnosed it, was "stop paying frontier prices for work that does not need frontier intelligence, and make the whole system legible."

The approach

I treated the AI bill as an architecture problem, not a vendor-negotiation problem. Five moves did the work:

1. A no-AI pre-filter. Before a single model runs, a cheap formula-based gate disqualifies a large share of inputs using structured signals. The expensive pipeline never touches low-value work.
2. A cascaded model ladder. Each task routes to the cheapest model that can do it: high-volume extraction on small open-source models, mid-tier analysis on a larger open-source model, and only the final high-judgment step on a frontier model, with aggressive prompt caching.
3. A two-tier scoring router. Most cases are clear-cut and route to the cheapest frontier tier; only the genuinely complex minority routes to the pricier one. A computed complexity signal decides, so you never overpay for an easy call.
4. A semantic cache. A similarity cache returns a stored answer for near-duplicate work, skipping the model entirely on a large share of calls.
5. Owned compute for the clerical layers, plus instrumentation. The high-volume cheap models run on the team's own compute, so volume stops driving the bill. Alongside the cost work I closed data leaks, rebuilt the attribution layer so revenue traces back to source, and split the system into clear layers so stakeholders could finally see what was running.

The specific routing logic and the agent design are the part I build inside a paid engagement. The principle is portable: match the cost of the model to the value of the task. The frontier model should do the last 5% of the work, not the first 95%.

The impact

• Monthly AI operating cost: $10,500 to $467. A reduction of more than 95%, with no measured drop in output quality.
• Approximately $120,000 in annualized run-rate savings, recurring rather than one-time.
• An attribution layer the team can read, so GTM spend ties back to outcomes instead of disappearing into a black box.
• Observable infrastructure the team owns and operates, rather than a set of opaque API calls only one person understood.

The savings alone returned many times the cost of the work, before counting the pipeline and attribution gains that came with it.

How this transfers to your team

This is what I mean by building infrastructure you keep. The savings are not a one-time consulting trick that decays the day I leave. They live in an architecture the team owns: a routing pattern that keeps cost matched to value as you scale, and a spend picture you can read without me in the room. When usage doubles, the bill does not.

If your AI or GTM costs are scaling faster than your results, that gap is usually an architecture problem hiding as a vendor problem. That is the kind of problem I take apart.