blog
Introducing 4-Layer Caching for all PVYai Models
The 0.07% Problem: Why Most AI Bills Are 10x Too High — And How We Fixed It
A true story about 2.7 billion tokens, five hidden cache-killers, and the four-layer caching architecture that now protects every model on our platform — automatically.
The Discovery That Changed Everything
When we audited our AI infrastructure at PVY.swiss, we uncovered something that should make every AI operator pause. Across 60 days of internal usage — architect agents, coding assistants, research models — our systems had processed 2.7 billion input tokens. The cache hit rate? 0.07%.
Not 7%. Zero-point-zero-seven.
We were paying full price for essentially every token, every request, every day. And here's the uncomfortable truth about the wider industry: almost nobody measures this. Most teams don't even know their hit rate, let alone optimize it.
We did measure. And then we did something about it.
Our first step came in Spring this year, when we introduced token caching — initially scoped exclusively to Anthropic AI models, since it was the only external commercial Provider we took in. That early work proved the concept and gave us our first real look at the numbers. But as we dug into the token usage of our own locally hosted and GPU-served models, and the ones we proxy to Moonshot AI and Z.ai, one thing became clear: a single-provider fix wasn't going to cut it.
So we stepped back and asked the harder question: how do we architect this holistically — so that caching covers every model on the platform, not just one Model or particular vendor? That meant designing it so our PVYai Sentinel safety layer and PVYai Memory system stay fully intact and unaffected, and so that our amazing native tool calls are cached too — not just the conversation text.
The result is a caching architecture that applies uniformly across local models, GPU-hosted models, and cloud-proxied models — and extends automatically to every new model we deploy. No per-model configuration. No remembering to enable it. It's simply how the platform works now.
The Problem: Same Prompts, Full Price, Every Time
Here's how modern LLM conversations actually work. Your AI coding assistant doesn't just send your latest question to the model. It sends everything: the system prompt defining the agent's persona, the tool definitions, the attached files, and the entire conversation history, the harness. In a long-run SWE Task, our architect agent resends roughly 250,000 tokens with every single message — and 99% of that is identical to what was sent one message earlier.
The major providers know this, and they offer a solution: prompt caching. Anthropic caches marked prefixes and charges cache hits at a fraction of normal rates. OpenAI and Z.ai cache repeated prefixes implicitly. PVYai Infrastructure reuse KV-caches on your own GPUs for dramatically faster responses.
So caching exists everywhere. Why do so many teams still measure 0.07%?
Because caching is fragile. Every provider caches a byte-identical prefix. Change one character anywhere early in the prompt — inject a timestamp, append a warning message, reorder a tool — and the entire cache invalidates. The request goes through as if it were brand new. Full price. Again.
And here's the part most providers won't tell you: many don't cache at all. A significant number of OpenRouter-style AI providers pass your tokens through with zero caching, zero reuse, zero savings. If you're routing through them, you're paying full price on every single request — and you may not even realize it.
Where It Breaks: The Multi-Client Reality
The problem compounds when, like us, you run many AI clients against many models: a web chat UI, coding agents in OpenCode, third-party tools speaking the OpenAI-compatible protocol — routed to cloud models (Anthropic Claude, Z.ai GLM, Moonshot Kimi) and our own or post-trained models on our GPU infrastructure in the datacenter.
Each layer introduces its own cache killers:
- Safety filters that append dynamic warning messages into the prompt — "this action looks risky" — right into the cache-critical prefix. On requests where nothing is wrong, the prefix still changes shape, because sometimes the message is there, sometimes it isn't.
- Memory systems injecting retrieved context with timestamps ("user said this on [date]") into position zero.
- Usage tracking that simply drops the cache statistics providers return. We had caching enabled for months, saw zero hits in our dashboards, and only deep inspection revealed the streaming path discarded the very fields that would have shown the hits.
- Tool catalogs refreshed on a timer, silently rewriting schemas mid-conversation.
Our audit found five distinct structural cache breakers in our own middleware. Yours probably has them too.
The Solution: Four Layers, One Discipline — For Every Model
We rebuilt our platform — PVYai, with a unified OpenAI-compatible API — around a simple principle: the prompt prefix must be stable within a session, and cache performance must be measured, not assumed.
And here's what makes our approach genuinely different: this isn't a single-provider optimization. Our four-layer caching applies holistically to every model on the platform — models running locally, models served on our GPUs, and models reached through our cloud proxy to Moonshot AI, Z.ai, and Ollama Cloud. And it applies automatically to any new model we deploy, with no per-model configuration required. You don't have to remember to enable it. It's simply how the platform works.
Layer 1 — Provider deep-caching, everywhere. Requests now carry cache breakpoints not just on the system prompt but across tools and the growing conversation history, so each new message reuses the computation of everything before it. This works across Anthropic, OpenAI, Z.ai, Moonshot, and our self-hosted models — one discipline, applied uniformly.
Layer 2 — Observability. Every response's cached-token count — from OpenAI, Z.ai, or Anthropic, each reporting it differently — is normalized into one standard field, stored per message, and aggregated into a dedicated Caching dashboard: hit rate per model, per user, per time period, with estimated savings. You cannot fix what you don't measure. Our 0.07% baseline is now the number every future change is judged against.
Layer 3 — Prefix stability. Our Sentinel safety layer now uses a static placeholder slot: when no warning is needed, a fixed marker occupies exactly the same position a warning would — byte-identical on every request. Dynamic content only appears when it must. Cache stays valid; safety stays intact.
Layer 4 — Self-hosted tuning. Local and GPU-served models get long keep-alive windows so their KV-caches persist between requests, cutting time-to-first-token on the infrastructure we already own.
One principle ruled the whole refactor: never break working features for theoretical gains. Position-zero system messages? Tested for weeks across every client we could find — untouched. A proposed optimization to our MCP tool-cache TTL? Dropped after code review showed it solved a problem that didn't exist while risking stale tool bindings. Stability beats cleverness.
The Result
With stable prefixes and deep caching, our architect-class workload projects from ~$440/month uncached to $90–130/month — a 70% reduction — before counting the latency win on self-hosted models.
Quality of long sessions improves too: a quality-threshold setting per model now detects when a context window fills past the point where output degrades (~65% on long-context models) and triggers a clean session handover instead of quietly getting worse.
The dashboard shows it happening. Cache hit rate: climbing, request by request. A Yes, we crafted a new elegant Dashboard for PVYai too, so any PVYai System Administrator can review the stats in ease of use and beautiful manner.
Why This Matters — And Why We're Ahead
Prompt caching is not a feature you enable. It's a property you maintain — across every filter, every memory injection, every tool refresh, every client. And the reality is that most of the market hasn't caught up:
- Many AI providers don't cache at all. If you're routing through an OpenRouter-style aggregator, there's a good chance your tokens are passing through uncached — full price, every time.
- Most teams don't measure. They can't tell you their hit rate, because they've never looked.
- Single-provider caching is fragile. Even teams that do enable caching often see it silently invalidated by their own middleware.
At PVY.swiss, we've built caching as a platform property, not a per-model afterthought. It covers every model we run — local, GPU, and cloud-proxied — and it extends automatically to every model we deploy next. That's not a feature. That's an architecture.
Measure your hit rate first. If it looks like our 0.07%, there's a very good chance a large part of your AI bill is paying twice for the same tokens.
Want to see how PVYai's four-layer caching can cut your AI infrastructure costs? Get in touch with the PVY.swiss team.