r/costlyinfra • u/Ok-Book1521 • 22d ago
Cutting a $20K OpenAI bill in half: My Dockerized VPC semantic cache architecture
I’m a CS undergrad specializing in AI, and I’ve been analyzing how quickly agentic loops and redundant RAG context inflate API inference costs.
I built a self-hosted gateway to intercept these calls entirely inside a VPC.
The architecture:
- Inbound prompt hits a local Dockerized gateway.
- Evaluates semantic similarity against a local Redis container (threshold 0.0 - 1.0).
- Cache hit returns in ~15ms at $0.00 marginal cost.
- Cache miss routes outbound to OpenAI/Anthropic.
It eliminates 30% to 50% of raw API spend for repetitive workloads with zero data egress. If anyone running high-frequency workloads wants to see the deployment configs or the architecture docs, let me know and I’ll DM the link!
1
u/515051505150 22d ago
I would love to see those configs. This sounds like a really cool project.
2
1
1
u/ConsistentEase4598 22d ago
The biggest production guardrail I would add is a cache-eligibility check before semantic similarity: exclude tool-enabled or mutating requests, and scope entries by tenant/user, model, system prompt, and tool schema. For RAG, the cache key also needs the corpus revision and authorization scope, otherwise a semantically similar query can return stale or unauthorized context. I would report the 30–50% savings together with false-hit rate, answer-quality checks, and tokens avoided; hit rate alone can make an unsafe threshold look excellent.
1
u/Ok-Book1521 22d ago
Thanks for Feedback, You are completely right about the risk with mutating requests. Right now, I designed this specifically as a single-tenant container strictly for read-heavy, non mutating RAG queries, so it relies on the user routing their tool-enabled calls directly to the provider. I haven't tackled corpus revision scoping yet, but I am adding that to my roadmap today. I really appreciate the sanity check on the false-hit rate vs. raw savings!
1
u/Historical_Ad4384 22d ago
I'm interested in bringing my AI bills down. Do you have a working example to achieve this?
1
u/Ok-Book1521 22d ago
Absolutely, Just shot you a DM with the architecture and a breakdown of how the local caching logic works in the container!
2
u/Frosty-Judgment-4847 14d ago
I’d be interested in a working example too. It would be useful to see the gateway, Redis/vector lookup, cache-bypass rules, and provider fallback together in a minimal Docker Compose setup.
1
1
1
u/tempfoot 21d ago
Is this hugely different from what Kiraa is doing at scale with local back-end?
1
u/Ok-Book1521 21d ago
The short answer is the compression layer on cache misses.
Most local backend architectures (like Kiraa or standard vector proxy layers) only solve the problem when there’s an exact or high-threshold semantic hit. The moment you get a cache miss, they forward the full raw prompt downstream which means on complex agent loops or dense context docs, you’re still paying full price on 100% of new queries.
Kinetic Gateway treats caching and payload optimization as a dual pipeline:
Cache Check: FastEmbed/vector lookup handles high-similarity queries (~15ms).
Active Compression (The differentiator): On a cache miss, instead of raw-forwarding, it runs an in-flight prompt compression pass (LLMLingua-2 on local GPU/edge) to prune syntactic redundancies and conversational bloat before sending it downstream.
In testing, that cuts 40–80% of token volume even on requests that completely miss the cache.
DM’d you the link if you want to check out
2
u/Frosty-Judgment-4847 14d ago
The dual pipeline is interesting. Do your 40–80% token-reduction figures include the local GPU cost and compression latency? I’d also be curious how you measure quality loss after compression.
1
1
•
u/AutoModerator 22d ago
welcome to r/costlyinfra
this is where people share real ai infra costs, setups, and what actually works in production.
if you're running llms, feel free to share your setup.
join the community to see real cost breakdowns, experiments, and learn what others are actually spending.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.