Vectorcache

A semantic cache for LLMs: serve the answer you’ve already paid for.

5
repos · full stack
384-d
vector embeddings
pgvector
+ FastAPI engine
npm
published SDK

A solo build, measured by craft rather than traction. See the sunset note on why I stopped.

The problem & the bet

People ask the same thing in different words. “How do I reset my password?” and “I forgot my password, what do I do?” are different strings but the same question. Traditional caching is exact-match, so it never catches the second one. Every near-duplicate query pays the full LLM bill and full latency again.

Vectorcache’s bet: a semanticcache. Embed each prompt into a vector, search for a previously-answered prompt that’s similar enough, and if one clears a tunable threshold, return the stored answer. No LLM call, milliseconds instead of seconds. Every catch is a call you don’t pay for.

The honest tradeoff: every request pays an embedding and vector-search cost first, so a miss is a little slower and costs a little more than calling the LLM directly. Semantic caching only wins when the hit rate is high enough to pay that tax back. That is exactly why the threshold, not the cache, is the real product.

See it decide

The whole idea lives in one interaction: the same query can be a hit or a miss depending on one dial. Pick a paraphrase, then drag the threshold and watch the LLM step get skipped.

Watch a cache hit happen

interactive simulation · precomputed · not a live LLM call

Already in the cache (asked & answered once):

  • How do I reset my password?
  • What are your business hours?
  • How do I cancel my subscription?

Now a user asks it a different way. Pick one:

0.85

Lower = more hits but riskier (false matches). Higher = safer but fewer hits. This dial is the core product trade-off. Default 0.85.

CACHE HIT89% similar
~45ms$0.0000
  1. 1
    Generate Embedding35ms

    Your text becomes a 384-dimensional vector (all-MiniLM-L6-v2) that captures its meaning.

  2. 2
    Search Cache5ms

    Postgres + pgvector scans cached vectors. An HNSW index keeps this sub-linear.

  3. 3
    Calculate Similarity2ms

    Cosine similarity scores the query against cached entries (0 = unrelated, 1 = identical).

    best match: 89% similar

  4. 4
    Cache Decision1ms

    If the best score clears the threshold, it's a hit: reuse the stored answer.

    Cache HIT: 0.89 ≥ 0.85

  5. 5
    LLM API Call

    On a miss, forward to the LLM (your own key) for a fresh answer. On a hit this is skipped. That's the whole point.

    skipped, served from cache

  6. 6
    Store Result

    A fresh answer is embedded and cached (encrypted) for next time. Skipped on a hit.

    skipped, already cached

  7. 7
    Return Response2ms

    The answer is returned with cache metadata.

Response

Go to Settings → Security → Reset Password, then follow the emailed link. Returned from cache, no LLM call, no token cost.

Architecture at a glance

A thin SDK talks to a FastAPI service that owns the intelligence; the embedding model and vector search run server-side, and customers bring their own LLM key so I never hold their spend.

SDK (HTTP, typed, zero-dep)
  └─► FastAPI service
        ├─ embed prompt ── all-MiniLM-L6-v2 (384-dim, server-side)
        ├─ search ─────── Postgres + pgvector, cosine, HNSW index
        ├─ decide ─────── similarity ≥ threshold ?
        │      HIT ─────► return cached answer   (encrypted at rest)
        │      MISS ────► forward to YOUR LLM key ► cache the result
        └─ log ───────── usage, hit-rate, est. cost saved

Decisions & trade-offs

The interesting part isn’t the feature list. It’s the calls made under real constraints, and what they cost.

01

Encrypt everything (except the embeddings)

Decision
Encrypt customer prompts, responses, and context at rest (Fernet), but deliberately leave the embedding vectors in plaintext.
Trade-off
pgvector similarity search operates on raw vectors, so encrypting them would break the core feature. The sensitive text is protected, but the vectors (a lossy, non-reversible-ish representation) are not.
Outcome
End-to-end encryption of the readable data at a measured ~0.2% latency cost, with the one honest caveat documented in the code rather than hidden.
If I did it again
I'd use a random per-key salt (the code notes the shortcut) and investigate whether embeddings leak enough to warrant vector encryption or an enclave.
02

The similarity threshold as the product's core lever

Decision
Make cache correctness a tunable, per-project threshold rather than a fixed constant, and add a quality gate so the cache never stores junk.
Trade-off
Too low and you serve confident-but-wrong answers (false hits); too high and you rarely hit and save nothing. There's no universally right number: it's a dial the customer owns.
Outcome
Per-project thresholds plus a gate that refuses to cache errors, apologies, and 'I don't know' responses, so a bad answer can't poison future hits.
If I did it again
I'd ship threshold guidance from real hit-rate data rather than a 0.85 default chosen by intuition. The demo on this page exists precisely because this trade-off is the whole product.
03

Assume someone will try to run up the bill (before launch)

Decision
Harden the cost-abuse surface before the beta, not after an incident.
Trade-off
More pre-launch work and complexity, in exchange for not waking up to a runaway compute/LLM bill from a single abusive caller.
Outcome
Dual-layer rate limiting (in-memory per-minute burst check in front of a database monthly quota), a 1MB payload cap, and a bring-your-own-key model that puts LLM spend on the caller's own account.
If I did it again
The in-memory limiter is per-instance, so it wouldn't hold across a horizontally-scaled fleet. I'd move that layer to Redis before scaling out.
04

Fail gracefully when a dependency degrades

Decision
Wrap the fragile dependencies (database, embedding model, LLM) in a circuit breaker with per-dependency thresholds.
Trade-off
A three-state breaker adds moving parts, but stops a slow dependency from cascading into a full outage. It deliberately does NOT trip on client 4xx errors, only real service failures.
Outcome
Fast-fail with a Retry-After and a customer-facing fallback ('consider calling your LLM provider directly while we restore service') instead of a bare error.
If I did it again
Honest limit: it's unit-tested but I never validated it under injected production failure. I'd run a real fault-injection drill before trusting it in anger.

Engineering for trust

For infrastructure people put their prompts (and API keys) through, credibility is the product. Beyond the encryption above: a quality gate that keeps bad answers out of the cache, and a deliberately small, typed, zero-dependency SDK with timeouts, request cancellation, and a real error taxonomy. These are the parts a developer feels within five minutes.

Built end to end

This wasn’t a prototype. It shipped as a full stack: a FastAPI + pgvector engine, a Next.js dashboard with project and key management, a published npm SDK , a documentation site , a quickstart, and three pricing tiers wired through Stripe.

Why I sunset it

I’d validated the two things that were actually uncertain: that the semantic-cache engine worked, and that the developer experience (SDK + docs + quickstart) was clean enough to adopt in minutes. The backlog that remained was mostly undifferentiated plumbing: storage quotas, cache eviction (FIFO), a full TTL lifecycle. Plus a real go-to-market push into a crowded infrastructure category.

Weighing the remaining lift against the signal I’d already gotten, I chose to stop and redeploy the time. It’s the same prioritization call a TPM makes to kill a low-ROI workstream rather than finish it on momentum. The SDK and docs stay published as reference.

An honest note on how it was built

Vectorcache was built solo with heavy AI assistance. What I owned, and what this page is about, is the problem framing, the architecture, the trade-off calls (encrypt-but-not-the- vectors, the threshold-as-a-dial, pre-launch abuse hardening), driving it to a real launch across five repos, and the decision to stop. Those are the parts that don’t come from a code generator.

Poke at the real thing

The SDK and docs are still published. The demo above mirrors the real workflow.

npm docs