js/ai/cache

js/ai/cache.ts

fino:ai/cache — exact and semantic cache wrappers for chat models.

cachedModel() wraps a provider-neutral Model and serves repeated requests from a local cache before calling the underlying provider. Two tiers are checked in order:

Both generate() and stream() are cached. A streamed miss forwards provider events unchanged while recording them; a hit replays an equivalent coalesced stream (one text delta, tool-call event groups, then usage and stop) rather than the original chunk boundaries.

Caveats

Semantic thresholds are domain-specific. Time-sensitive answers should pass a bypass predicate, and tool-calling turns bypass semantic hits by default because tool context usually needs exact matching.

Cache hits report provider spend as zero and put avoided tokens in localCacheReadInputTokens and localCacheReadOutputTokens. Every result is stamped with providerMetadata.finoCache so callers can observe hit rates: { hit: true, tier, score? } on hits, { hit: false, tier: null } on misses.

import { memoryCache } from 'fino:cache';
import { cachedModel } from 'fino:ai/cache';
import { openai } from 'fino:ai/model';

const base = openai({ model: 'gpt-4o' });
const model = cachedModel(base, { cache: memoryCache(), ttlMs: 60_000 });
const result = await model.generate({ messages: [{ role: 'user', content: 'hello' }] });

Interfaces

interface SemanticCacheOptions {

Semantic cache configuration for cachedModel().

The semantic tier only runs when the exact tier misses, and it is skipped entirely — for both lookup and writes — on requests with tool context (declared tools, or tool_use/tool_result message parts).

By default the index of embeddings and results lives in cache under a reserved key and is scanned with brute-force cosine similarity, which suits small working sets. Setting path adds a persistent SQLite store queried via sqlite-vec KNN; when the sqlite-vec extension is unavailable or the embedder reports zero dimensions, lookup silently falls back to the in-cache index.

import { memoryCache } from 'fino:cache';
import { cachedModel } from 'fino:ai/cache';
import { openai } from 'fino:ai/model';

const embedder = openai({ model: 'gpt-4o' });
const model = cachedModel(openai({ model: 'gpt-4o' }), {
  cache: memoryCache(),
  semantic: {
    cache: memoryCache(),
    embedder,
    threshold: 0.9,
    path: './semantic-cache.db',
  },
});

Properties

cache: Cache

Cache holding the semantic index of embeddings and cached results.

embedder: EmbeddingModel

Embedding model used to compare request text.

threshold?: number

Minimum similarity score for a semantic hit. Defaults to 0.92.

path?: string

Optional SQLite database path for sqlite-vec backed semantic lookup.

fs?: FileSystem

Optional filesystem provider for the semantic SQLite store.

interface CachedModelOptions {

Options for cachedModel().

Only cache is required; it backs the always-on exact tier. Add semantic for similarity-based reuse and bypass to exempt requests that must always reach the provider (time-sensitive prompts, per-user answers, and so on).

import { memoryCache } from 'fino:cache';
import { cachedModel } from 'fino:ai/cache';
import { openai } from 'fino:ai/model';

const model = cachedModel(openai({ model: 'gpt-4o' }), {
  cache: memoryCache(),
  ttlMs: 5 * 60_000,
  bypass: (req) => req.messages.some(
    (m) => typeof m.content === 'string' && m.content.includes('today'),
  ),
});

Properties

cache: Cache

Exact cache backend.

ttlMs?: number

TTL applied to stored exact entries and to the in-cache semantic index. Rows in a path-backed semantic SQLite store are not expired.

semantic?: SemanticCacheOptions

Optional semantic tier.

bypass?: (req: GenerateRequest) => boolean

Return true to bypass all cache lookup and writes for a request. Bypassed requests always hit the provider and are never stored.

Functions

function cachedModel(base: Model, opts: CachedModelOptions): Model

Wrap a chat model with exact and optional semantic local caching.

The returned model preserves the identity of base (id, name, provider, capabilities) and caches both generate() and stream(). Each request first checks the exact tier, then the semantic tier when configured; a hit returns the stored result with usage zeroed and avoided tokens reported as localCacheReadInputTokens / localCacheReadOutputTokens. On a miss the provider result is stored and returned stamped with providerMetadata.finoCache = { hit: false, tier: null }, so the same conversation replayed later — even with request properties in a different order — is served locally.

Streamed hits replay a coalesced event stream synthesized from the cached result: a single text delta, start/delta/end groups per tool call, then usage and stop events. Streamed misses forward the provider's events unmodified while recording the final result for future hits.

Because the cache key includes the model identity, wrappers around different models can safely share one Cache instance.

import { memoryCache } from 'fino:cache';
import { cachedModel } from 'fino:ai/cache';
import { openai } from 'fino:ai/model';

const model = cachedModel(openai({ model: 'gpt-4o' }), {
  cache: memoryCache(),
  ttlMs: 60_000,
});

const req = { messages: [{ role: 'user' as const, content: 'What is TAP output?' }] };
const first = await model.generate(req);   // calls the provider
const second = await model.generate(req);  // served locally
console.log(second.providerMetadata?.finoCache); // { hit: true, tier: 'exact' }
console.log(second.usage.localCacheReadOutputTokens); // tokens avoided