model

type Role = 'user' | 'assistant' | 'system'

Chat message role understood by all providers.

user and assistant carry the conversation turns. A system message placed in GenerateRequest.messages is routed to the provider's native system-prompt channel and takes precedence over GenerateRequest.system.

type ContentPart = TextPart | ImagePart | ToolUsePart | ToolResultPart | DocumentPart

Content part accepted in a model message.

Message content is either a plain string or an ordered list of these parts; the string form is shorthand for a single TextPart.

type StopReason = 'end_turn' | 'tool_use' | 'max_tokens' | 'stop_sequence' | 'error' | 'refusal' | 'content_filter'

Reason a model stopped generating.

end_turn is a normal completion. tool_use means the model is waiting on tool results. max_tokens and stop_sequence indicate the output was cut by a limit from the request. error, refusal, and content_filter are abnormal terminations surfaced by the provider.

type StreamEvent = { type: 'text_delta'; index: number; text: string; } | { type: 'tool_call_start'; index: number; id: string; name: string; } | { type: 'tool_call_delta'; index: number; json: string; } | { type: 'tool_call_end'; index: number; } | { type: 'usage'; usage: Usage; } | { type: 'stop'; reason: StopReason; } | { type: 'error'; message: string; }

Streaming event emitted by provider adapters.

text_delta carries an index so parallel content blocks concatenate in the right order. Tool calls arrive as a tool_call_start naming the tool, tool_call_delta events carrying argument-JSON fragments, and a tool_call_end. usage may be emitted more than once; the latest values win. stop reports the stop reason, and error signals an abnormal termination — assembleResult() converts it into a thrown Error.

for await (const event of model.stream({ messages })) {
  if (event.type === 'text_delta') write(event.text);
  else if (event.type === 'stop') console.log('done:', event.reason);
}

type Model = ChatModel

Provider-neutral model adapter used by chat and agent APIs.

Providers may also implement EmbeddingModel; memory and semantic eval APIs depend on that narrower embedding contract instead of requiring every chat model to expose embeddings.

interface TextPart {

Text content part.

The most common part; a plain-string message body is shorthand for a single text part. Use the part form when a message mixes text with images, documents, or tool traffic, or when a span of text should be a prompt-cache breakpoint.

import type { ModelMessage } from 'fino:ai/model';

const message: ModelMessage = {
  role: 'user',
  content: [
    { type: 'text', text: manualText, cache: true },
    { type: 'text', text: 'Which chapter covers installation?' },
  ],
};

Properties

interface ImagePart {

Inline image content part.

data is base64-encoded bytes without a data URI prefix. Send images only to models whose capabilities.input.image is true; providers reject or silently drop modalities they do not accept.

import type { ModelMessage } from 'fino:ai/model';

const message: ModelMessage = {
  role: 'user',
  content: [
    { type: 'image', mediaType: 'image/png', data: screenshotBase64 },
    { type: 'text', text: 'What error does this screenshot show?' },
  ],
};

Properties

interface ToolUsePart {

Model-requested tool call content part.

Appears in assistant messages when replaying conversation history that included tool calls: echo the model's request back as a tool_use part, then answer it with a matching tool_result part in the following user message.

import type { ModelMessage } from 'fino:ai/model';

const history: ModelMessage[] = [
  { role: 'user', content: 'What is the weather in Lisbon?' },
  {
    role: 'assistant',
    content: [{ type: 'tool_use', id: 'call_1', name: 'weather', args: { city: 'Lisbon' } }],
  },
  {
    role: 'user',
    content: [{ type: 'tool_result', toolCallId: 'call_1', content: '19C and sunny' }],
  },
];

Properties

interface ToolResultPart {

Tool result content part sent back to the model.

Sent in a user message after the assistant requested a tool call. Set isError when the tool run failed so the model can recover instead of treating the output as a successful result.

import type { ModelMessage } from 'fino:ai/model';

const reply: ModelMessage = {
  role: 'user',
  content: [{
    type: 'tool_result',
    toolCallId: call.id,
    content: JSON.stringify(rows),
  }],
};

Properties

interface DocumentPart {

Inline document content part.

data is base64-encoded bytes without a data URI prefix. Providers may map this to files, documents, or other native document upload fields. Send documents only to models whose capabilities.input.document is true.

import type { ModelMessage } from 'fino:ai/model';

const message: ModelMessage = {
  role: 'user',
  content: [
    { type: 'document', mediaType: 'application/pdf', data: contractBase64, name: 'contract.pdf' },
    { type: 'text', text: 'List the termination clauses.' },
  ],
};

Properties

interface ResponseFormat {

Native structured response format request.

Adapters translate this to the provider's native structured-output transport: response_format on OpenAI-compatible APIs and output_config on Anthropic. Check capabilities.structuredOutput.native before relying on it; higher layers fall back to prompt-based JSON extraction for models without native support.

const result = await model.generate({
  messages: [{ role: 'user', content: 'Extract the invoice fields.' }],
  responseFormat: {
    type: 'json_schema',
    name: 'invoice',
    schema: {
      type: 'object',
      properties: { total: { type: 'number' }, currency: { type: 'string' } },
      required: ['total', 'currency'],
    },
  },
});
const invoice = JSON.parse(result.text);

Properties

interface ModelMessage {

Provider-neutral chat message.

content is either plain text or an ordered list of content parts for multimodal and tool-carrying turns.

import type { ModelMessage } from 'fino:ai/model';

const messages: ModelMessage[] = [
  { role: 'user', content: 'Summarize the release notes.' },
  { role: 'assistant', content: 'The release adds SSE routes and fixes two loader bugs.' },
  { role: 'user', content: 'Shorter, one sentence.' },
];

Properties

interface ToolDefinition {

Provider-neutral tool definition sent with model requests.

This is the wire shape adapters transmit; higher-level tool registration and dispatch live in fino:ai/tool. parameters is a JSON Schema object describing the tool's arguments.

import type { ToolDefinition } from 'fino:ai/model';

const weather: ToolDefinition = {
  name: 'weather',
  description: 'Look up current weather for a city.',
  parameters: {
    type: 'object',
    properties: { city: { type: 'string' } },
    required: ['city'],
  },
};

Properties

interface GenerateRequest {

Provider-neutral generation request.

The same request shape is accepted by Model.generate() and Model.stream(). Sampling fields set here override any defaults configured on the model at construction time.

const controller = new AbortController();
const result = await model.generate({
  system: 'You are a terse release-notes editor.',
  messages: [{ role: 'user', content: draft }],
  maxTokens: 512,
  temperature: 0,
  signal: controller.signal,
});

Properties

interface Usage {

Token usage reported by a provider.

The cache* fields describe the provider's native prompt cache; the localCache* fields describe requests answered by a local Fino response cache (fino:ai/cache) without touching the provider at all. Optional fields are present only when the provider or cache reported them.

const result = await model.generate({ messages });
const { inputTokens, outputTokens, cacheReadInputTokens = 0 } = result.usage;
recordCost(inputTokens - cacheReadInputTokens, cacheReadInputTokens, outputTokens);

Properties

interface ToolCall {

Assembled tool call returned by generate() or ModelStream.result().

This is the folded form of the incremental tool_call_start / tool_call_delta / tool_call_end stream events, with args already JSON-parsed. Answer each call with a tool_result part and continue the conversation.

const result = await model.generate({ messages, tools });
if (result.stopReason === 'tool_use') {
  for (const call of result.toolCalls) {
    const output = await dispatch(call.name, call.args);
    messages.push(
      { role: 'assistant', content: [{ type: 'tool_use', ...call }] },
      { role: 'user', content: [{ type: 'tool_result', toolCallId: call.id, content: output }] },
    );
  }
}

Properties

interface GenerateResult {

Complete non-streamed model result.

Returned by Model.generate() and produced by folding a stream through ModelStream.result() or assembleResult() — all three paths yield this same shape.

const result = await model.generate({ messages, maxTokens: 256 });
if (result.stopReason === 'max_tokens') {
  console.warn('Output truncated after', result.usage.outputTokens, 'tokens');
}
console.log(result.text);

Properties

interface ModelStreamState {

Current retained view of a model stream.

This is a lossy read model for UIs and progress meters. Use the stream itself when every provider event must be processed. The state advances only as events are pulled from the stream, so something must be consuming it.

const stream = model.stream({ messages });
const unsubscribe = stream.state.subscribe(({ text }) => render(text));
const result = await stream.result();
unsubscribe();

Properties

interface ModelCapabilities {

Feature metadata exposed by chat model adapters.

Capabilities are semantic flags, not provider names. Agent code uses these fields to decide whether to request tools, native JSON Schema output, multimodal input, and optional sampling controls. Every field is optional; an absent field means the adapter made no claim, so treat it as unknown rather than unsupported.

const nativeJson = model.capabilities?.structuredOutput?.native === true;
const result = await model.generate({
  messages,
  ...(nativeJson ? { responseFormat: { type: 'json_schema', schema } } : {}),
});

Properties

interface EmbeddingCapabilities {

Feature metadata exposed by embedding model adapters.

const batchSize = embedder.capabilities?.maxBatchSize ?? 64;
for (const batch of chunk(texts, batchSize)) {
  vectors.push(...await embedder.embed(batch));
}

Properties

interface ModelStream extends AsyncIterable<StreamEvent> {

Async stream of model events.

Iterate it to process every provider event, or call result() to drain the remaining events and receive the folded GenerateResult. The state signal is a retained, lossy view for UIs; it only advances while events are being pulled.

const stream = model.stream({ messages });
for await (const event of stream) {
  if (event.type === 'text_delta') write(event.text);
}
const { toolCalls, usage } = await stream.result();

Readonly Properties

Methods

interface ChatModel {

Provider-neutral chat model adapter.

This is the contract agents and evals consume; anthropic(), openai(), and local() all return implementations of it. Provider HTTP failures surface as ModelError.

import { anthropic } from 'fino:ai/model';

const model = anthropic();
const result = await model.generate({
  messages: [{ role: 'user', content: 'Name three prime numbers.' }],
  maxTokens: 128,
});
console.log(result.text);

Readonly Properties

Methods

interface EmbeddingModel {

Provider-neutral embedding model adapter.

Kept separate from ChatModel so chat-only providers and test doubles do not need fake embedding methods. Memory and semantic-eval APIs accept this narrower contract.

import type { EmbeddingModel } from 'fino:ai/model';

async function indexChunks(embedder: EmbeddingModel, chunks: string[]) {
  const vectors = await embedder.embed(chunks);
  for (let i = 0; i < chunks.length; i++) {
    await store.put(chunks[i], vectors[i]);
  }
}

Readonly Properties

Methods

interface ModelCreateOptions {

Options applied when constructing a Model from a provider or registry.

These become per-model defaults: each GenerateRequest can still override the sampling values for an individual call.

const model = await registry.create('gpt-4o-mini', {
  maxTokens: 1024,
  temperature: 0,
});

Properties

interface ModelInfo {

Discovered model entry returned by a ModelProvider.

create() constructs a provider-neutral Model for this exact model id using the provider configuration that discovered it.

const models = await registry.list({ provider: 'anthropic' });
const info = models.find((m) => m.capabilities?.toolCalling);
if (!info) throw new Error('No tool-calling model available');
const model = await info.create({ temperature: 0.2 });

Readonly Properties

Methods

interface ModelProvider {

Provider that can discover available model ids and construct Model adapters for those ids.

Providers are usually consumed through a ModelRegistry, but they can be used directly when only one provider is in play.

import { anthropicProvider } from 'fino:ai/model';

const provider = anthropicProvider();
const [info] = await provider.listModels();
const model = await provider.createModel(info.id, { maxTokens: 2048 });

Readonly Properties

Methods

interface ProviderOptions {

Common provider options accepted by bundled providers.

All fields are optional: API keys fall back to environment variables and baseUrl defaults to the provider's public endpoint. Point baseUrl at any compatible server — the OpenAI adapter works against OpenAI-compatible endpoints such as vLLM or llama-server.

import { openai } from 'fino:ai/model';

const model = openai({
  baseUrl: 'http://localhost:8000/v1',
  apiKey: 'local',
  model: 'llama-3.1-8b-instruct',
  maxTokens: 2048,
});

Properties

class ModelRegistry {

Registry for discovering and constructing provider-neutral models.

A registry aggregates several ModelProvider instances behind one lookup surface so application code can list, find, and construct models without caring which provider serves them. listModels() results are cached per provider instance; pass refresh: true to bypass the cache for a call.

import { anthropicProvider, modelRegistry, openaiProvider } from 'fino:ai/model';

const registry = modelRegistry([openaiProvider(), anthropicProvider()]);
for (const info of await registry.list()) {
  console.log(info.provider, info.id);
}
const model = await registry.create('claude-opus-4-8', { temperature: 0.2 });

Constructors

Methods

function modelRegistry(providers: ModelProvider[] = []): ModelRegistry

Create a model registry from one or more providers.

Convenience wrapper around new ModelRegistry(providers).

import { modelRegistry, openaiProvider } from 'fino:ai/model';

const registry = modelRegistry([openaiProvider()]);
const model = await registry.create('gpt-4o-mini');

function assembleResult(events: AsyncIterable<StreamEvent>): Promise<GenerateResult>

Assemble streamed provider events into a complete GenerateResult.

Text deltas are concatenated per content index and joined in index order, tool-call argument fragments are JSON-parsed (falling back to {} when the JSON is empty or malformed), and the latest usage and stop values win. Throws when the stream emits an error event. ModelStream.result() uses this internally, so calling it directly is only needed for bare event iterables such as recorded or transformed streams.

import { assembleResult } from 'fino:ai/model';

const result = await assembleResult(recordedEvents());
console.log(result.text, result.stopReason);

function anthropic(opts: ProviderOptions = {}): Model

Create an Anthropic-backed Model.

apiKey defaults to ANTHROPIC_API_KEY (throws when neither is set), baseUrl defaults to the public Anthropic API, and model defaults to claude-opus-4-8.

import { anthropic } from 'fino:ai/model';

const model = anthropic({ model: 'claude-opus-4-8', maxTokens: 4096 });
const result = await model.generate({
  messages: [{ role: 'user', content: 'Explain kqueue in two sentences.' }],
});

function anthropicProvider(opts: ProviderOptions = {}): ModelProvider

Create an Anthropic provider that can list and construct models.

apiKey defaults to ANTHROPIC_API_KEY. listModels() calls GET /v1/models on baseUrl and infers capabilities per model id.

import { anthropicProvider } from 'fino:ai/model';

const provider = anthropicProvider();
const models = await provider.listModels();
const model = await provider.createModel(models[0].id);

function openai(opts: ProviderOptions = {}): Model

Create an OpenAI-backed Model.

apiKey defaults to OPENAI_API_KEY (throws when neither is set), baseUrl defaults to the public OpenAI API, and model defaults to gpt-4o. Point baseUrl at any OpenAI-compatible server such as vLLM or llama-server. The returned model also implements EmbeddingModel backed by the OpenAI embeddings endpoint.

import { openai } from 'fino:ai/model';

const model = openai({ model: 'gpt-4o-mini' });
const stream = model.stream({
  messages: [{ role: 'user', content: 'Stream a limerick about io_uring.' }],
});
for await (const event of stream) {
  if (event.type === 'text_delta') write(event.text);
}

function openaiProvider(opts: ProviderOptions = {}): ModelProvider

Create an OpenAI-compatible provider that can list and construct models.

apiKey defaults to OPENAI_API_KEY. listModels() calls GET /models on baseUrl; servers without a models endpoint make it throw ModelListingUnsupportedError.

import { openaiProvider } from 'fino:ai/model';

const provider = openaiProvider({ baseUrl: 'http://localhost:8000/v1', apiKey: 'local' });
const models = await provider.listModels();
const model = await provider.createModel(models[0].id);

const ModelError

Error thrown for provider HTTP failures.

Carries the HTTP status, the raw response body when available, and retryAfterMs parsed from a Retry-After header — useful for backoff on 429 and 5xx responses.

import { ModelError } from 'fino:ai/model';

try {
  await model.generate({ messages });
} catch (err) {
  if (err instanceof ModelError && err.status === 429) {
    await delay(err.retryAfterMs ?? 1000);
  } else {
    throw err;
  }
}

const ModelListingUnsupportedError

Error thrown when a provider endpoint does not support model listing.

Raised by ModelProvider.listModels() when the models endpoint responds with 404 — common for OpenAI-compatible servers that implement chat but not discovery. Carries the provider name, HTTP status, and response body.

const hasLlamaCpp

Whether a default system libllama was found when this module was evaluated.

This is a capability gate for optional local model support. It reflects only default lookup paths such as LLAMA_CPP_LIBRARY and FINO_LLAMA_LIBRARY; callers can still pass an explicit libraryPath to local().

import { anthropic, hasLlamaCpp, local } from 'fino:ai/model';

const model = hasLlamaCpp
  ? await local({ model: './models/qwen2.5-0.5b-instruct-q4_k_m.gguf' })
  : anthropic();

const LocalModelLibraryError

Error thrown when local llama.cpp support is requested but no usable libllama library is available.

Carries the libraryPath that failed to load, when one was given. Check hasLlamaCpp first to avoid the throw on default lookup paths.

const LocalModelUnsupportedError

Error thrown for local model request features that the llama.cpp adapter does not implement.

Currently raised for requests that use tools, toolChoice, or responseFormat, and for embed() calls on local models.

const local

Create a local llama.cpp-backed Model.

The model source can be a local GGUF path or an explicit Hugging Face GGUF file. Construction is async because Hugging Face sources may need to be downloaded into the local cache before llama.cpp opens them. Throws LocalModelLibraryError when no usable libllama can be loaded. Close long-lived models with model.close?.() to release the llama.cpp handle.

import { local } from 'fino:ai/model';

const model = await local({
  model: { repo: 'Qwen/Qwen2.5-0.5B-Instruct-GGUF', file: 'qwen2.5-0.5b-instruct-q4_k_m.gguf' },
  contextSize: 8192,
  gpuLayers: 99,
});
const result = await model.generate({
  messages: [{ role: 'user', content: 'Write a haiku about rivers.' }],
});

const localProvider

Create a local provider for configured GGUF models.

Local providers do not discover remote model catalogs. listModels() returns the configured entries so registries can present local and remote models through one interface.

import { localProvider, modelRegistry } from 'fino:ai/model';

const registry = modelRegistry([
  localProvider({
    models: [
      { id: 'qwen-0.5b', model: './models/qwen2.5-0.5b-instruct-q4_k_m.gguf' },
    ],
  }),
]);
const model = await registry.create('qwen-0.5b');