local

type LocalModelSource = string | { path: string; } | { repo: string; file: string; revision?: string; }

Source accepted by the local llama.cpp adapter.

A plain string or the { path } form names a GGUF file on disk. The { repo, file, revision } form names an explicit GGUF file in a Hugging Face repository, downloaded into the cache directory on first use (revision defaults to main). Every form must end in .gguf — the adapter refuses to guess quantizations or resolve non-GGUF artifacts and throws for anything else.

interface LocalModelOptions extends ModelCreateOptions {

Options for creating a local llama.cpp-backed model.

Inherited ModelCreateOptions fields act as generation defaults: requests that omit maxTokens or temperature fall back to the values given here, then to the module defaults (512 tokens, temperature 0.8).

import { local } from 'fino:ai/model/local';

const model = await local({
  model: {
    repo: 'bartowski/SmolLM2-135M-Instruct-GGUF',
    file: 'SmolLM2-135M-Instruct-Q2_K.gguf',
  },
  cacheDir: '/tmp/fino-models',
  contextSize: 2048,
  threads: 4,
  maxTokens: 64,
  temperature: 0,
  stopSequences: ['\n'],
});

Properties

interface LocalModelProviderEntry {

Configured model entry exposed by localProvider().

Entries are static configuration: listModels() reports each one as a ModelInfo row, and the model's GGUF source is only resolved and opened when the entry is instantiated through createModel() or ModelInfo.create().

import type { LocalModelProviderEntry } from 'fino:ai/model/local';

const entry: LocalModelProviderEntry = {
  id: 'smol',
  model: { repo: 'bartowski/SmolLM2-135M-Instruct-GGUF', file: 'SmolLM2-135M-Instruct-Q2_K.gguf' },
  displayName: 'SmolLM2 135M',
};

Properties

interface LocalProviderOptions extends Omit<LocalModelOptions, 'model'> {

Options for a local provider.

Everything except models is shared LocalModelOptions configuration — library path, cache directory, sampling defaults, and so on — applied to every model the provider creates. Per-call ModelCreateOptions passed to createModel() layer on top of these shared options.

import { localProvider } from 'fino:ai/model/local';

const provider = localProvider({
  cacheDir: '/tmp/fino-models',
  contextSize: 2048,
  models: [{ id: 'tiny', model: '/models/tiny.Q4_K_M.gguf' }],
});

Properties

class LocalModelLibraryError extends Error {

Error thrown when libllama cannot be loaded.

local() throws this when an explicit libraryPath fails to open, or when no libraryPath is given and default discovery found no libllama (that is, hasLlamaCpp is false).

import { local, LocalModelLibraryError } from 'fino:ai/model/local';

try {
  await local({ model: '/tmp/model.gguf', libraryPath: '/opt/llama/lib/libllama.so' });
} catch (err) {
  if (err instanceof LocalModelLibraryError) {
    console.error(`libllama unavailable at ${err.libraryPath}`);
  }
}

Properties

Constructors

class LocalModelUnsupportedError extends Error {

Error thrown for model request features that the local adapter does not support.

Raised for tool definitions or any explicit toolChoice, native responseFormat requests, non-text content parts (images, documents), and embed() calls. Catch it to fall back to a remote provider for requests that need those capabilities.

import { local, LocalModelUnsupportedError } from 'fino:ai/model/local';

const model = await local({ model: './models/tiny.Q4_K_M.gguf' });
try {
  await model.generate({
    messages: [{ role: 'user', content: 'What is the weather?' }],
    tools: [{ name: 'weather', description: 'Look up weather', parameters: {} }],
  });
} catch (err) {
  if (err instanceof LocalModelUnsupportedError) {
    // route this request to a remote model instead
  }
}

Constructors

const hasLlamaCpp

Whether the default libllama was found during module evaluation.

This boolean only reflects the default lookup paths. Passing libraryPath to local() can still succeed when this is false.

async function local(opts: LocalModelOptions): Promise<Model>

Create a llama.cpp-backed local model.

Construction resolves the model source, downloads Hugging Face GGUF files into the cache when necessary, loads the optional libllama, and opens the model. Requests fall back to the option defaults given here, then to the module defaults (512 max tokens, temperature 0.8, top-k 40, top-p 0.95); a temperature of 0 or below switches to deterministic greedy sampling. Concurrent generate() and stream() calls are serialized on the single llama.cpp context. Close long-lived models with model.close?.() when your application is done with them.

Throws LocalModelLibraryError when libllama cannot be loaded, and a plain Error when the source is not a .gguf file, a Hugging Face download fails, or the model itself fails to open.

import { local } from 'fino:ai/model/local';

const model = await local({
  model: {
    repo: 'bartowski/SmolLM2-135M-Instruct-GGUF',
    file: 'SmolLM2-135M-Instruct-Q2_K.gguf',
  },
  contextSize: 2048,
  maxTokens: 64,
});

for await (const event of model.stream({
  messages: [{ role: 'user', content: 'Tell me a joke.' }],
})) {
  if (event.type === 'text_delta') console.log(event.text);
}

function localProvider(opts: LocalProviderOptions = {}): ModelProvider

Create a local provider for configured GGUF models.

listModels() returns the static models entries supplied here. It does not scan local directories or query Hugging Face. createModel(id) — or the create() callback on a listed ModelInfo — opens the entry's GGUF source through local() with the provider's shared options, and throws when id was not configured.

import { localProvider } from 'fino:ai/model/local';

const provider = localProvider({
  cacheDir: '/tmp/fino-models',
  models: [
    {
      id: 'smol',
      model: {
        repo: 'bartowski/SmolLM2-135M-Instruct-GGUF',
        file: 'SmolLM2-135M-Instruct-Q2_K.gguf',
      },
      displayName: 'SmolLM2 135M',
    },
  ],
});

const [info] = await provider.listModels();
const model = await info.create({ maxTokens: 128 });