eval

interface EvalCase< In = unknown, Out = unknown > {

One evaluation case: a named input plus an optional expected output.

name becomes the test name inside the suite that evaluate() registers. expected feeds scorers that compare against a reference value (exactMatch, semanticSimilarity, llmJudge); scorers that only inspect the output ignore it. Extra properties are allowed and travel with the case into every scorer, so metadata such as tags or difficulty can inform custom scorers.

import type { EvalCase } from 'fino:ai/eval';

const c: EvalCase<string, string> = {
  name: 'refund policy',
  input: 'What is the refund window?',
  expected: '30 days',
  difficulty: 'easy',
};

Properties

interface ScoreResult {

Result produced by a scorer for one case.

value is the numeric score, conventionally in [0, 1]. pass records the scorer's own verdict, but evaluate() decides case pass/fail from the mean of scorer values against its threshold — the per-scorer flag is reporting metadata, not the test assertion.

import type { ScoreResult } from 'fino:ai/eval';

const result: ScoreResult = {
  value: 0.5,
  pass: false,
  explanation: 'answer omitted the refund window',
};

Properties

interface EvalCaseReport< In = unknown, Out = unknown > {

Report for one evaluated case, delivered to EvalReporter.onCase().

scores is keyed by scorer name and holds each scorer's normalized result; score is the mean of those values and pass records whether the mean reached the evaluation threshold.

import { EvalReporter, type EvalCaseReport } from 'fino:ai/eval';

class FailureLogger extends EvalReporter {
  override async onCase(r: EvalCaseReport): Promise<void> {
    if (!r.pass) console.log(`${r.name} scored ${r.score}`, r.scores);
  }
}

Properties

interface EvalSummary {

Aggregate result for an evaluation suite, delivered to onFinish().

mean averages the per-case mean scores, passed counts cases whose score cleared the threshold, and total is the number of cases in the suite.

import { EvalReporter, type EvalSummary } from 'fino:ai/eval';

class SummaryLogger extends EvalReporter {
  override async onFinish(s: EvalSummary): Promise<void> {
    console.log(`${s.name}: ${s.passed}/${s.total} passed, mean ${s.mean.toFixed(3)}`);
  }
}

Properties

interface EvalProgress {

Snapshot of evaluation progress published by EvalProgressReporter.

completed counts cases scored so far, passed counts those that cleared the threshold, and mean is the running average score over completed cases. After the suite finishes, the snapshot matches the final summary with completed equal to total.

import { EvalProgressReporter, type EvalProgress } from 'fino:ai/eval';

const reporter = new EvalProgressReporter();
reporter.progress.subscribe((p: EvalProgress) => {
  console.log(`${p.completed}/${p.total} done, mean ${p.mean.toFixed(3)}`);
});

Properties

interface JsonEvalReport {

JSON-serializable evaluation report produced by JsonEvalReporter.toJSON().

suite appears once onStart() has fired and summary once onFinish() has; cases is always present and sorted by case name so serialized reports diff cleanly between runs.

import { JsonEvalReporter, type JsonEvalReport } from 'fino:ai/eval';

const reporter = new JsonEvalReporter();
// ... run an evaluation with `report: reporter` ...
const report: JsonEvalReport = reporter.toJSON();
console.log(report.summary?.mean, report.cases.length);

Properties

interface JsonEvalReporterOptions {

Options for JsonEvalReporter.

File output only happens when both path and fs are provided; otherwise the report is available in memory through toJSON().

import { JsonEvalReporter } from 'fino:ai/eval';
import { DiskFileSystem } from 'fino:file';

const reporter = new JsonEvalReporter({
  path: './eval-report.json',
  fs: new DiskFileSystem(),
});

Properties

interface EvalOptions< In = unknown, Out = unknown > {

Options for evaluate().

import { evaluate, exactMatch, contains, type EvalOptions } from 'fino:ai/eval';

const opts: EvalOptions<string, string> = {
  name: 'city extraction',
  target: async (input) => extractCity(input),
  cases: [{ name: 'simple', input: 'I live in Paris', expected: 'Paris' }],
  scorers: { exact: exactMatch(), mentionsCity: contains('Paris') },
  threshold: 1,
};
evaluate(opts);

Properties

interface OpenTelemetryReporterOptions {

Options for OpenTelemetryReporter.

Provide exporter (a pre-built exporter instance) or endpoint (an OTLP/HTTP JSON collector URL) to have the reporter start and own a dedicated OpenTelemetry SDK for the run. With neither, the reporter emits through whatever logger and meter providers are already registered.

import { OpenTelemetryReporter } from 'fino:ai/eval';

const reporter = new OpenTelemetryReporter({
  endpoint: 'http://localhost:4318',
  headers: { authorization: `Bearer ${token}` },
});

Properties

type Scorer<Out = unknown> = ( output: Out, evalCase: EvalCase<unknown, Out> ) => number | ScoreResult | Promise<number | ScoreResult>

Function that scores one evaluation output.

Receives the target's output and the full case (including expected and any extra metadata). It may return a bare number — which evaluate() normalizes into a ScoreResult whose pass is value >= threshold — or a complete ScoreResult, synchronously or as a promise. A scorer that throws fails the case's test outright.

When scorers are passed to evaluate() as an array, the reported scorer name comes from a scorerName property on the function (all built-ins set one), falling back to the function's name. Passing a Record<string, Scorer> names each scorer by its key instead.

import type { Scorer } from 'fino:ai/eval';

const noApologies: Scorer<string> = (output) => ({
  value: output.includes('sorry') ? 0 : 1,
  pass: !output.includes('sorry'),
  explanation: 'responses should not apologize',
});

abstract class EvalReporter {

Base class for evaluation report sinks.

evaluate() calls onStart() just before the first case executes, onCase() after each case is scored, and onFinish() from the trailing summary test. Every hook is awaited, so a slow reporter delays the tests it observes. The base implementations are no-ops; subclasses override only the hooks they care about.

import { EvalReporter, type EvalCaseReport } from 'fino:ai/eval';

class ConsoleReporter extends EvalReporter {
  override async onCase(r: EvalCaseReport): Promise<void> {
    console.log(`${r.name}: ${r.score.toFixed(3)} ${r.pass ? 'PASS' : 'FAIL'}`);
  }
}

Methods

class EvalProgressReporter extends EvalReporter {

Reporter that publishes live evaluation progress as a retained signal.

Each hook updates the progress signal: onStart() resets it with the suite name and total, onCase() advances the completed/passed counts and running mean, and onFinish() snaps the snapshot to the final summary. Useful for driving UIs or log output that watches a long evaluation run.

import { evaluate, exactMatch, EvalProgressReporter } from 'fino:ai/eval';

const reporter = new EvalProgressReporter();
reporter.progress.subscribe((p) => {
  console.log(`${p.completed}/${p.total} (${p.passed} passed)`);
});

evaluate({
  name: 'formatter',
  cases: [{ name: 'upper', input: 'a', expected: 'A' }],
  target: async (input: string) => input.toUpperCase(),
  scorers: [exactMatch()],
  report: reporter,
});

Getters

Methods

class JsonEvalReporter extends EvalReporter {

Eval reporter that stores deterministic JSON-friendly reports.

Use this reporter when local CI should write or compare an eval artifact without sending telemetry to an external service. toJSON() returns cloned data sorted by case name for stable output, and when both path and fs are configured the report is also written to disk after onFinish().

import { evaluate, exactMatch, JsonEvalReporter } from 'fino:ai/eval';
import { DiskFileSystem } from 'fino:file';

const reporter = new JsonEvalReporter({
  path: './eval-report.json',
  fs: new DiskFileSystem(),
});

evaluate({
  name: 'formatter',
  cases: [{ name: 'upper', input: 'a', expected: 'A' }],
  target: async (input: string) => input.toUpperCase(),
  scorers: [exactMatch()],
  report: reporter,
});

Constructors

Methods

class OpenTelemetryReporter extends EvalReporter {

Eval reporter that emits OpenTelemetry GenAI evaluation telemetry.

For each scorer result of each case it records a gen_ai.client.evaluation.score histogram point and emits a gen_ai.evaluation.result log event carrying the score value, a pass/fail label, and the explanation when present. Case-level conversationId/responseId become gen_ai.conversation.id and gen_ai.response.id attributes.

When constructed with an exporter or endpoint, onStart() boots a dedicated OtelSDK and onFinish() flushes and shuts it down, so each run is self-contained. Without either option the reporter emits through the ambient logger/meter providers and never owns an SDK. Test outcomes are unchanged either way — the telemetry is purely additive.

import { evaluate, exactMatch, OpenTelemetryReporter } from 'fino:ai/eval';

evaluate({
  name: 'formatter',
  cases: [{ name: 'upper', input: 'a', expected: 'A' }],
  target: async (input: string) => input.toUpperCase(),
  scorers: [exactMatch()],
  report: new OpenTelemetryReporter({ endpoint: 'http://localhost:4318' }),
});

Constructors

Methods

function evaluate< In = unknown, Out = unknown >(opts: EvalOptions<In, Out>): void

Register an evaluation suite with the Fino test runner.

Call it at module load in a test file; fino test then runs one test per case plus a trailing summary test. Each case test awaits target(input), runs every scorer against the output, records the mean scorer value as the case score (a case with no scorers scores 1), attaches the score as test metadata, and asserts the mean reaches threshold. The summary test asserts that the mean over all case scores also reaches the threshold.

When a reporter is supplied, onStart() fires just before the first case executes, onCase() after each case, and onFinish() from the summary test. A target or scorer that throws fails that case's test rather than being converted into a score.

import { evaluate, exactMatch, contains } from 'fino:ai/eval';

evaluate({
  name: 'city extraction',
  target: async (input: string) => extractCity(input),
  cases: [
    { name: 'simple', input: 'I live in Paris', expected: 'Paris' },
    { name: 'trailing punctuation', input: 'Moving to Tokyo!', expected: 'Tokyo' },
  ],
  scorers: { exact: exactMatch(), nonEmpty: contains('') },
  threshold: 1,
});

function exactMatch(): Scorer

Scorer for exact equality between the output and the case's expected value.

Compares JSON.stringify(output) against JSON.stringify(expected), so deep object equality works but property order matters. Scores 1/pass on a match and 0/fail otherwise. Reports under the name exactMatch.

import { evaluate, exactMatch } from 'fino:ai/eval';

evaluate({
  name: 'classifier',
  cases: [{ name: 'positive review', input: 'Loved it!', expected: { label: 'positive' } }],
  target: async (input) => classify(input),
  scorers: [exactMatch()],
});

function contains(substr: string): Scorer

Scorer that checks whether the output text contains substr.

Non-string outputs are JSON-stringified before the check. Scores 1/pass when the substring is present and 0/fail otherwise; the case's expected value is not consulted. Reports under the name contains(<substr>).

import { evaluate, contains } from 'fino:ai/eval';

evaluate({
  name: 'support answer',
  cases: [{ name: 'refund policy', input: 'refund window?' }],
  target: async (input) => askSupportBot(input),
  scorers: [contains('30 days')],
});

function schemaScorer(schema: unknown): Scorer

Scorer that validates the output against a JSON schema.

The schema is compiled once with fino:validate when the scorer is created. Valid outputs score 1/pass; invalid outputs score 0/fail with an explanation listing each validation issue as path: message. Reports under the name schemaScorer.

import { evaluate, schemaScorer } from 'fino:ai/eval';

evaluate({
  name: 'structured extraction',
  cases: [{ name: 'contact card', input: 'Ada Lovelace <ada@example.com>' }],
  target: async (input) => extractContact(input),
  scorers: [schemaScorer({
    type: 'object',
    properties: { name: { type: 'string' }, email: { type: 'string' } },
    required: ['name', 'email'],
  })],
});

function llmJudge(model: Model, rubric: string): Scorer

Scorer that asks a model to grade the output against a rubric.

Builds a single-model agent and prompts it with the rubric, the case input, the expected value (or none when absent), and the output, requesting a JSON object of the form {"score": 0.0, "explanation": "..."}. The score is clamped to [0, 1]; a response that is not valid JSON scores 0. The scorer's own pass flag is set only for a perfect 1, but evaluate() judges the case by mean score against its threshold, so partial credit still counts. Reports under the name llmJudge.

Judged scores inherit the judge model's variance — pin or stub the model in CI and treat the rubric as policy rather than ground truth.

import { evaluate, llmJudge } from 'fino:ai/eval';
import { anthropic } from 'fino:ai/model';

evaluate({
  name: 'tone check',
  cases: [{ name: 'greeting', input: 'Say hi to Ada', expected: 'A warm greeting addressed to Ada' }],
  target: async (input) => runAssistant(input),
  scorers: [llmJudge(anthropic(), 'The answer must be polite and match the expected description.')],
  threshold: 0.8,
});

function semanticSimilarity(model: EmbeddingModel, min: number): Scorer

Scorer that compares the output and expected value by embedding similarity.

Embeds both texts with the given embedding model (non-strings are JSON-stringified first) and scores their cosine similarity, passing when the similarity reaches min. A case without an expected value scores 0/fail with a 'no expected value' explanation. Reports under the name semanticSimilarity(<min>).

Because the raw cosine value is the score, pair this scorer with a threshold below 1 — identical wording is rarely required, only meaning.

import { evaluate, semanticSimilarity } from 'fino:ai/eval';
import { openai } from 'fino:ai/model';

evaluate({
  name: 'paraphrase quality',
  cases: [{ name: 'refund', input: 'refund window?', expected: 'Refunds are accepted within 30 days.' }],
  target: async (input) => paraphraseBot(input),
  scorers: [semanticSimilarity(openai(), 0.85)],
  threshold: 0.85,
});