js/data/parquet

js/data/parquet/index.ts

fino:data/parquet - a native Apache Parquet reader and writer producing and consuming Arrow.

Built from discrete parts rather than a bundled engine: Thrift compact metadata (internal:format/thrift), page compression (fino:compress: snappy/gzip/zstd/brotli), and Arrow (fino:data/arrow) as the in-memory representation. readParquet decodes a file to an Arrow Table, concatenating every row group; writeParquet serializes an Arrow Table/RecordBatch to Parquet bytes as a single row group, with ParquetWriteOptions selecting the compression codec, value encoding, dictionary encoding, and data page version. Malformed or unsupported input is reported by throwing ParquetError.

Coverage: the full primitive and logical type system (bool, signed/unsigned ints, float/double/float16, string/binary, fixed-length binary, decimals over every physical backing, date, time, all timestamp units, and INT96); nested columns (list/struct/map and arbitrary nesting) via repetition/definition levels; DATA_PAGE v1 and v2; and every value encoding (PLAIN, dictionary, RLE, the DELTA family, BYTE_STREAM_SPLIT). Compression codecs are those fino:compress provides (LZO and raw-block LZ4 excepted).

import { writeParquet, readParquet } from 'fino:data/parquet';
import { RecordBatch } from 'fino:data/arrow';

const batch = RecordBatch.from({ id: [1, 2, 3], name: ['a', 'b', 'c'] });
const bytes = writeParquet(batch, { compression: 'zstd' });
const table = readParquet(bytes);
table.getChild('name')!.toArray(); // ['a', 'b', 'c']

Useful references: - Parquet file format: https://parquet.apache.org/docs/file-format/ - parquet.thrift: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift

Functions

function readParquet(input: Uint8Array | ArrayBuffer): Table

Decode a complete Parquet file into an Arrow Table.

Accepts the file's bytes as a Uint8Array or ArrayBuffer (an ArrayBuffer is viewed in place, not copied). Each row group in the file becomes one RecordBatch in the returned table; a file with no row groups yields a table containing a single zero-row batch, so the schema is always preserved. Values arrive already converted to their Arrow logical types — strings, decimals, dates, timestamps, nested lists/structs/maps, and so on.

Throws ParquetError if the input is too short or missing the PAR1 magic, if the footer length is corrupt, if a row group's column-chunk count does not match the schema's leaf count, or if a column chunk carries no metadata. Errors from deeper layers (unsupported codec, malformed pages, bad level streams) propagate as ParquetError too.

import { readParquet } from 'internal:data/parquet/reader';
import { DiskFileSystem } from 'fino:file';

const file = await new DiskFileSystem().open('/data/events.parquet', 'r');
const table = readParquet(await file.bytes());
await file.close();
for (const row of table) console.log(row);

function writeParquet(source: Table | RecordBatch, options?: ParquetWriteOptions): Uint8Array

Serialize an Arrow table or batch to a complete Parquet file in memory.

A RecordBatch is wrapped into a single-batch Table; a multi-batch Table has its batches concatenated column-wise, so the output always contains exactly one row group holding every row. The returned bytes are the full file — magic, column chunks, footer — suitable for writing to disk as-is or feeding straight back to readParquet.

The Parquet schema is derived from the Arrow schema, including nested list/struct/map columns, nullability (definition levels), and logical type annotations. Encoding and compression choices come from options; see ParquetWriteOptions for the per-column fallback rules.

Throws a ParquetError if the requested compression codec's system library is not available.

import { writeParquet } from 'internal:data/parquet/writer';
import { RecordBatch, Schema, Field, list, int32, utf8, vectorFromArray } from 'fino:data/arrow';
import { DiskFileSystem } from 'fino:file';

const scoresType = list(new Field('item', int32(), true));
const batch = new RecordBatch(
  new Schema([new Field('user', utf8(), false), new Field('scores', scoresType, true)]),
  [vectorFromArray(['ada', 'lin'], utf8()), vectorFromArray([[1, 2], [3]], scoresType)],
);
const bytes = writeParquet(batch, { compression: 'gzip', pageVersion: 2 });
await new DiskFileSystem().writeFile('scores.parquet', bytes);