Downloads 2.7kthis week

GitHub stars 0

Extension descriptor (YAML)

Stream any DuckDB query into Lance datasets with Blob v2 storage and scalar, vector, text, and Bloom filter indexes

Compatibility Matrix

This table covers all DuckDB releases that have not reached end-of-life, including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.

Installing and Loading

INSTALL lance_conversion FROM community;
LOAD lance_conversion;

Example

INSTALL lance_conversion FROM community;
LOAD lance_conversion;

COPY (
  SELECT *
  FROM read_parquet('events/**/*.parquet')
) TO 'events.lance' (FORMAT LANCE);

About lance_conversion

lance_conversion registers Lance as a native DuckDB COPY destination. DuckDB reads, filters, joins, casts, and projects the source query, then streams its output into the Rust Lance writer without an intermediate Parquet export or full-input materialization.

Any source that DuckDB can query can be written to Lance, including DuckDB tables and views, Parquet, CSV, JSON, and S3-backed data:

COPY events TO 'events.lance' (FORMAT LANCE);

COPY (
  SELECT *
  FROM read_json_auto('events.json')
) TO 'events_from_json.lance' (FORMAT LANCE);

The extension also provides read_huggingface for Hugging Face Parquet datasets and read_warc for local or S3 WARC/WARC.GZ files:

COPY (
  SELECT * FROM read_huggingface('lhoestq/demo1')
) TO 'demo1.lance' (FORMAT LANCE);

COPY (
  SELECT id, type, date, content_type, body
  FROM read_warc('archive.warc.gz')
) TO 'archive.lance' (FORMAT LANCE);

read_huggingface fetches only the projected Parquet columns and decodes row groups in the background. max_read_parallelism bounds concurrent row-group reads, and read_ahead_bytes caps the decoded data each one may queue (0 disables read-ahead):

SELECT id, package_name
FROM read_huggingface(
  'lhoestq/demo1',
  max_read_parallelism = 8,
  read_ahead_bytes = 67108864
);

Storage timeouts and retries for S3 and Hugging Face I/O are configured through the lance_conversion_storage_* settings:

SET lance_conversion_storage_io_timeout_ms = 30000;
SET lance_conversion_storage_retry_max_times = 5;

Lance writes support Create (the default), Append, and Overwrite modes:

COPY new_events TO 'events.lance' (FORMAT LANCE, APPEND);
COPY replacement TO 'events.lance' (FORMAT LANCE, OVERWRITE);

When row order need not be preserved, every DuckDB thread runs its own Lance writer, and all writers commit as one dataset version. Use PRESERVE_ORDER false for the fastest writes:

COPY events TO 'events.lance' (FORMAT LANCE, PRESERVE_ORDER false);

SAMPLE_PERCENT and SAMPLE_ROWS write a random sample of the source rows instead of every row:

COPY events TO 'events_10pct.lance' (FORMAT LANCE, SAMPLE_PERCENT 10);
COPY events TO 'events_sample.lance' (FORMAT LANCE, SAMPLE_ROWS 1000);

Blob v2 layout, data file sizing, and indexes are configurable directly from the COPY statement:

COPY (
  SELECT id, event_id, category, description, embedding, asset_uri
  FROM read_parquet('catalog/**/*.parquet')
) TO 'catalog.lance' (
  FORMAT LANCE,
  TARGET_FILE_SIZE 536870912,
  BLOB_INLINE_SIZE_THRESHOLD 2097152,
  BLOB_DEDICATED_SIZE_THRESHOLD 16777216,
  BLOB_COLUMNS (asset_uri),
  SCALAR_INDEX_COLUMNS (id, category),
  VECTOR_INDEX_COLUMNS (embedding),
  TEXT_INDEX_COLUMNS (description),
  BLOOM_FILTER_INDEX_COLUMNS (event_id)
);

Local and S3-compatible Lance destinations are supported. S3 credentials, endpoint, region, TLS, and URL style are resolved from DuckDB secrets and passed to the OpenDAL-backed Lance object store.

Added Functions

Overloaded Functions

This extension does not add any function overloads.

Added Types

This extension does not add any types.

Added Settings