Search Shortcut cmd + k | ctrl + k

Read HDF5 datasets and attributes

Maintainer(s): jokasimr

Compatibility Matrix

This table covers all DuckDB releases that have not reached end-of-life, including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.

Platform 1.4.4 1.4.5 1.5.5 1.5.6 2.0.0
Linux AMD64
Linux ARM64
macOS AMD64
macOS ARM64
Windows AMD64
Windows ARM64
WebAssembly

Installing and Loading

INSTALL h5db FROM community;
LOAD h5db;

Example

FROM h5_read('file.h5', '/some/dataset', '/another');

About h5db

This extension provides functions for reading data and metadata from HDF5 files.

Features include:

  • h5_read() table function to read datasets.
  • h5_tree() table function to list groups and datasets in a file, optionally including projected attributes.
  • h5_attributes() table function to read attributes.
  • h5_ls() table and scalar functions to list entries in groups.
  • Multiple datasets: Read multiple datasets into separate columns.
  • Multi-dimensional arrays: Support for 1D to 4D datasets.
  • Projection pushdown: Read only the datasets that are actually needed.
  • Index column: Optionally adds an index column that supports predicate pushdown of constant range filters (>, <=, BETWEEN, etc.) for efficient selective reads.
  • Reads datasets that are larger than memory.
  • Remote reads over HTTPS, S3, etc., via the DuckDB httpfs extension.
  • Remote reads over SFTP via a built-in SFTP client.
  • Globbing: Combine datasets from multiple files vertically (UNION ALL).

For full documentation, see the h5db repository.

Added Functions

function_name function_type description comment examples
h5_alias scalar Renames a column definition. NULL [FROM h5_read('data.h5', h5_alias('temperature', '/entry/temp'))]
h5_attr scalar Creates a projected HDF5 attribute definition for h5_tree() and h5_ls(). NULL [FROM h5_tree('data.h5', h5_attr('NX_class'))]
h5_attributes scalar Reads all attributes from one HDF5 object as a MAP. NULL [SELECT h5_attributes('data.h5', '/measurements')]
h5_attributes table Reads all attributes from an HDF5 object or file root. NULL [FROM h5_attributes('data.h5', '/measurements')]
h5_first_file scalar Returns the first concrete HDF5 filename from an exact path, glob, or list for planning-time use with h5_read(). NULL [FROM h5read(h5_first_file('runs/run*.h5'), '/detector_geometry')]
h5_index scalar Creates a virtual row-index column definition for h5_read(). NULL [FROM h5_read('data.h5', h5_index(), '/measurements')]
h5_ls scalar Lists entries immediately under an HDF5 group as a MAP. NULL [SELECT h5_ls('data.h5', '/entry')]
h5_ls table Lists entries immediately under an HDF5 group as rows. NULL [FROM h5_ls('data.h5', '/entry')]
h5_ls_swmr scalar Lists entries immediately under an HDF5 group as a MAP using SWMR read mode. NULL [SELECT h5_ls_swmr('data.h5', '/entry')]
h5_read scalar Reads one HDF5 dataset as a VARIANT. NULL [SELECT h5_read('data.h5', '/entry/dataset')]
h5_read table Reads one or more HDF5 datasets as DuckDB columns. NULL [FROM h5_read('data.h5', '/measurements')]
h5_ree scalar Creates a run-end encoded column definition for h5_read(). NULL [FROM h5_read('data.h5', '/time', h5_ree('/state_run_ends', '/state_values'))]
h5_rse scalar Creates a run-start encoded column definition for h5_read(). NULL [FROM h5_read('data.h5', '/time', h5_rse('/state_run_starts', '/state_values'))]
h5_tree table Recursively lists entries in an HDF5 file. NULL [FROM h5_tree('data.h5')]
h5db_version scalar Returns h5db, linked HDF5, and SFTP backend library versions. NULL [SELECT h5db_version()]

Overloaded Functions

This extension does not add any function overloads.

Added Types

This extension does not add any types.

Added Settings

name description input_type scope aliases
h5db_batch_size Target batch size for h5_read read-ahead caching (e.g. 1MB, 8MB) VARCHAR GLOBAL []
h5db_scalar_read_memory_limit Estimated peak memory limit for scalar h5_read materialization (e.g. 64MB, none) VARCHAR GLOBAL []
h5db_swmr_default Default to SWMR read mode for h5db table functions BOOLEAN GLOBAL []