Remote storage filesystem for DuckDB via Apache OpenDAL — S3, GCS, Azure, Hugging Face, WebDAV, SFTP, and more, with DuckDB secret-based auth.
Maintainer(s):
dentiny
Compatibility Matrix
This table covers all DuckDB releases that have not reached end-of-life, including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.
| Platform | 1.4.4 | 1.4.5 | 1.5.5 | 1.5.6 | 2.0.0 |
|---|---|---|---|---|---|
| Linux AMD64 | |||||
| Linux ARM64 | |||||
| macOS AMD64 | |||||
| macOS ARM64 | |||||
| Windows AMD64 | |||||
| Windows ARM64 | |||||
| WebAssembly |
Installing and Loading
INSTALL duckdb_opendalfs FROM community;
LOAD duckdb_opendalfs;
Example
FORCE INSTALL duckdb_opendalfs FROM community;
LOAD duckdb_opendalfs;
CREATE SECRET production_s3 (
TYPE opendal_s3,
SCOPE 's3://analytics',
ACCESS_KEY_ID 'access-key',
SECRET_ACCESS_KEY 'secret-key',
REGION 'us-east-1'
);
SELECT * FROM read_parquet('s3://analytics/events.parquet');
About duckdb_opendalfs
duckdb_opendalfs is an extension which supports many backends (S3, GCS, Azure Blob, Hugging Face, etc)
and secret management via opendal_<backend> secret types.
Added Functions
This extension does not add any functions.
Overloaded Functions
This extension does not add any function overloads.
Added Types
This extension does not add any types.
Added Settings
| name | description | input_type | scope | aliases | typed_value |
|---|---|---|---|---|---|
| opendal_io_timeout_ms | OpenDAL I/O-operation timeout in milliseconds | UBIGINT | GLOBAL | [] | 10000 |
| opendal_retry_factor | OpenDAL exponential retry factor | DOUBLE | GLOBAL | [] | 2.0 |
| opendal_retry_jitter | Add jitter to OpenDAL retry delays | BOOLEAN | GLOBAL | [] | false |
| opendal_retry_layer_enabled | Enable the OpenDAL retry layer | BOOLEAN | GLOBAL | [] | true |
| opendal_retry_max_delay_ms | Maximum OpenDAL retry delay in milliseconds | UBIGINT | GLOBAL | [] | 60000 |
| opendal_retry_max_times | Maximum number of OpenDAL retry backoff delays | UBIGINT | GLOBAL | [] | 3 |
| opendal_retry_min_delay_ms | Minimum OpenDAL retry delay in milliseconds | UBIGINT | GLOBAL | [] | 1000 |
| opendal_timeout_layer_enabled | Enable the OpenDAL timeout layer | BOOLEAN | GLOBAL | [] | true |
| opendal_timeout_ms | OpenDAL control-operation timeout in milliseconds | UBIGINT | GLOBAL | [] | 60000 |
| temp_directory | Set the directory to which temporary files are written. | VARCHAR | GLOBAL | [] | post.db.tmp |