Provides IO observability to filesystem
Compatibility Matrix
This table covers all DuckDB releases that have not reached end-of-life, including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.
| Platform | 1.4.4 | 1.4.5 | 1.5.5 | 1.5.6 | 2.0.0 |
|---|---|---|---|---|---|
| Linux AMD64 | |||||
| Linux ARM64 | |||||
| macOS AMD64 | |||||
| macOS ARM64 | |||||
| Windows AMD64 | |||||
| Windows ARM64 | |||||
| WebAssembly |
Installing and Loading
INSTALL observefs FROM community;
LOAD observefs;
Example
COPY (SELECT observefs_get_profile()) TO '/tmp/output.txt';
About observefs
This extension provides observability to duckdb filesystems. It supports a few key features:
- 100% compatible with duckdb httpfs
- Record latency for most of the IO operations (open, read, write, etc)
- Provides both process-wise and bucket-wise latency stats (including histogram and quantile estimation) to all read operations
- Provides cache access insight to duckdb external file cache
- Allows registering ANY duckdb compatible filesystems (i.e., azure filesystem)
Added Functions
| function_name | function_type | description | comment | examples |
|---|---|---|---|---|
| observefs_clear | scalar | Clears collected I/O observability metrics for all wrapped filesystems. | NULL | [observefs_clear()] |
| observefs_clear_external_file_cache_access_record | scalar | Clears the recorded external file cache hit and miss counts. | NULL | [observefs_clear_external_file_cache_access_record()] |
| observefs_external_file_cache_access_record | table | Returns recorded external file cache hit, miss, and partial-hit counts. | NULL | [observefs_external_file_cache_access_record()] |
| observefs_get_profile | scalar | Returns a human-readable profile of I/O metrics for wrapped filesystems. | NULL | [observefs_get_profile()] |
| observefs_list_registered_filesystems | table | Lists the filesystem implementations registered with DuckDB. | NULL | [observefs_list_registered_filesystems()] |
| observefs_wrap_filesystem | scalar | Wraps a registered filesystem with I/O observability instrumentation. | NULL | [observefs_wrap_filesystem('AzureBlobStorageFileSystem')] |
Overloaded Functions
This extension does not add any function overloads.
Added Types
This extension does not add any types.
Added Settings
| name | description | input_type | scope | aliases | typed_value |
|---|---|---|---|---|---|
| observefs_enable_external_file_cache_stats | Whether to enable stats record for external file cache. | BOOLEAN | GLOBAL | [] | 1 |
| temp_directory | Set the directory to which temporary files are written. | VARCHAR | GLOBAL | [] | post.db.tmp |