Search Shortcut cmd + k | ctrl + k
hedged_request_fs
Downloads 5.2kthis week
GitHub stars 2
Extension descriptor (YAML)

Provides hedged request for filesystem operations

Maintainer(s): dentiny

Compatibility Matrix

This table covers all DuckDB releases that have not reached end-of-life, including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.

Platform 1.4.4 1.4.5 1.5.5 1.5.6 2.0.0
Linux AMD64
Linux ARM64
macOS AMD64
macOS ARM64
Windows AMD64
Windows ARM64
WebAssembly

Installing and Loading 

INSTALL hedged_request_fs FROM community;
LOAD hedged_request_fs;

Example 

SELECT * FROM hedged_fs_list_filesystems();
SELECT hedged_fs_wrap('S3FileSystem');

About hedged_request_fs

This extension enhances DuckDB filesystem capability by hedged request to cut down tail latency.

  • It allows users to wrap an existing filesystem, which achieves hedged request with no user code change.
  • It supports all metadata and directory operations, the slow ones of which will be attempted hedged request to cut down tail latency.
  • Users are able to tune hedged requests trigger thresholds and max attempt counts.

Added Functions

function_name function_type description comment examples
hedged_fs_list_filesystems table Lists the names of all filesystem subsystems currently registered with DuckDB. NULL [SELECT * FROM hedged_fs_list_filesystems();]
hedged_fs_wrap scalar Wraps a registered filesystem subsystem with hedged requests and returns true on success. NULL [SELECT hedged_fs_wrap('S3FileSystem');]

Overloaded Functions

This extension does not add any function overloads.

Added Types

This extension does not add any types.

Added Settings

name description input_type scope aliases typed_value
hedged_fs_create_directory_delay_ms Delay in milliseconds before starting hedged request for CreateDirectory and CreateDirectoriesRecursive UBIGINT GLOBAL [] 3000
hedged_fs_delete_delay_ms Delay in milliseconds before starting hedged request for RemoveFile, TryRemoveFile, and RemoveDirectory UBIGINT GLOBAL [] 3000
hedged_fs_directory_exists_delay_ms Delay in milliseconds before starting hedged request for DirectoryExists operation UBIGINT GLOBAL [] 3000
hedged_fs_file_exists_delay_ms Delay in milliseconds before starting hedged request for FileExists operation UBIGINT GLOBAL [] 3000
hedged_fs_get_file_size_delay_ms Delay in milliseconds before starting hedged request for GetFileSize operation UBIGINT GLOBAL [] 3000
hedged_fs_get_file_type_delay_ms Delay in milliseconds before starting hedged request for GetFileType operation UBIGINT GLOBAL [] 3000
hedged_fs_get_last_modified_time_delay_ms Delay in milliseconds before starting hedged request for GetLastModifiedTime operation UBIGINT GLOBAL [] 3000
hedged_fs_get_stats_delay_ms Delay in milliseconds before starting hedged request for Stats operation UBIGINT GLOBAL [] 3000
hedged_fs_get_version_tag_delay_ms Delay in milliseconds before starting hedged request for GetVersionTag operation UBIGINT GLOBAL [] 3000
hedged_fs_glob_delay_ms Delay in milliseconds before starting hedged request for Glob operation UBIGINT GLOBAL [] 5000
hedged_fs_list_files_delay_ms Delay in milliseconds before starting hedged request for ListFiles operation UBIGINT GLOBAL [] 5000
hedged_fs_max_hedged_request_count Maximum number of hedged requests to spawn for each operation UBIGINT GLOBAL [] 3
hedged_fs_open_file_delay_ms Delay in milliseconds before starting hedged request for OpenFile operation UBIGINT GLOBAL [] 3000
temp_directory Set the directory to which temporary files are written. VARCHAR GLOBAL [] post.db.tmp