httpfs with connection pool, HTTP/2 and async IO.
Maintainer(s):
dentiny
Compatibility Matrix
This table covers all DuckDB releases that have not reached end-of-life, including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.
| Platform | 1.4.4 | 1.4.5 | 1.5.5 | 1.5.6 | 2.0.0 |
|---|---|---|---|---|---|
| Linux AMD64 | |||||
| Linux ARM64 | |||||
| macOS AMD64 | |||||
| macOS ARM64 | |||||
| Windows AMD64 | |||||
| Windows ARM64 | |||||
| WebAssembly |
Installing and Loading
INSTALL curl_httpfs FROM community;
LOAD curl_httpfs;
Example
SELECT length(content) AS char_count FROM read_text('https://raw.githubusercontent.com/dentiny/duck-read-cache-fs/main/test/data/stock-exchanges.csv');
About curl_httpfs
This extension rewrites HTTP layer for httpfs based on libcurl and epoll, it provides a few features:
- 100% compatible with httpfs; use curl-based implementation by default, but also allows users to fallback to httplib.
- Enable HTTP/2 by default.
- Implements TCP connection pool.
- All network IO operations are performed in asynchronously.
Added Functions
| function_name | function_type | description | comment | examples |
|---|---|---|---|---|
| curl_httpfs_get_tcp_connection | table | Returns active system TCP connection counts grouped by remote IP address. | NULL | [SELECT * FROM curl_httpfs_get_tcp_connection();] |
| curl_httpfs_http_util_name | scalar | Returns the name of the HTTP client implementation currently used by curl_httpfs. | NULL | [SELECT curl_httpfs_http_util_name();] |
Overloaded Functions
This extension does not add any function overloads.
Added Types
This extension does not add any types.
Added Settings
| name | description | input_type | scope | aliases | typed_value |
|---|---|---|---|---|---|
| curl_httpfs_client_implementation | Select the HTTPUtil implementation to be used. Supports multi_curl, curl, httplib, and default (multi_curl). |
VARCHAR | GLOBAL | [] | default |
| curl_httpfs_enable_verbose_logging | Turn on and off curl-based http util verbose logging. | BOOLEAN | GLOBAL | [] | 0 |
| temp_directory | Set the directory to which temporary files are written. | VARCHAR | GLOBAL | [] | post.db.tmp |