HTTP client extension for DuckDB with GET/POST/PUT/PATCH/DELETE and byte-range requests
Maintainer(s):
onnimonni
Installing and Loading
INSTALL http_request FROM community;
LOAD http_request;
Example
-- Simple GET request
SELECT http_get('https://example.com/');
-- Access response fields
SELECT
r.status,
r.content_type,
r.content_length,
r.cookies[1].name as first_cookie
FROM (SELECT http_get('https://example.com/') as r);
-- GET with custom headers
SELECT http_get('https://httpbin.org/get', headers := {'Accept': 'application/json'}).body;
-- Byte-range request for partial content
SELECT http_get(
'https://example.com/largefile.gz',
{'Range': byte_range(0, 1024)}
);
About http_request
HTTP client extension providing scalar and table functions for making HTTP requests.
Features:
- All HTTP methods: GET, HEAD, POST, PUT, PATCH, DELETE
- Parallel execution: requests within a chunk run concurrently
- Custom headers via STRUCT parameter
- Query parameters via STRUCT
- Byte-range requests with helper function
- Auto-decompression of gzip/zstd responses
- Form-encoded POST with http_post_form()
- Multipart/form-data uploads with http_post_multipart()
- Parsed Set-Cookie headers into structured cookies array
- Convenience fields: content_type, content_length
- Respects duckdb http and proxy settings
- Configurable concurrency via http_max_concurrency setting (default: 32)
- Global User-Agent override via http_user_agent setting
- Request caching to prevent duplicate requests (http_request_cache setting)
- Headers support both STRUCT and MAP syntax
- Redirect control via http_follow_redirects setting (default: true)
- HTTP_HEADERS type with case-insensitive [] access (headers['Content-Type'] = headers['content-type'])
Uses DuckDB's built-in httplib for HTTP connections.
Added Functions
| function_name | function_type | description | comment | examples |
|---|---|---|---|---|
| byte_range | scalar | NULL | NULL | |
| http_delete | scalar | NULL | NULL | |
| http_delete | table | NULL | NULL | |
| http_get | scalar | NULL | NULL | |
| http_get | table | NULL | NULL | |
| http_head | scalar | NULL | NULL | |
| http_head | table | NULL | NULL | |
| http_patch | scalar | NULL | NULL | |
| http_patch | table | NULL | NULL | |
| http_post | scalar | NULL | NULL | |
| http_post | table | NULL | NULL | |
| http_post_form | scalar | NULL | NULL | |
| http_post_form | table | NULL | NULL | |
| http_post_multipart | scalar | NULL | NULL | |
| http_post_multipart | table | NULL | NULL | |
| http_put | scalar | NULL | NULL | |
| http_put | table | NULL | NULL | |
| http_range_header | scalar | NULL | NULL |
Overloaded Functions
| function_name | function_type | description | comment | examples |
|---|---|---|---|---|
| map_extract_value | scalar | Returns the value for a given key or NULL if the key is not contained in the map. The type of the key provided in the second parameter must match the type of the map’s keys else an error is returned | NULL | [map_extract_value(map(['key'], ['val']), 'key')] |
Added Types
| type_name | type_size | logical_type | type_category | internal |
|---|---|---|---|---|
| HTTP_HEADERS | 16 | MAP | COMPOSITE | true |
Added Settings
| name | description | input_type | scope | aliases |
|---|---|---|---|---|
| http_follow_redirects | Automatically follow HTTP redirects (default: true) | BOOLEAN | GLOBAL | [] |
| http_max_concurrency | Maximum number of concurrent HTTP requests per scalar function call (default: 32) | UBIGINT | GLOBAL | [] |
| http_request_cache | Cache HTTP responses within query to prevent duplicate requests (default: true) | BOOLEAN | GLOBAL | [] |
| http_user_agent | Custom User-Agent header for all HTTP requests (default: DuckDB) | VARCHAR | GLOBAL | [] |