Search Shortcut cmd + k | ctrl + k
boilstream

Secure remote secrets storage with OPAQUE PAKE authentication, email/password registration, MFA login, and automatic ducklake mounting

Maintainer(s): dforsber

Installing and Loading

INSTALL boilstream FROM community;
LOAD boilstream;

Example

-- Setup: Install and load boilstream extension
INSTALL httpfs; LOAD httpfs;
INSTALL boilstream FROM community;
LOAD boilstream;

-- Option 1: Login with email/password/MFA (establishes OPAQUE session)
--     NOTE: Remember to clean up your ~/.duckdb/history
PRAGMA boilstream_login('https://api.example.com/[email protected]', 'password', '123456');
┌────────────────────────┬─────────────────────┬────────────────────┐
         status              expires_at       ducklakes_attached 
├────────────────────────┼─────────────────────┼────────────────────┤
 Session token obtained  2025-11-21 10:30:00                   2 
└────────────────────────┴─────────────────────┴────────────────────┘

-- Option 2: Authenticate with bootstrap token (from SSO/web UI)
PRAGMA boilstream_bootstrap_session('https://api.example.com/secrets:TOKEN');
┌────────────────────────┬─────────────────────┬────────────────────┐
         status              expires_at       ducklakes_attached 
├────────────────────────┼─────────────────────┼────────────────────┤
 Session token obtained  2025-11-21 10:30:00                   2 
└────────────────────────┴─────────────────────┴────────────────────┘

-- List all available ducklakes
FROM boilstream_ducklakes();
┌──────────────┬──────────────┬─────────────────┐
 catalog_id    catalog_name    description   
├──────────────┼──────────────┼─────────────────┤
 5a805647-...  my_analytics  Sales analytics 
└──────────────┴──────────────┴─────────────────┘

-- List all cached secrets with expiration
FROM boilstream_secrets();
┌─────────────────────┬──────────┬─────────────────────┐
        name            type        expires_at      
├─────────────────────┼──────────┼─────────────────────┤
 my_analytics         ducklake  2026-11-19 20:00:00 
 my_analytics_s3      s3        2026-11-19 20:00:00 
 my_analytics_pg      postgres  2026-11-19 20:00:00 
└─────────────────────┴──────────┴─────────────────────┘

-- Create new ducklake
PRAGMA boilstream_create_ducklake('new_catalog', 'My new ducklake');

-- Query ducklake (automatically attached)
USE my_analytics;
SHOW TABLES;

About boilstream

Boilstream extension provides enterprise-grade remote secrets storage for multi-tenant DuckDB deployments.

Features:

  • 🔐 OPAQUE PAKE Authentication - Password-less authentication (IETF RFC 9807)
  • 🔒 End-to-end Encryption - AES-256-GCM/ChaCha20-Poly1305 for secrets and responses
  • 🔑 Cryptographic Key Derivation - HKDF-SHA256 with domain separation
  • 🛡️ Defense in Depth - HMAC verification before AEAD decryption
  • 🎯 Session Protection - Lock-step sequence protocol prevents hijacking
  • 🗄️ Ducklake Support - Auto-attach multi-tenant data catalogs
  • 🌐 SSO Integration - Bootstrap token exchange for web environments

Security:

  • Facebook's audited OPAQUE implementation (Rust + C++ FFI)
  • Protects against MITM attacks even if TLS is compromised
  • One-time bootstrap tokens with anonymous authentication
  • Session resumption with encrypted refresh tokens
  • Comprehensive test vectors and security specification

API Functions:

  • PRAGMA boilstream_register_user(url_with_email, password) - Register new user with MFA (displays QR code)
  • PRAGMA boilstream_verify_mfa(totp_code) - Complete registration and get backup codes
  • PRAGMA boilstream_login(url_with_email, password, mfa_code) - Email/password/MFA login
  • PRAGMA boilstream_bootstrap_session(url_with_token) - Authenticate with bootstrap token
  • boilstream_ducklakes() - List available ducklakes
  • boilstream_secrets() - List cached secrets with expiration
  • boilstream_buckets() - List available storage buckets
  • PRAGMA boilstream_create_ducklake(name, description) - Create new ducklake
  • PRAGMA boilstream_help - Show all available commands

Requirements:

  • httpfs extension (auto-loaded)
  • postgres_scanner extension (auto-loaded)
  • ducklake extension (auto-loaded)
  • Compatible REST API server (see API specification)

For detailed documentation, security specification, and setup instructions, visit the extension repository.

Added Functions

function_name function_type description comment examples
boilstream_bootstrap_session pragma NULL NULL  
boilstream_buckets table NULL NULL  
boilstream_create_ducklake pragma NULL NULL  
boilstream_ducklakes table NULL NULL  
boilstream_help pragma NULL NULL  
boilstream_login pragma NULL NULL  
boilstream_register_user pragma NULL NULL  
boilstream_secrets table NULL NULL  
boilstream_verify_mfa pragma NULL NULL  
ducklake_add_data_files table NULL NULL  
ducklake_cleanup_old_files table NULL NULL  
ducklake_current_snapshot table NULL NULL  
ducklake_delete_orphaned_files table NULL NULL  
ducklake_expire_snapshots table NULL NULL  
ducklake_flush_inlined_data table NULL NULL  
ducklake_last_committed_snapshot table NULL NULL  
ducklake_list_files table NULL NULL  
ducklake_merge_adjacent_files table NULL NULL  
ducklake_options table NULL NULL  
ducklake_rewrite_data_files table NULL NULL  
ducklake_scan table NULL NULL  
ducklake_set_commit_message table NULL NULL  
ducklake_set_option table NULL NULL  
ducklake_settings table NULL NULL  
ducklake_snapshots table NULL NULL  
ducklake_table_changes table_macro NULL NULL  
ducklake_table_deletions table NULL NULL  
ducklake_table_info table NULL NULL  
ducklake_table_insertions table NULL NULL  
murmur3_32 scalar NULL NULL  
pg_clear_cache table NULL NULL  
postgres_attach table NULL NULL  
postgres_configure_pool table NULL NULL  
postgres_execute table NULL NULL  
postgres_query table NULL NULL  
postgres_scan table NULL NULL  
postgres_scan_pushdown table NULL NULL  
read_postgres_binary table NULL NULL  

Overloaded Functions

This extension does not add any function overloads.

Added Types

This extension does not add any types.

Added Settings

name description input_type scope aliases
ducklake_default_data_inlining_row_limit Default row limit for data inlining (0 disables inlining) UBIGINT GLOBAL []
ducklake_max_retry_count The maximum amount of retry attempts for a ducklake transaction UBIGINT GLOBAL []
ducklake_retry_backoff Backoff factor for exponentially increasing retry wait time DOUBLE GLOBAL []
ducklake_retry_wait_ms Time between retries UBIGINT GLOBAL []
ducklake_write_deletion_vectors [EXPERIMENTAL] Write Iceberg V3 deletion vectors (puffin) instead of positional delete files (parquet) BOOLEAN GLOBAL []
pg_array_as_varchar Read Postgres arrays as varchar - enables reading mixed dimensional arrays BOOLEAN GLOBAL []
pg_connection_cache Whether or not to use the connection pooling. This option is deprecated, instead to disable the connection pooling use "SET pg_pool_max_connections=0" for newly attached Postgres databases and "FROM postgres_configure_pool(catalog_name='my_attached_postgres_db', max_connections=0)" for already attached Postgres databases. BOOLEAN GLOBAL []
pg_connection_limit The maximum amount of concurrent Postgres connections. This option is deprecated, instead use "SET pg_pool_max_connections = 42" for newly attached Postgres databases and "FROM postgres_configure_pool(catalog_name='my_attached_postgres_db', max_connections=42)" for already attached Postgres databases. UBIGINT GLOBAL []
pg_debug_show_queries DEBUG SETTING: print all queries sent to Postgres to stdout BOOLEAN GLOBAL []
pg_experimental_filter_pushdown Whether or not to use filter pushdown BOOLEAN GLOBAL []
pg_idle_in_transaction_timeout_millis Postgres idle in transaction timeout in milliseconds to set on scan connections UINTEGER GLOBAL []
pg_null_byte_replacement When writing NULL bytes to Postgres, replace them with the given character VARCHAR GLOBAL []
pg_pages_per_task The amount of pages per task UBIGINT GLOBAL []
pg_pool_enable_reaper_thread Whether to enable the connection pool reaper thread, that periodically scans the pool to check the 'max_lifetime_millis' and 'idle_timeout_millis' and closes the connection which exceed the specified values. Either 'max_lifetime_millis' or 'idle_timeout_millis' must be set to a non-zero value for this option to be effective. This option only applies to newly attached Postgres databases, to configure a database that is already attached use "FROM postgres_configure_pool(catalog_name='my_attached_postgres_db', enable_reaper_thread=TRUE)" BOOLEAN GLOBAL []
pg_pool_enable_thread_local_cache Whether to enable the connection caching in thread-local cache. Such connections are getting pinned to the threads and are not made available to other threads, while still taking the place in the pool. This option only applies to newly attached Postgres databases, to configure a database that is already attached use "FROM postgres_configure_pool(catalog_name='my_attached_postgres_db', enable_thread_local_cache=FALSE)" BOOLEAN GLOBAL []
pg_pool_health_check_query The query that is used to check that the connection is healthy. Setting this option to an empty string disables the health check. This option only applies to newly attached Postgres databases, to configure a database that is already attached use "FROM postgres_configure_pool(catalog_name='my_attached_postgres_db', health_check_query=SELECT 42)" VARCHAR GLOBAL []
pg_pool_idle_timeout_millis Maximum number of milliseconds the connection can be kept idle in the pool. This number is checked when the connection is taken from the pool. When the connection pool reaper thread is enabled ('pg_pool_enable_reaper_thread' option), then this number is checked in background periodically. This option only applies to newly attached Postgres databases, to configure a database that is already attached use "FROM postgres_configure_pool(catalog_name='my_attached_postgres_db', idle_timeout_millis=300000)" UBIGINT GLOBAL []
pg_pool_max_connections Maximum number of connections that are allowed to be cached in a connection pool for each attached Postgres database. This number can be temporary exceeded when parallel scans are used. This option only applies to newly attached Postgres databases, to configure a database that is already attached use "FROM postgres_configure_pool(catalog_name='my_attached_postgres_db', max_connections=42)" UBIGINT GLOBAL []
pg_pool_max_lifetime_millis Maximum number of milliseconds the connection can be kept open. This number is checked when the connection is taken from the pool and returned to the pool. When the connection pool reaper thread is enabled ('pg_pool_enable_reaper_thread' option), then this number is checked in background periodically. This option only applies to newly attached Postgres databases, to configure a database that is already attached use "FROM postgres_configure_pool(catalog_name='my_attached_postgres_db', max_lifetime_millis=600000)" UBIGINT GLOBAL []
pg_pool_wait_timeout_millis Maximum number of milliseconds to wait when acquiring a connection from a pool where all available connections are already taken. This option only applies to newly attached Postgres databases, to configure a database that is already attached use "FROM postgres_configure_pool(catalog_name='my_attached_postgres_db', wait_timeout_millis=60000)" UBIGINT GLOBAL []
pg_statement_timeout_millis Postgres statement timeout in milliseconds to set on scan connections UINTEGER GLOBAL []
pg_use_binary_copy Whether or not to use BINARY copy to read data BOOLEAN GLOBAL []
pg_use_ctid_scan Whether or not to parallelize scanning using table ctids BOOLEAN GLOBAL []
pg_use_text_protocol Whether or not to use TEXT protocol to read data. This is slower, but provides better compatibility with non-Postgres systems BOOLEAN GLOBAL []