Downloads 2.4kthis week

GitHub stars 16

Extension descriptor (YAML)

SQL-native web crawler with HTML extraction and MERGE support

Compatibility Matrix

This table covers all DuckDB releases that have not reached end-of-life, including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.

Installing and Loading

INSTALL crawler FROM community;
LOAD crawler;

Example

SELECT url, jq(html.document, 'h1').text as title
FROM crawl(['https://example.com']);

About crawler

The crawler extension provides SQL-native web crawling capabilities for DuckDB.

Features:

• crawl() table function with automatic rate limiting and robots.txt compliance
• LATERAL join support: FROM urls u, LATERAL crawl(u.url)
• sitemap() for XML sitemap parsing
• read_html() for Google Sheets-style IMPORTHTML (tables, lists, JS variables)
• jq() and htmlpath() functions for CSS selector-based extraction
• html.readability for article extraction
• html.schema for JSON-LD/microdata parsing
• CRAWLING MERGE INTO syntax for upsert operations

Example with read_html (like Google Sheets =IMPORTHTML):

SELECT * FROM read_html('https://en.wikipedia.org/wiki/...', 'table.wikitable', 1);
SELECT * FROM read_html('https://example.com/page', 'js=jobs');

Example with extraction:

SELECT
    url,
    jq(html.document, '.price', 'data-amount') as price,
    html.readability.title as article_title
FROM crawl(['https://example.com/products']);

Example with MERGE:

CRAWLING MERGE INTO pages
USING crawl(['https://example.com']) AS src
ON (src.url = pages.url)
WHEN MATCHED THEN UPDATE BY NAME
WHEN NOT MATCHED THEN INSERT BY NAME;

For full documentation see: https://github.com/midwork-finds-jobs/duckdb-crawler

Added Functions

Overloaded Functions

This extension does not add any function overloads.

Added Types

This extension does not add any types.

Added Settings