funnelweb: hyperconfigurable async web scraper

funnelweb allows you to scrape arbitrary domains via configuration files written in YAML, TOML or JSON. It can leverage openai, jq, trafilatura, XPath and regex to extract metadata from pages of interest, which end up in a pandas DataFrame.

Code is written in asynchronous Python, providing parallelised scraping/spidering via aiohttp, playwright and requests (if need be). It is a no-code tool for most applications, though is extensible with custom classes and callbacks if need be.

Visit the main documentation page for instructions on installation and use.