Connection check
local install
thecrawler
Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support.
Tools
8
GitHub stars
2
Installs / wk
—
Licence
AGPL-3.0
Transport
stdio
Last checked
never
Tools & capabilities
8 toolsRead from the published package source — this server runs locally, so there is no endpoint to query. Names are taken from the code, not observed at runtime, and descriptions are often absent.
crawl
from source
crawl_markdown
from source
Extract clean markdown for a single URL. Strips boilerplate (nav/header/footer/cookie banners). Returns markdown text only — no surrounding metadata. Ideal for feeding a single pag… Extract clean markdown for a single URL. Strips boilerplate (nav/header/footer/cookie banners). Returns markdown text only — no surrounding metadata. Ideal for feeding a single page to an LLM. For multi-URL or rich-data extraction, use
crawl_sitemap
from source
extract_extraction_contract
from source
extract_structured
from source
list_extraction_contracts
from source
search_and_crawl
from source
thecrawler
from source