Extract a site

Discovery order: sitemap → robots.txt hints → RSS/Atom → link crawl fallback.

Site URL

Extraction schema

LLM provider

Model override

Max pages (blank = whole site)

Concurrency

Run LLM extraction Crawl fallback RSS/Atom discovery Respect robots.txt

Providers

gemini · gemini-3.7-flash · ready openai · gemini-3.7-flash · no key anthropic · gemini-3.7-flash · no key echo · gemini-3.7-flash · ready

Set LLM_PROVIDER and the matching API key in .env. Without a key the pipeline still runs and falls back to JSON-LD / meta extraction.

Site	Status	Schema	Provider	Routes	Extracted	Failed
https://www.namshi.com	cancelled	ecommerce	gemini	51	16	0
https://www.daraz.pk	done	ecommerce	gemini	11	0	1
https://www.daraz.pk	done	roa_panel	gemini	11	0	1
https://www.daraz.pk	done	roa_panel	gemini	1	1	0
https://theostrich.pk	done	roa_panel	gemini	11	11	0
https://enroba.com	done	roa_panel	gemini	10	9	1