Feature
Multi-page crawl + 24h cache for AI tool generation
AI tool generation now crawls up to 5 representative pages per site (homepage plus product, cart, checkout, search, account) instead of analyzing the homepage alone. Pages are discovered via sitemap.xml with fallback to homepage link extraction. Robots.txt is honored.
- Per-page LLM call with deduplication across pages.
- Each suggestion stamped with its source URL (visible in the dashboard).
- Crawl results cached for 24h per site. Force refresh button bypasses the cache.
- Crawl report panel shows which pages succeeded and how many tools came from each.
- Pro: Gemini 2.5 Pro. Free: Gemini 2.5 Flash.
Read the deep dive on the pipeline.