Back to home

Aigentably-Bot

Aigentably-Bot is the crawler we run when an authenticated Aigentably user requests AI tool generation for a site they own. This page documents what it does, what it doesn't do, and how to control access.

Identification

User-agent stringAigentably-Bot/1.0 (+https://aigentably.com/bot)
Operated byAigentably (aigentably.com)
PurposeOn-demand site analysis for WebMCP tool generation, triggered by authenticated site owners.
FrequencyOn request only, and never on a schedule. Crawl results are cached for 24 hours, so repeated analysis of an unchanged site does not re-fetch it; a site owner can force a fresh crawl.
Pages per crawlUp to 5 pages discovered from your sitemap or homepage links, plus up to 10 pages the site owner names explicitly, plus the homepage. At most 21 page requests, inside a 30-second budget. A page that has to be rendered is loaded in a browser, which fetches its subresources as a visitor would, and may be loaded twice so we can tell which element ids your framework generates fresh on each render.

What it does

  • Fetches /robots.txt and honors Sitemap: directives. Disallow and Allow rules are applied to every page the crawler discovers on its own, with the usual precedence: the longest matching rule decides, and Allow wins a tie. * and $ work as wildcards. Those rules are not applied to your homepage, or to a page the site owner typed in by hand: those are treated as an explicit instruction from the person who operates the site.
  • Honors Crawl-delay, waiting that long between requests even when it means we run out of our own time budget and analyse fewer pages. Without one, requests are still spaced about a quarter of a second apart rather than sent all at once.
  • Reads /sitemap.xml (or /sitemap_index.xml) when no Sitemap: entry is declared.
  • Fetches up to 5 representative HTML pages with standard GET requests.
  • Honors <meta name="robots" content="noindex"> (and googlebot / aigentably-bot variants), so those pages are not passed to the model.
  • Sends If-None-Match / If-Modified-Since on refresh: unchanged pages return 304 with no body transfer.
  • Extracts structured signals (forms, buttons, framework, JSON-LD) from each page and discards the raw HTML.
  • Caches the crawl result for 24 hours to avoid re-fetching unchanged sites.
  • Renders the page in a headless browser when plain HTML turns out to be an empty shell, which is the only way a single-page application has any readable content at all. This runs the page's own JavaScript, exactly as a visitor's browser would. Nothing of ours is injected and no code from the page is kept.

What it does not do

  • Does not submit forms, complete purchases, or take any action that changes data on your site.
  • Does not attempt to log in, guess credentials, or bypass authentication. A page returning 401 or 403 is recorded as inaccessible and skipped. Where a site owner supplies a session cookie for their own site, it is sent only to the paths they nominated, and it is never stored beyond the crawl.
  • Does not crawl sites on a schedule or follow links across the web. Every crawl is triggered by a signed-in Aigentably user for a site in their own account.
  • Does not store full page content beyond the structured signal extraction.
  • Does not click anything unless the site owner turns that on for their own site. When they do, it is limited to at most three elements that disclose content already on the page, such as an accordion or a "show more" control, and elements whose label suggests any action at all are excluded outright.

Blocking Aigentably-Bot

Add the following to your robots.txt to block the crawler entirely:

User-agent: Aigentably-Bot
Disallow: /

To block specific paths only:

User-agent: Aigentably-Bot
Disallow: /admin
Disallow: /internal/

Rules under User-agent: * are also honored.

Contact

Questions, abuse reports, or unexpected behavior: contact us.