Identification
User agent: WebScopeBot/1.0 (+https://example.com/bot)
The production contact URL and product domain are configurable. Requests use this recognizable user agent and ordinary HTTP first; browser rendering is a separate, throttled job.
Robots and load
We fetch and honor robots.txt before crawling. Domain-level concurrency, page count, response size, redirect count and time are bounded. Interactive anonymous requests inspect only the homepage. Crawl-delay is honored where reasonably expressed.
Blocking the crawler
To block all access, add User-agent: WebScopeBot and Disallow: / to robots.txt. Existing public observations can be reviewed through the data-correction contact channel.
Safety
The crawler refuses localhost, private, link-local, reserved and cloud-metadata destinations. It validates every redirect and pins connections to a validated public address to resist DNS rebinding.