How it behaves
- It obeys robots.txt, including rules for the
SEOReviewBotuser agent and Crawl-delay. - It does not crawl a site when robots.txt returns a server error.
- It crawls a single website per analysis, with a small page limit, low concurrency and a delay between requests.
- It slows down and retries later when a server responds with 429 or 503.
- It does not execute JavaScript, submit forms or send cookies.
Blocking the crawler
To stop SEOReviewBot from crawling your site, add the following to your robots.txt:
User-agent: SEOReviewBot Disallow: /
Contact
Contact details have not been added yet.