AI agent directory / SiteAuditBot
SiteAuditBot
Crawler from Semrush
Crawls sites for Semrush's Site Audit tool, which checks on-page SEO, technical and usability issues.
SiteAuditBot is the crawler behind Semrush's Site Audit tool, which checks a site for on-page SEO, technical and usability issues. It runs when a Semrush user sets up or reruns an audit for a domain.
It is an SEO tool, not an AI agent. Semrush lets the user choose a desktop or mobile version, and mobile is the default.
| Run by | Semrush |
|---|---|
| Type | Crawler |
| Purpose | Undeclared |
| Acts for | Its operator |
| robots.txt token | SiteAuditBot |
| Documentation | Semrush's documentation |
Verifying it's really SiteAuditBot
Semrush says Site Audit crawls from 85.208.98.128/25, a subnet it uses for Site Audit only, over ports 80 and 443. A request claiming to be SiteAuditBot from outside that range is worth a closer look.
How Airglow identifies it
- DeclaredIt names itself in the User-Agent:
SiteAuditBot.
Every visit in Airglow says which of these applied, so you can decide how much to trust it.
Seeing it on your site
It reads pages without running JavaScript, so Airglow counts it through a server connection (Cloudflare). It's labeled as context, not replayed.
Allowing or blocking SiteAuditBot
SiteAuditBot reads the SiteAuditBot group in your robots.txt. To keep it out of the whole site:
User-agent: SiteAuditBot Disallow: /
Airglow shows, for every agent that visits, whether your robots.txt lets it in, and helps you write rules by purpose (training, search, acting for a person).
Which user agents Site Audit uses
Semrush's setup guide lists three user agent choices: SiteAuditBot (Mobile), the default, SiteAuditBot (Desktop) and OpenAI-Search. The last one tests whether OpenAI's search bot could crawl your site. Older campaigns set to Googlebot keep crawling as Googlebot until the user changes it. So some Site Audit traffic may not say SiteAuditBot at all.
Semrush shows the full string inside the tool but does not publish it on its documentation pages.
Web Bot Auth on hosts like Shopify
Site Audit can attach a Web Bot Auth signature that the site's host provides, for hosts that block unknown bots by default. The user enters the host's Signature-Agent URL (Semrush gives https://shopify.com as an example), signature input and signature. Semrush does not publish its own signing key directory.
Questions
What is SiteAuditBot?
It is the crawler for Semrush's Site Audit tool. It checks pages for SEO, technical and usability problems when a Semrush user audits a site.
How do I block SiteAuditBot?
Add User-agent: SiteAuditBot and Disallow: / to your robots.txt. Blocking SemrushBot alone does not stop it.
How do I allowlist SiteAuditBot?
Semrush asks site owners to allow 85.208.98.128/25 on ports 80 and 443 in their firewall, CDN and plugins, and to make sure robots.txt does not block SiteAuditBot.
Can a site owner let Site Audit bypass robots.txt?
Yes. If the site owner uploads a verification file Semrush provides to the site root, SiteAuditBot may crawl pages blocked by robots.txt disallow rules or robots meta tags.
Other agents from Semrush
Sources
- Semrush: SemrushBot
- Semrush: Configuring Site Audit
- Semrush: Site Audit troubleshooting
- Semrush: Page is not accessible note
Last reviewed November 10, 2026.
See when SiteAuditBot visits your site
Airglow names every AI agent on your website, replays what it did and links the visitors it sends you. Free for one website.