AI agent directory / SemrushBot-ESI
SemrushBot-ESI
Crawler from Semrush
Crawls sites for Semrush Enterprise Site Intelligence, which audits on-page SEO, technical and usability issues.
SemrushBot-ESI is the Semrush bot for Semrush Enterprise Site Intelligence. Semrush says the product analyzes on-page SEO, technical and usability issues.
Semrush's bot page gives only the robots.txt token for this bot, not a full user-agent string.
| Run by | Semrush |
|---|---|
| Type | Crawler |
| Purpose | Undeclared |
| Acts for | Its operator |
| robots.txt token | SemrushBot-ESI |
| Documentation | Semrush's documentation |
How Airglow identifies it
- DeclaredIt names itself in the User-Agent:
SemrushBot-ESI.
Every visit in Airglow says which of these applied, so you can decide how much to trust it.
Seeing it on your site
It reads pages without running JavaScript, so Airglow counts it through a server connection (Cloudflare). It's labeled as context, not replayed.
Allowing or blocking SemrushBot-ESI
SemrushBot-ESI reads the SemrushBot-ESI group in your robots.txt. To keep it out of the whole site:
User-agent: SemrushBot-ESI Disallow: /
Airglow shows, for every agent that visits, whether your robots.txt lets it in, and helps you write rules by purpose (training, search, acting for a person).
Blocking one Semrush tool doesn't block the others
Semrush uses a separate robots.txt token for each tool. A group for SemrushBot-ESI does not cover SemrushBot, SiteAuditBot or the other tokens on Semrush's bot page. Add a group for each one you want to stop.
Questions
What is SemrushBot-ESI?
It is the bot behind Semrush Enterprise Site Intelligence, which audits sites for on-page SEO, technical and usability issues.
How do I block SemrushBot-ESI?
Add User-agent: SemrushBot-ESI and Disallow: / to the robots.txt in your site's top directory. Semrush documents this token on its bot page.
Is it an AI crawler?
No. Semrush describes it as part of its SEO tools. It does not say the data is used to train AI models.
Other agents from Semrush
Sources
Last reviewed November 10, 2026.
See when SemrushBot-ESI visits your site
Airglow names every AI agent on your website, replays what it did and links the visitors it sends you. Free for one website.