AI agent directory / ShapBot
ShapBot
Crawler from Parallel Web Systems
Crawls public pages to build Parallel's search index, which AI apps query through Parallel's web APIs.
| Run by | Parallel Web Systems |
|---|---|
| Type | Crawler |
| Purpose | Search |
| Acts for | Its operator |
| robots.txt token | ShapBot |
| IP ranges | Published by Parallel Web Systems |
| Documentation | Parallel Web Systems's documentation |
How Airglow identifies it
- SignedIt can sign its requests (Web Bot Auth), so its identity can be verified.
- DeclaredIt names itself in the User-Agent:
ShapBot.
Every visit in Airglow says which of these applied, so you can decide how much to trust it.
Seeing it on your site
It reads pages without running JavaScript, so Airglow counts it through a server connection (Cloudflare). It's labeled as context, not replayed.
Allowing or blocking ShapBot
ShapBot reads the ShapBot group in your robots.txt. To keep it out of the whole site:
User-agent: ShapBot Disallow: /
Parallel Web Systems runs a separate bot for each purpose, so blocking ShapBot doesn't block its other agents listed below.
Airglow shows, for every agent that visits, whether your robots.txt lets it in, and helps you write rules by purpose (training, search, acting for a person).
Other agents from Parallel Web Systems
See when ShapBot visits your site
Airglow names every AI agent on your website, replays what it did and links the visitors it sends you. Free for one website.