AI agent directory / Meta web indexer
Meta web indexer
Crawler from Meta
Renders pages to improve search results in Meta AI.
| Run by | Meta |
|---|---|
| Type | Crawler |
| Purpose | Search |
| Acts for | Its operator |
| robots.txt token | meta-webindexer |
| Documentation | Meta's documentation |
How Airglow identifies it
- DeclaredIt names itself in the User-Agent:
meta-webindexer.
Every visit in Airglow says which of these applied, so you can decide how much to trust it.
Seeing it on your site
It runs JavaScript, so the Airglow script sees it on any website, with no server setup. Its visits are replayed step by step: the pages it opened, what it clicked and where it gave up.
Allowing or blocking Meta web indexer
Meta web indexer reads the meta-webindexer group in your robots.txt. To keep it out of the whole site:
User-agent: meta-webindexer Disallow: /
Meta runs a separate bot for each purpose, so blocking Meta web indexer doesn't block its other agents listed below.
Airglow shows, for every agent that visits, whether your robots.txt lets it in, and helps you write rules by purpose (training, search, acting for a person).
Other agents from Meta
See when Meta web indexer visits your site
Airglow names every AI agent on your website, replays what it did and links the visitors it sends you. Free for one website.