agenttru.st

The agenttru.st crawler

If you found agenttrust-collector in your logs, this page is the answer.

What it does

agenttru.st discovers AI agents by reading public Certificate Transparency logs — the append-only record every publicly trusted certificate is written to — and then asking each hostname whether it publishes an agent. Asking means fetching a small, fixed set of documents:

That is the whole list. It does not scan ports, guess paths, follow links, crawl your site, or submit forms. It sends a POST only to an endpoint your own DNS record or your own published document told it to contact.

How to stop it

Add this to your robots.txt:

User-agent: agenttrust-collector
Disallow: /

It is honoured on every fetch above, and a host that declines is not asked again for six months. A rule naming agenttrust-collector takes precedence over your wildcard rule, so you can block this crawler without changing anything else — or the reverse, if you block everything and want the discovery documents readable.

If you are already listed and want the listing gone as well, request removal. That takes the page down and stops the probing; they are not separate requests.

How to identify it

User-Agent
agenttrust-collector (+https://agenttru.st/crawler)
Rate
At most 2 requests in flight per registrable domain, at least 300ms apart. Whatever else arrives is not this.
Addresses
Not published for this deployment.

Why it exists

Nobody can currently enumerate what agents are running on the public internet, which makes it hard to say anything true about them. How this works covers the method and its limits in full.