agenttru.st

The agenttru.st crawler

If you found agenttrustbot in your logs, this page is the answer.

What it does

agenttru.st discovers AI agents by reading public Certificate Transparency logs — the append-only record every publicly trusted certificate is written to — and then asking each hostname whether it publishes an agent. Asking means fetching a small, fixed set of documents:

That is the whole list on first contact. One addition for hosts already in the directory: when a listing is re-verified, /.well-known/ai-catalog.json is fetched as well — the host's ARD catalog, if it publishes one, shown on its listing as the publisher's own claims. Beyond these documents it does not scan ports, guess paths, follow links, crawl your site, or submit forms. It sends a POST only to an endpoint your own DNS record or your own published document told it to contact.

It also asks DNS about you, which for a host that does not exist is the only thing it does — three lookups and no requests at all:

_ans.<host> TXT
the ANS discovery record, which names an agent card URL
<host> SVCB
the DNS-AID declaration, which names the protocols a host speaks
<host> A and AAAA
to connect at all — and to refuse the address if it is not public
_ans-badge.<host> TXT
a transparency-log badge, only when an ANS record exists
_443._tcp.<host> TLSA
DANE, only for a host that already verified
_ans-identity._tls.<host> TLSA
the ANSv2 identity binding, only for a host that already verified

No zone transfers, no subdomain guessing, no wildcard probing, and no reverse lookups of your visitors. The last three run only for a host that already verified, so an ordinary name costs the first three and nothing more.

A host that publishes none of them costs 6 requests — over one connection where your server allows reuse — once, and is then left alone for 30 days — longer each time the answer stays the same. If you are reading this because of a burst in your logs, that burst was the whole visit.

How to stop it

Add this to your robots.txt:

User-agent: agenttrustbot
Disallow: /

It is honoured on every fetch above, and a host that declines is not asked again for 180 days. This crawler previously identified as agenttrust-collector; a rule naming that token is still honoured, and always will be — a rename must not orphan a rule an operator wrote once and had no reason to revisit. A rule naming agenttrustbot takes precedence over your wildcard rule, so you can block this crawler without changing anything else — or the reverse, if you block everything and want the discovery documents readable.

Two other ways to say no, neither of which needs a file:

How it behaves

Being identifiable is only worth something if the behaviour behind the name is bounded. What is bounded, and how:

Per site
At most 2 requests in flight per registrable domain, with visits to a domain starting at least 300ms apart. The spacing is between visits: a same-site redirect — your apex forwarding to www — is followed promptly inside the visit it belongs to, not treated as a new one. And because a domain and a server are two different questions: at most 1 walk at a time against one IP address, so if two of your sites share an ordinary server, they are not visited simultaneously.
One honest exception: an address observed serving many unrelated domains — shared hosting, CDN edges — is not held to one walk at a time, because serialising a platform protects nobody in particular. Every domain on it keeps its own bound above, which is the one that protects an operator.
These bounds apply across every machine we probe from, not per machine: what your server receives in total is what is bounded, however many machines this fleet runs.
Per visit
The DNS lookups listed above, 6 small GETs at most — one more, the catalog, when re-verifying an existing listing — over one connection — where your server permits reuse; a server that closes after each response is redialled per document, which is its choice to make — response bodies capped, one redirect hop, and a short timeout. A host that does not answer is not retried path by path — one failure ends the visit.
Not asked at all
Some hostnames are recognised and never contacted, because the provider that generated them has already decided the answer: object-storage API endpoints, which reject any non-object path by design, and cPanel service names such as cpcalendars and webdisk, which exist on every hosted domain and serve only an authentication prompt. These reach the certificate logs in bulk and asking them would be noise you pay for.
Over time
Every hostname that is asked is deduplicated and asked once. A confirmed agent is re-checked every 7 days. Anything else waits 30 days and then longer each time the answer holds, out to 90 days for a host that keeps saying no and 180 for one that stops answering at all.
Never
No port scanning, no path guessing, no vulnerability probing, no form submission, no authentication attempts, no cookies, no JavaScript execution, no evasion — one honest User-Agent, and no second identity if you block the first.

How to verify it is us

Anyone can put our User-Agent on their own traffic, so do not treat the name as evidence. Check the address instead: this fleet probes from addresses with forward-confirmed reverse DNS under agenttru.st, the same check Google and Bing document for their own crawlers.

Take the address from your log, look up its name, then look that name back up:

dig +short -x 51.83.6.2
server.agenttru.st.

dig +short server.agenttru.st
51.83.6.2

Both steps matter, and the second is the whole check. Whoever controls an address can make its reverse record claim any name they like — but only we can make a name under agenttru.st resolve back to the address that claimed it. If the second lookup does not return the address you started with, it is not this crawler, whatever the first one said. IPv6 answers the same way.

User-Agent
AgentTrustBot/1.0 (+https://agenttru.st/crawler; purpose=AI Agent Discovery; contact=admin@agenttru.st) — a courtesy, so you know who to look up. Not proof of anything.
Current addresses
  • 51.83.6.2
  • 74.208.40.151
  • 54.39.22.255
  • 51.38.70.50
  • 162.19.251.210
  • 109.123.229.136
  • 129.121.109.249
  • 50.6.54.51
  • 129.121.99.17
  • 129.121.133.131
Published so you know what to expect, but a list is only as fresh as its last edit and the fleet changes hosts. The reverse-DNS check above is true whenever it is true, which is why it is the one to rely on.
Anything else
Traffic outside the list of paths above, at a higher rate, or from an address that does not forward-confirm is not this crawler, whatever it calls itself. If something claiming this name is misbehaving — or if this one is — tell us. An address in an abuse report is one we can check against our own records, which is most of why any of this is published.

If we fetched your icon

A listed agent's favicon is fetched once and re-served from this site rather than linked to yours. That is deliberate, and it is for your visitors as much as ours: hot-linking would hand every listed operator the address of everyone browsing this directory, and let a listed party watch who is looking at them. Removal deletes the stored copy.

Why it exists

Nobody can currently enumerate what agents are running on the public internet, which makes it hard to say anything true about them. How this works covers the method and its limits in full.