agenttru.st

The agenttru.st crawler

If you found agenttrustbot in your logs, this page is the answer.

What it does

agenttru.st finds AI agents by reading hostnames from public Certificate Transparency logs — the record every publicly trusted certificate is written to — and asking each one whether it publishes an agent. A few names reach us another way — from a listed agent's own catalog, or from the certificate a listed agent is served under on its own site, which can name other hosts too — and are asked the same way. A name a certificate puts forward is added only if it is not already one of the hosts we check: one we already know is not asked again, or sooner, for being named. Asking means fetching:

Those are the only fixed paths a first visit asks for; a host whose own DNS records or documents name an endpoint is also asked for that endpoint (see If you publish an agent). For the overwhelming majority of hosts, which publish no agent and have none listed under their domain, that is the whole list ever.

A host that publishes none of them, and answers each one directly rather than with a redirect, costs 6 requests — one after another, over one connection where your server allows reuse — and that name is then left alone for at least 30 days, and for longer once the same answer repeats. When the host we came to visit answers 403 to our request for its own robots.txt — or redirects that request, every hop staying on that host or its other name (www or bare) and on the same port, and is answered 403 where the redirect ends — nothing after it is asked of that host or its other name. A redirect of one of the documents, to another path on the same host or between your apex and its www name, is followed once, so each redirected document costs one more request to wherever you sent it: your homepage or a login page, if that is where unknown paths go, or the same path at your other name, after that name's robots.txt. A redirect of robots.txt itself is followed up to five times, one request per hop, to any path on your host or its www name, or to another host's /robots.txt, and what it leads to is read as the file. Your apex and its www name are two names with a schedule each, so when one redirects to the other, the other hears from us on both schedules.

If you are reading this because of a burst in your logs and your host publishes no agent, that burst was the whole visit. If it found one, this site checks it for itself soon after — asking also the documents a listing's re-check asks for — and, once the listing is published, visits once more for its icon.

It also asks DNS about you. For a name that does not exist, that is all it does:

_ans.<host> TXT
the Agent Name Service (ANS) record, in which an agent can name its protocol and the URL of its card or endpoint
<host> SVCB
a DNS-AID record, in which a host can declare the agent protocols it speaks; by default asked only of names we have reason to think are agents, and of listings
<host> A and AAAA
to connect, and to refuse any address that is not public; asked of each name a visit reaches
_ans-badge.<host> TXT
the ANS badge record, which can point to a public transparency log; only when an ANS record exists
_<port>._tcp.<host> TLSA
DANE, a certificate the domain pins in DNS; only for an agent that has already verified, asked of the host and port it was found on (443, unless the URL in its ANS record names another port), which may be another party's
_ans-identity._tls.<host> TLSA
the Agent Name Service's own form of that pin; only for an agent that has already verified, and not when the DANE check already matched

No zone transfers, no subdomain guessing, no wildcard probing and no reverse lookups. A name that publishes nothing costs three queries per visit — the ANS record, A and AAAA — or four when the SVCB question is asked, and two more for each further name a visit reaches, such as a redirect to www. Those are the questions we put to our own resolver; what reaches your nameservers depends on its cache and its DNSSEC validation.

How to stop it

Add this to your robots.txt:

User-agent: agenttrustbot
Disallow: /

It is read per hostname and port — rules at example.com do not cover api.example.com, or example.com:8443 — and honoured on every visit this page describes; only the file's own redirects are followed without it, as the robots.txt standard provides. The name is matched whole and regardless of case, up to the first character that is not a letter, a hyphen or an underscore: AgentTrustBot/1.0 is us, agenttrustbot-images is not. A rule naming our former token, agenttrust-collector, is still honoured and always will be. To keep other crawlers out and leave the discovery documents open to us, see robots.txt on the about page.

If your file refuses /.well-known/agent-card.json, or the page a request for it is redirected to, as Disallow: / does, and nothing else we may read turns out to be an agent, that hostname is not checked for itself again for 180 days, other than on the rare occasion given under Limits; if it is listed, the listing comes down at its next 7-day check. Three things delay this: an endpoint your ANS record names that does not answer keeps the host on the ordinary schedule; a listing comes down only after further checks when that happens, when the lookup of your ANS record fails, or while your site answers 429; and a listing whose host is on a network we have been asked not to contact is not checked at all, so it comes down only after 182 days, unless you request removal. A visit made for another name can still reach it — by a redirect, for the icon of a listing under your registrable domain, or to check a site whose records or documents name your host — and reads your robots.txt first and obeys it, unless it is your other name's robots.txt that redirects there. A refusal of anything else is simply obeyed, and the host is revisited on the ordinary schedule.

A robots.txt that does not exist — any 4xx but 429 — means no rules, with one exception: a 403 to our request for the robots.txt of the host we came to visit — that file itself, or where its own redirect leads when every hop stays on that host or its other name and on the same port — ends that visit there, with nothing else asked of that host or its other name, although the standard would let us read it as having no rules. A 403 from a file a redirect led to by way of any host but yours and your other name, or another port, or from your other name's file when a document's redirect sends us there and we ask for that file on its own account, means no rules like any other 4xx. If the host is listed, a check that ends this way counts as one that went unanswered, and the listing comes down only after several in a row go unanswered. One we cannot read — a 5xx, a timeout, a refused connection — means the visit ends there with nothing else asked, as RFC 9309, the robots.txt standard, requires. If you are listed, a file your own server answers badly is not held against the listing, and once it has been unreadable for 30 days it is read as having no rules for one visit, as the same RFC allows. How we read robots.txt gives the exact cases, and how its redirects are followed.

Two other ways to say no, neither of which needs a file:

If you run a network — a hosting provider, say, whose customers' names keep reaching us — you can ask us to exclude it whatever those names are. Within a few minutes of it going on our list, our crawlers and the hub's own checks open no connection to any address in it: a crawler too old to know the list is given no work while there is one. Its names are still looked up in DNS, since that is how we learn an address, so if your network runs its customers' nameservers, those lookups still reach it.

How to verify it is us

Anyone can put our User-Agent on their own traffic, so do not treat the name as evidence. Check the address instead. Most of our addresses have forward-confirmed reverse DNS under agenttru.st, the same check Google and Bing document for their own crawlers. Look up the name of the address in your log, then look that name back up:

dig +short -x 51.83.6.2
server.agenttru.st.

dig +short server.agenttru.st
51.83.6.2

The second lookup is the real check. Whoever controls an address can make its reverse record claim any name they like, but only we can make a name under agenttru.st resolve back to that address. An address that fails this check but appears in the list below is still ours. If it does neither, it is not this crawler. IPv6 works the same way.

User-Agent
AgentTrustBot/1.0 (id=3f9a2b1c8d4e; +https://agenttru.st/crawler; purpose=AI agent directory; contact=admin@agenttru.st) — a courtesy, so you know who to look up. Not proof of anything.
Current addresses
  • 51.83.6.2
  • 74.208.40.151
  • 54.39.22.255
  • 51.38.70.50
  • 162.19.251.210
  • 109.123.229.136
  • 129.121.109.249
  • 50.6.54.51
  • 129.121.99.17
  • 129.121.133.131
Every address on this list is ours, including any whose reverse DNS does not confirm yet.
Anything else
None of these is this crawler, whatever it calls itself: a request for a path — a Certificate Transparency log's own API aside — that is neither listed on this page nor named by a record, document, icon link or redirect pointing at your host; more visits at once than this page allows; or a request from an address that neither forward-confirms under agenttru.st nor appears in the list above. If something using this name misbehaves — or this one does — tell us.

Every request our crawlers make on a visit carries an id, and quoting it back is the fastest way to get an answer. In AgentTrustBot/1.0 (id=3f9a2b1c8d4e; +https://agenttru.st/crawler; purpose=AI agent directory; contact=admin@agenttru.st), the twelve hexadecimal characters identify that one visit. Send us that line, or just the id, and we can tell you which host was visited, when, from which of our addresses and how many requests the visit made — and stop it. We can usually look one up for 2 days after the visit. Without it, tell us the hostname that was visited, and we can list our crawlers' visits made for it over the same 2 days, each with its time, our address and its request count — though not a visit whose result never reached us, nor one that reached your host on another name's behalf, which is listed under that name, nor this site's own visits, which carry no id. A report naming only one of our addresses and a time leaves hundreds of candidates or more — every host that address visited in the minute around it — and we usually cannot tell which was yours.

The id is random: it identifies the visit to us, and says nothing about you to anyone. Only our crawlers' visits carry one. Nothing this site sends does — not its check of a host a crawler found, its re-checks of listings or its icon visits — and nor does reading a Certificate Transparency log.

A rule matching our whole User-Agent exactly cannot work, because the id changes with every visit. Match the AgentTrustBot/1.0 token or this page's address instead; neither changes from visit to visit.

Limits

Per site
At most 2 visits at once per registrable domain — example.com, for www.example.com or api.example.com — and at most 1 visit at a time to any one IP address, so two of your sites on one ordinary server are not visited at once. These limits apply across every machine we probe from, not per machine.
An address we have seen serving 8 or more unrelated registrable domains — a CDN edge, a large shared host — is not held to one visit at a time. Every domain on it still keeps its own limit, and if that is still too much for yours, answer 429. If our crawlers lose contact with one another, each keeps to these limits only among the visits it starts while out of touch, so until they are back in step a site can receive more visits at once than these limits.
Per visit
One request at a time, over one connection where your server allows reuse. Response bodies are capped, timeouts are short, and a GET follows at most one redirect, only to the same host or its www name — robots.txt up to five. A host that stops answering is not asked path after path: if a connection cannot be made, the visit ends at once; if requests on an open connection go unanswered, the second one ends it; and so does a 429, and a 403 to our request for the visited host's own robots.txt, or at the end of that file's own redirect when every hop stays on the host or its other name and on the same port. A 403 from its other name's file, when a document's redirect sends us there, or from another host's, means no rules, and a 403 to any other request refuses only that document.
Over time
Each hostname is asked once, then again on a schedule: every 7 days while we find an agent there, listed or not; anything else after 30 days, and usually longer each time the answer holds: out to 90 days for a host that keeps saying there is no agent, and 180 for one that keeps answering with an error, or not at all. A 402, asking to be paid for the card, earns 180 days, as a robots.txt refusal of it usually does. When a re-check takes a listing down for any reason but one of those two, its host is asked once more 7 days later. When no agent is found and robots.txt, or the first request after it, was answered 429, or 503 with a Retry-After, our crawlers ask that host again after 30 days each time; a 429 later in the visit can leave it on the longer schedule instead.
A crawler visit we cut short ourselves — because a site it depends on had asked us for quiet, or to stay within the limits above — or whose result never reached us is repeated no sooner than a day later; one cut short after it had sent a request is repeated at most 6 times before that schedule resumes. This site's own check of a newly found agent can be repeated sooner. When the agent, or a site it depends on, asks for quiet, the check waits as long as asked — from 1 minute up to 7 days — and runs again, up to 26 times. When we cut it short ourselves, or a robots.txt it needs was answered badly, it normally runs again within minutes if it had sent nothing, or an hour later if it had, at intervals doubling up to a day — for a robots.txt, for up to 30 days. A re-check is repeated as given under If you publish an agent. A host can occasionally be visited outside this schedule — for example when its operator asks us to look again — even inside the 180 days a refusal earns, while it is listed, or inside the 30 days a “no icon” is believed. Such a visit reads robots.txt first and obeys it, and keeps to every limit on this page.
Never
No port scanning, no path guessing, no vulnerability probing, no form submission, no authentication attempts, no cookies, no JavaScript execution, no evasion — one honest User-Agent, and no second identity if you block the first. No request we send carries an Authorization header, and a URL with credentials in it is refused rather than fetched.

If you publish an agent

A host that does publish something pays for what it publishes. Each discovery record or document that names an endpoint costs a request to that endpoint — one GET for an agent card, at most 2 small POSTs for an MCP (Model Context Protocol) server, counted per endpoint, not per record — for up to 4 records, and for the one endpoint each of your own MCP server cards names. A POST only ever goes to an endpoint a host's own DNS record or published document named.

An endpoint on a name the visit has not asked yet costs that name's robots.txt first, and is held to it. One on any host but the one being visited and its www name — another subdomain of yours included — is contacted only on port 443 or 8443. Your DNS records and robots.txt redirects can send one visit to at most 2 such parties, and your own MCP server cards to at most 2 more, on the same ports.

Once a host is listed, our crawlers leave it alone and this site re-checks it instead, in a single visit every 7 days — or two back to back when your robots.txt has been unreadable for 30 days, the second reading it as having no rules. That re-check asks for the documents above, and also:

/.well-known/ai-catalog.json
your Agentic Resource Discovery (ARD) catalog, if you publish one, shown on your listing as your own claims; the URLs in it are not fetched, but other hostnames they name are queued like any name from the certificate logs
/llms.txt
noted as evidence of what you publish for agents; the links in it are not fetched
/.well-known/http-message-signatures-directory
the key directory a bot that signs its requests publishes (Web Bot Auth), noted the same way

The icon shown beside a listing comes from a short visit of its own, made, normally, at most every 30 days once the host has answered one, with the exceptions given below. It asks for:

/favicon.ico
the site icon
/
the homepage, read only for its <link rel=icon>, and only when /favicon.ico gave us nothing we can use — the one ordinary web page this crawler ever asks for

When your homepage links an icon — rel=icon, apple-touch-icon and the like — the first such link we can read, over https to the same host or its www name, is fetched too, if the robots.txt there allows its path; later links are not tried, and nothing else on the page is followed. When the listed host has no icon we can use, the same is asked of your registrable domain — unless it, or its www name, was removed at its operator's request — since an agent often lives on a subdomain with no homepage of its own.

So a listed host can receive two visits from us in the same week — the re-check and the icon visit — as separate sessions, each opening its own connection and asking for robots.txt first. It can receive more when a re-check has to be repeated: one we cut short ourselves, to stay within the limits above, is run again, usually an hour later and then at doubling intervals up to once a day, until one is completed. Every request count on this page is per visit, not per week.

If we fetched your icon

A listed host that has answered an icon visit is normally sent another at most once every 30 days. When it has no icon we can use, your registrable domain is asked instead, and its answer, icon or none, is believed for 30 days and reused for every listing under it. A homepage, or the icon file it names, that we could not read — an error status, no reply, or an image in a format we do not use, such as SVG — is no answer: the domain is asked again after 7 days, doubling up to 30, though if the domain is itself listed, its own listing's icon visit may ask it sooner. A visit that was turned away — your 429, our own pacing, a robots.txt we could not read — records no answer, icon or none, for the host it was sent to; how soon it is tried again depends on who turned it away.

We serve a copy of your icon from this site rather than linking to yours, so a listed site cannot see who is browsing the directory. Removal deletes that listing's stored copy, and the copy kept for your registrable domain too unless another listing under the domain is still published; a domain's copy is in any case deleted 60 days after it was last fetched. A copy another listing under the domain took from it stays with that listing, and is deleted when that listing is removed. The link-preview image that shows your icon stops being served the moment the listing does. We ask caches we do not own — a CDN, someone's chat app — to keep either picture no longer than 24 hours.

Why it exists

Nobody can currently enumerate what agents are running on the public internet, which makes it hard to say anything true about them. How this works explains the method and its limits.