Agent Ready Crawlers agent bronze

List crawlers and agents

Lists rows of the Crawler Index (id, name, operator, robots.txt token, purpose, how the operator says it treats robots.txt, verification method, block rates, profile URL). Filter by operator, purpose, verification method, robots.txt reading or words. Call it to find a crawler id before get_crawler, or to answer which crawlers do X. Send a data part {"tool":"list_crawlers","arguments":{...}}; the reply is a completed task whose data artifact is the tool result.

• {"tool":"list_crawlers","arguments":{"purpose":"training"}}

One crawler in full

Returns one Crawler Index row by id, robots.txt token or name (for example gptbot, GPTBot or ClaudeBot): operator, exact user-agent string, purpose, the operator statement on robots.txt with its source URL, verification method and IP list URL, rate behaviour, the allow and refuse lines, the vendor rule names, both block rates and the profile page URL to cite. Send a data part {"tool":"get_crawler","arguments":{...}}; the reply is a completed task whose data artifact is the tool result.

• {"tool":"get_crawler","arguments":{"id":"gptbot"}}

Block rate of one robots.txt token

Returns how many sites in the crawl refuse one robots.txt token (for example GPTBot or ClaudeBot) at the path /: the parsed-site count, the refused count, the part where a robots.txt group names the token, the shares, and the 401 and 403 share of home pages in the same cut. Without stratum it returns all four site lists; with category it returns that category. Send a data part {"tool":"get_block_rate","arguments":{...}}; the reply is a completed task whose data artifact is the tool result.

• {"tool":"get_block_rate","arguments":{"token":"GPTBot"}}

How an operator lets you verify its requests

Returns one Verification Ledger row: the IP list URLs, the reverse DNS pattern, the Web Bot Auth key directory, the Cloudflare verified status, the card-network registry lines, a summary, the source URL and the date read, plus the operator's crawlers with their verification method. Call it to check a request is really from an operator. Send a data part {"tool":"get_verification","arguments":{...}}; the reply is a completed task whose data artifact is the tool result.

• {"tool":"get_verification","arguments":{"operator":"OpenAI"}}

The robots.txt lines for a set of crawlers

Returns the allow lines or the refuse lines the Crawler Index holds for up to 20 crawlers (robots.txt tokens or crawler ids), per crawler and joined into one robots.txt body in the order asked. It only joins the rows' own lines; crawlers without a robots.txt token are listed as unmatched. Call it when writing a robots.txt policy for named crawlers. Send a data part {"tool":"get_robots_lines","arguments":{...}}; the reply is a completed task whose data artifact is the tool result.

• {"tool":"get_robots_lines","arguments":{"tokens":["GPTBot","ClaudeBot"],"action":"refuse"}}

Search Agent Ready Crawlers

Send a text message with a question or search words; the reply is a completed task whose artifact lists the matching records and pages of Agent Ready Crawlers, each with its source URL.

• ClaudeBot block rate

Protected-resource metadata

RFC 9728 metadata stating that the API, the MCP servers and the ask endpoint take no credential (authorization_servers is empty).

Agent Ready Crawlers A2A agent

A2A 0.3.0 agent card: message/send at /api/a2a searches the site or runs one read-only tool.

Ask Agent Ready Crawlers

NLWeb 0.55 /ask over the site's records and pages: POST query.text or GET ?query=; results are schema.org objects with a source URL each.

Agent Ready Crawlers API

Read-only REST API over the site's datasets: one GET endpoint per tool under /api/v1/, a batch endpoint, documented at https://agentreadycrawlers.com/developers.

Agent Ready Crawlers documentation MCP server

Agent Ready Crawlers documentation: search the pages, read one as markdown, list the sections.

Agent Ready Crawlers MCP server

AI crawler index, robots.txt block rates and verification ledger, read-only.