Check robots.txt crawl rules — v1 robots
Limited track record
How we rate this
Delivery health: unknown. Paid successes: 0. Paid failures: 0. Schema confidence: high.
Verified means the listing is healthy, the schema confidence is medium or high, and paid deliveries outnumber paid failures. Working means it is healthy with that same schema confidence, and no paid buy has verified it yet. Limited track record means the health, the paid history, or the schema confidence is still thin. Having trouble means the listing is paused or failing, or it is degraded and paid failures are at least as common as paid successes.
Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule
More
that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.
Example
What agents use this for. Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Category: x402/robots.txt parser.
Request
POST https://web.openverbs.com/v1/robots
curl
curl -sS -X POST \
'https://web.openverbs.com/v1/robots' \
-H 'Content-Type: application/json' \
-d '{"url":"https://bitcoin.org/"}' \
# Example only. No L402 token. Sends no sats.
MCP buy
buy({
"sku": "bazaar-5764818818c7",
"params": {
"url": "https://bitcoin.org/"
}
})
Response
No response schema is listed, and no example sample is stored.
web.openverbs.com