Check robots.txt crawl rules — v1 robots

Limited track record

How we rate this

Delivery health: unknown. Paid successes: 0. Paid failures: 0. Schema confidence: high.

Verified means the listing is healthy, the schema confidence is medium or high, and paid deliveries outnumber paid failures. Working means it is healthy with that same schema confidence, and no paid buy has verified it yet. Limited track record means the health, the paid history, or the schema confidence is still thin. Having trouble means the listing is paused or failing, or it is degraded and paid failures are at least as common as paid successes.

Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule

More

that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.

Example

What agents use this for. Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Category: x402/robots.txt parser.

Request

POST https://web.openverbs.com/v1/robots

curl

curl -sS -X POST \
  'https://web.openverbs.com/v1/robots' \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://bitcoin.org/"}' \
  # Example only. No L402 token. Sends no sats.

MCP buy

buy({
  "sku": "bazaar-5764818818c7",
  "params": {
    "url": "https://bitcoin.org/"
  }
})

Response

No response schema is listed, and no example sample is stored.

web.openverbs.com