SiteCheck Web Intel — security robots-security-txt

Limited track record

How we rate this

Delivery health: unknown. Paid successes: 0. Paid failures: 0. Schema confidence: high.

Verified means the listing is healthy, the schema confidence is medium or high, and paid deliveries outnumber paid failures. Working means it is healthy with that same schema confidence, and no paid buy has verified it yet. Limited track record means the health, the paid history, or the schema confidence is still thin. Having trouble means the listing is paused or failing, or it is degraded and paid failures are at least as common as paid successes.

Reads a site's public robots.txt (crawl rules, sitemaps, AI-crawler rules) and security.txt (RFC 9116: contact, expiry, policy), and says whether security.txt is present, current

More

or expired. Missing files are reported as absent, not as errors. Usually answers in about 6 s, up to 23 s on slower calls; allow a 45 s timeout.

Example

What agents use this for. Reads a site's public robots.txt (crawl rules, sitemaps, AI-crawler rules) and security.txt (RFC 9116: contact, expiry, policy), and says whether security.txt is present, current or expired. Category: x402/security.

Request

GET https://api.webintel-api.workers.dev/api/security/robots-security-txt?domain=bitcoin.org

curl

curl -sS -X GET \
  'https://api.webintel-api.workers.dev/api/security/robots-security-txt?domain=bitcoin.org' \
  -H 'Accept: application/json' \
  # Example only. No L402 token. Sends no sats.

MCP buy

buy({
  "sku": "bazaar-e4dfcaf3f54c",
  "params": {
    "domain": "bitcoin.org"
  }
})

Response

No response schema is listed, and no example sample is stored.

api.webintel-api.workers.dev