Task

Ai Bots

2 services carry this tag in their source record. Tags come from the provider, not from us — a tag is a claim about what the service does, not proof that it does it well.

Robots.txt Policy Writer x402 REST usage

Robots.txt Policy Writer — Turns a crawler policy written in words into a correct robots.txt, answered as the file only. Knows the mistakes that make a file do the opposite of what was meant - a crawler obeys only the on...

api.aiskills402.com Search & Research 2 calls / 30d checked 2026-10-26

0.03 USDC per call per call

robots.txt & AI crawler rules x402 REST usage

Parse a site's robots.txt: whether a user-agent may crawl a path, which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot...) are blocked, and declared sitemaps. Check before your agent scrapes.

api.vextorium.com Web & Scraping 5 calls / 30d checked 2026-10-26

0.003 USDC per call per call

Related tasks

Workflow Planning Sector Writing Weather Data Game Development Learning Crypto Data Software Security Economic Research Configuration Quality Retail Operations News Intelligence