robots.txt & Sitemap Analyzer x402 REST usage
Give it domains. For each it parses robots.txt (user-agents, allow/disallow, crawl-delay, declared sitemaps, blocks-all) and the XML sitemap(s) (URL counts, nested sitemaps, lastmod coverage), and flags crawlability issu...