Crawler or Scanner? (Claude Skill)
Free Claude Skill, MIT licensed. Path classifier, crawler map and rDNS verification ported from the site's analytics code.
Short description
Tells a real Googlebot, GPTBot or ClaudeBot from a credential scanner wearing its name, using forward-confirmed reverse DNS and the requested path.
Detailed description
A triage playbook for scary log lines like /.env, /.aws/credentials and /.git. Check the response status first, classify the path, then verify the claimed identity with forward-confirmed rDNS (verified, forged, cloud-vm or unverified), and only then decide on action. Includes a log analyser and guidance for building crawler alerts that don't cry wolf. Where it comes from: My Slack alerts once said “Applebot read /.env.example”. It was a scanner on rented Google Cloud VMs spoofing crawler names, and every path returned 404.
What I use it for
Answering “was I hacked?” from access logs, and building honest server-side AI-crawler alerting.
Downloads
How to install
# Claude Code (personal, all projects) unzip crawler-or-scanner.zip -d ~/.claude/skills/ # Claude Code (one project) unzip crawler-or-scanner.zip -d .claude/skills/ # Claude.ai / Claude desktop Settings → Capabilities → Skills → Upload skill → choose the zip