gospider
Gospider is a flexible CLI crawler powered by Go. It's a fast an intuative way to scrape simple web scraping targets that do not require complex logic or anti bot bypass.
Feature highlights: - Fast web crawling - Brute force and parse sitemap.xml - Parse robots.txt - Generate and verify link from JavaScript files - Link Finder - Find AWS-S3 from response source - Find subdomains from response source - Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault - Format output easy to Grep - Support Burp input - Crawl multiple sites in parallel - Random mobile/web User-Agent
Example Use
# crawl and find all links
$ gospider -s "https://webscraping.fyi/lib/language/go/"
# crawl speicific depth and save output
$ gospider -s "https://webscraping.fyi/lib/language/go/" --output data --depth 2