Skip to content

gospider

2,341 3 51 MIT
v1.1.6 (27 Jun 2021) Feb 03 2020 58.1 thousand (month)

Gospider is a flexible CLI crawler powered by Go. It's a fast an intuative way to scrape simple web scraping targets that do not require complex logic or anti bot bypass.

Feature highlights: - Fast web crawling - Brute force and parse sitemap.xml - Parse robots.txt - Generate and verify link from JavaScript files - Link Finder - Find AWS-S3 from response source - Find subdomains from response source - Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault - Format output easy to Grep - Support Burp input - Crawl multiple sites in parallel - Random mobile/web User-Agent

Example Use


# crawl and find all links
$ gospider -s "https://webscraping.fyi/lib/language/go/"

# crawl speicific depth and save output
$ gospider -s "https://webscraping.fyi/lib/language/go/" --output data --depth 2

Alternatives / Similar


Other Languages