Skip to content


2,394 3 56 MIT
v1.1.6 (27 Jun 2021) Feb 03 2020 58.1 thousand (month)

Gospider is a flexible CLI crawler powered by Go. It's a fast an intuative way to scrape simple web scraping targets that do not require complex logic or anti bot bypass.

Feature highlights: - Fast web crawling - Brute force and parse sitemap.xml - Parse robots.txt - Generate and verify link from JavaScript files - Link Finder - Find AWS-S3 from response source - Find subdomains from response source - Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault - Format output easy to Grep - Support Burp input - Crawl multiple sites in parallel - Random mobile/web User-Agent

Example Use

# crawl and find all links
$ gospider -s ""

# crawl speicific depth and save output
$ gospider -s "" --output data --depth 2

Alternatives / Similar

Other Languages

Was this page helpful?