katanavswombat
Katana is a next-generation web crawling and spidering framework written in Go by ProjectDiscovery. It is designed for fast, comprehensive endpoint and asset discovery and is widely used in the security research and bug bounty communities.
Katana offers multiple crawling modes:
- Standard mode Fast HTTP-based crawling without a browser. Parses HTML, JavaScript files, and other resources to discover endpoints and links.
- Headless mode Uses a headless Chrome browser for crawling JavaScript-rendered pages and single-page applications (SPAs).
- Passive mode Discovers URLs from external sources (Wayback Machine, CommonCrawl, etc.) without actively visiting the target.
Key features include:
- Scope control Configurable crawl scope with regex patterns for including/excluding URLs, domains, and file extensions.
- JavaScript parsing Extracts endpoints from JavaScript files, inline scripts, and AJAX requests even in standard (non-headless) mode.
- Customizable output Filter and format output with field selection, JSON output, and custom templates.
- Rate limiting Built-in rate limiting and concurrency control to avoid overwhelming targets.
- Proxy support HTTP and SOCKS5 proxy support with rotation.
- Form filling Can detect and auto-fill forms to discover endpoints behind form submissions.
While Katana was designed for security research and reconnaissance, its fast crawling capabilities and JavaScript parsing make it equally useful for web scraping discovery and sitemap generation.
Wombat is a Ruby gem that makes it easy to scrape websites and extract structured data from HTML pages. It is built on top of Nokogiri, a popular Ruby gem for parsing and searching HTML and XML documents, and it provides a simple and intuitive API for defining and running web scraping operations.
One of the main features of Wombat is its ability to extract structured data from HTML pages using a simple, CSS-like syntax. It allows you to define a set of rules for extracting data from a page, and then automatically applies those rules to the page's HTML to extract the desired data. This makes it easy to extract data from even complex and dynamic pages, without having to write a lot of custom code.
In addition to its data extraction capabilities, Wombat also provides a variety of other features that can simplify the web scraping process. It can automatically follow links and scrape multiple pages, it can handle pagination and AJAX requests, and it can handle cookies and authentication. It also provides a built-in support for parallelism and queueing to speed up the scraping process.