Skip to content

spidr

792 2 16 MIT
0.7.1 (25 Jan 2024) Jul 25 2009 1.6 thousand (month)

Spidr is a Ruby gem that provides a simple and flexible way to spider and scrape websites. It allows you to easily navigate through a website, following links and scraping data as you go. It is built on top of Nokogiri, a popular Ruby gem for parsing and searching HTML and XML documents, and it provides a simple and intuitive API for defining and running web scraping operations.

One of the main features of Spidr is its ability to spider a website, following all the links on a page and visiting all the pages of a website. This allows you to easily and quickly scrape large amounts of data from a website, without having to manually specify which pages to visit.

In addition to its spidering capabilities, Spidr also provides a variety of other features that can simplify the web scraping process. It can automatically filter which links to follow and which pages to visit, it can handle cookies and authentication, and it can automatically store the scraped data in a database or file. It also provides a built-in support for parallelism and queueing to speed up the scraping process.

Example Use


require 'spidr'

Spidr.start_at("http://example.com") do |spider|
  spider.every_page do |page|
    puts "Visiting: #{page.url}"

    # Extract data from the page using Nokogiri
    doc = Nokogiri::HTML(page.body)
    title = doc.css("title").text
    puts "Title: #{title}"
  end
end

Alternatives / Similar


1,304 3.0.0 (1 year, 8 months ago) Dec 27 2011 compare

Other Languages

22,276 v2.1.0 (3 years ago) May 14 2018 compare
7,533 v1.3.4 (4 years ago) Feb 15 2020 compare
2,488 2024-05-09 (3 days ago) Jun 06 2019 compare
645 2024-05-11 (16 hours ago) Feb 09 2017 compare
51,043 2.11.1 (2 months ago) Jul 26 2019 compare
1,476 1.0.4 (1 year, 8 months ago) Nov 22 2014 compare
2,857 1.4.3 (7 months ago) Sep 04 2013 compare
5,630 v0.18.0 (1 year, 1 month ago) Aug 06 2019 compare
2,026 (2 years ago) Nov 20 2016 compare
6,623 1.5.0 (23 hours ago) Sep 10 2012 compare
2,891 v2.1.1 (5 months ago) Jul 17 2018 compare
5,964 1.1.14 (1 year, 9 months ago) Jul 26 2019 compare
3,012 1.5.0 (2 months ago) Sep 30 2018 compare
152 2.2.4 (3 years ago) Dec 22 2019 compare
3,219 0.9.13 (9 months ago) Jul 04 2017 compare
235 1.33.0 (2 months ago) Feb 05 2023 compare
1,733 0.8.5 (1 year, 8 months ago) Oct 17 2018 compare
10,534 1.1.9 (5 years ago) Aug 24 2018 compare
1,321 v3.2.0 (a month ago) Dec 27 2021 compare
493 3.0.0 (a month ago) May 04 2020 compare
197 1.0.0-beta8.4 (10 months ago) Apr 18 2019 compare
1,324 v0.7.2 (5 months ago) Mar 16 2013 compare
412 0.1.3 (9 months ago) Feb 20 2022 compare
298 v1.7.2 (a month ago) Apr 18 2022 compare