Skip to content

wombat

1,298 2 23 MIT
3.0.0 (23 Aug 2022) Dec 27 2011 1.5 thousand (month)

Wombat is a Ruby gem that makes it easy to scrape websites and extract structured data from HTML pages. It is built on top of Nokogiri, a popular Ruby gem for parsing and searching HTML and XML documents, and it provides a simple and intuitive API for defining and running web scraping operations.

One of the main features of Wombat is its ability to extract structured data from HTML pages using a simple, CSS-like syntax. It allows you to define a set of rules for extracting data from a page, and then automatically applies those rules to the page's HTML to extract the desired data. This makes it easy to extract data from even complex and dynamic pages, without having to write a lot of custom code.

In addition to its data extraction capabilities, Wombat also provides a variety of other features that can simplify the web scraping process. It can automatically follow links and scrape multiple pages, it can handle pagination and AJAX requests, and it can handle cookies and authentication. It also provides a built-in support for parallelism and queueing to speed up the scraping process.

Example Use


require 'wombat'

Wombat.crawl do
  base_url "https://www.github.com"
  path "/"

  headline xpath: "//h1"
  subheading css: "p.alt-lead"

  what_is({ css: ".one-fourth h4" }, :list)

  links do
    explore xpath: '/html/body/header/div/div/nav[1]/a[4]' do |e|
      e.gsub(/Explore/, "Love")
    end

    features css: '.nav-item-opensource'
    business css: '.nav-item-business'
  end
end
will result in:
{
  "headline"=>"How people build software",
  "subheading"=>"Millions of developers use GitHub to build personal projects, support their businesses, and work together on open source technologies.",
  "what_is"=>[
    "For everything you build",
    "A better way to work",
    "Millions of projects",
    "One platform, from start to finish"
  ],
  "links"=>{
    "explore"=>"Love",
    "features"=>"Open source",
    "business"=>"Business"
  }
}

Alternatives / Similar


769 0.7.0 (10 months ago) Jul 25 2009 compare

Other Languages

21,215 v2.1.0 (3 years ago) May 14 2018 compare
7,476 v1.3.4 (3 years ago) Feb 15 2020 compare
2,296 2023-11-24 (4 days ago) Jun 06 2019 compare
622 2023-11-24 (4 days ago) Feb 09 2017 compare
49,364 2.11.0 (2 months ago) Jul 26 2019 compare
1,428 1.0.3 (2 years ago) Nov 22 2014 compare
2,734 1.4.3 (2 months ago) Sep 04 2013 compare
6,541 1.4.0 (10 months ago) Sep 10 2012 compare
5,513 v0.18.0 (8 months ago) Aug 06 2019 compare
2,021 (2 years ago) Nov 20 2016 compare
2,829 v2.1.0 (5 months ago) Jul 17 2018 compare
5,654 1.1.14 (1 year, 4 months ago) Jul 26 2019 compare
150 2.2.4 (2 years ago) Dec 22 2019 compare
1,701 0.8.5 (1 year, 2 months ago) Oct 17 2018 compare
2,864 1.4.1 (2 months ago) Sep 30 2018 compare
3,118 0.9.13 (4 months ago) Jul 04 2017 compare
10,120 1.1.9 (5 years ago) Aug 24 2018 compare
1,271 2.0.1 (9 months ago) Dec 27 2021 compare
224 1.27.1 (3 months ago) Feb 05 2023 compare
443 2.0.0 (5 months ago) May 04 2020 compare
1,312 v0.7.1 (3 months ago) Mar 16 2013 compare
186 1.0.0-beta8.4 (5 months ago) Apr 18 2019 compare
386 0.1.3 (3 months ago) Feb 20 2022 compare
277 v1.3.0 (a month ago) Apr 18 2022 compare