Skip to content

ferret

5,513 7 52 Apache-2.0
v0.18.0 (28 Mar 2023) Aug 06 2019 58.1 thousand (month)

Ferret is a web scraping system. It aims to simplify data extraction from the web for UI testing, machine learning, analytics and more. ferret allows users to focus on the data. It abstracts away the technical details and complexity of underlying technologies using its own declarative language. It is extremely portable, extensible, and fast.

Features

  • Declarative language
  • Support of both static and dynamic web pages
  • Embeddable
  • Extensible

Ferret is always implemented in Python through pyfer

Example Use


// Example scraper for Google in Ferret:
LET google = DOCUMENT("https://www.google.com/", {
    driver: "cdp",
    userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.87 Safari/537.36"
})

HOVER(google, 'input[name="q"]')
WAIT(RAND(100))
INPUT(google, 'input[name="q"]', @criteria, 30)
WAIT(RAND(100))
CLICK(google, 'input[name="btnK"]')

WAITFOR EVENT "navigation" IN google

WAIT_ELEMENT(google, "#res")

LET results = ELEMENTS(google, X("//*[text() = 'Search Results']/following-sibling::*/*"))

FOR el IN results
    RETURN {
        title: INNER_TEXT(el, 'h3')?,
        description: INNER_TEXT(el, X("//em/parent::*")),
        url: ELEMENT(el, 'a')?.attributes.href
    }

Alternatives / Similar


21,215 v2.1.0 (3 years ago) May 14 2018 compare
7,476 v1.3.4 (3 years ago) Feb 15 2020 compare
2,296 2023-11-24 (4 days ago) Jun 06 2019 compare
622 2023-11-24 (4 days ago) Feb 09 2017 compare
2,021 (2 years ago) Nov 20 2016 compare

Other Languages

49,364 2.11.0 (2 months ago) Jul 26 2019 compare
1,428 1.0.3 (2 years ago) Nov 22 2014 compare
2,734 1.4.3 (2 months ago) Sep 04 2013 compare
6,541 1.4.0 (10 months ago) Sep 10 2012 compare
2,829 v2.1.0 (5 months ago) Jul 17 2018 compare
5,654 1.1.14 (1 year, 4 months ago) Jul 26 2019 compare
1,298 3.0.0 (1 year, 3 months ago) Dec 27 2011 compare
769 0.7.0 (10 months ago) Jul 25 2009 compare
150 2.2.4 (2 years ago) Dec 22 2019 compare
1,701 0.8.5 (1 year, 2 months ago) Oct 17 2018 compare
2,864 1.4.1 (2 months ago) Sep 30 2018 compare
3,118 0.9.13 (4 months ago) Jul 04 2017 compare
10,120 1.1.9 (5 years ago) Aug 24 2018 compare
1,271 2.0.1 (9 months ago) Dec 27 2021 compare
224 1.27.1 (3 months ago) Feb 05 2023 compare
443 2.0.0 (5 months ago) May 04 2020 compare
1,312 v0.7.1 (3 months ago) Mar 16 2013 compare
186 1.0.0-beta8.4 (5 months ago) Apr 18 2019 compare
386 0.1.3 (3 months ago) Feb 20 2022 compare
277 v1.3.0 (a month ago) Apr 18 2022 compare