Skip to content

geziyor

2,667 1 27 MPL-2.0
2025-02-18 (18 Feb 2025) Jun 06 2019

Geziyor is a blazing fast web crawling and web scraping framework. It can be used to crawl websites and extract structured data from them. Geziyor is useful for a wide range of purposes such as data mining, monitoring and automated testing.

Features:

  • JS Rendering
  • 5.000+ Requests/Sec
  • Caching (Memory/Disk/LevelDB)
  • Automatic Data Exporting (JSON, CSV, or custom)
  • Metrics (Prometheus, Expvar, or custom)
  • Limit Concurrency (Global/Per Domain)
  • Request Delays (Constant/Randomized)
  • Cookies, Middlewares, robots.txt
  • Automatic response decoding to UTF-8
  • Proxy management (Single, Round-Robin, Custom)

Example Use


// This example extracts all quotes from quotes.toscrape.com and exports to JSON file.
func main() {
    geziyor.NewGeziyor(&geziyor.Options{
        StartURLs: []string{"http://quotes.toscrape.com/"},
        ParseFunc: quotesParse,
        Exporters: []export.Exporter{&export.JSON{}},
    }).Start()
}

func quotesParse(g *geziyor.Geziyor, r *client.Response) {
    r.HTMLDoc.Find("div.quote").Each(func(i int, s *goquery.Selection) {
        g.Exports <- map[string]interface{}{
            "text":   s.Find("span.text").Text(),
            "author": s.Find("small.author").Text(),
        }
    })
    if href, ok := r.HTMLDoc.Find("li.next > a").Attr("href"); ok {
        g.Get(r.JoinURL(href), quotesParse)
    }
}

Alternatives / Similar


23,747 v2.1.0 (4 years ago) May 14 2018 compare
7,580 v1.3.4 (5 years ago) Feb 15 2020 compare
676 2025-02-16 (a day ago) Feb 09 2017 compare
5,716 v0.18.0 (1 year, 10 months ago) Aug 06 2019 compare
2,039 (3 years ago) Nov 20 2016 compare

Other Languages

54,211 2.12.0 (3 months ago) Jul 26 2019 compare
1,498 1.0.4 (2 years ago) Nov 22 2014 compare
2,980 1.5.0 (4 months ago) Sep 04 2013 compare
6,733 2.0.2 (7 months ago) Sep 10 2012 compare
2,977 v2.2.0 (19 days ago) Jul 17 2018 compare
6,638 1.1.14 (2 years ago) Jul 26 2019 compare
247 1.34.0 (2 months ago) Feb 05 2023 compare
813 0.7.2 (15 days ago) Jul 25 2009 compare
3,218 1.6.0 (2 days ago) Sep 30 2018 compare
3,365 0.9.13 (1 year, 7 months ago) Jul 04 2017 compare
1,316 3.0.0 (2 years ago) Dec 27 2011 compare
1,754 0.8.5 (2 years ago) Oct 17 2018 compare
11,149 1.1.9 (6 years ago) Aug 24 2018 compare
156 2.2.4 (3 years ago) Dec 22 2019 compare
1,384 v3.2.0 (10 months ago) Dec 27 2021 compare
428 0.1.3 (1 year, 6 months ago) Feb 20 2022 compare
213 1.0.0-beta8.4 (1 year, 7 months ago) Apr 18 2019 compare
554 3.0.0 (10 months ago) May 04 2020 compare
1,335 v0.7.2 (1 year, 2 months ago) Mar 16 2013 compare
356 v3.2.3 (21 days ago) Apr 18 2022 compare
Was this page helpful?