Skip to content

colly

22,347 5 185 Apache-2.0
v2.1.0 (8 Jun 2020) May 14 2018

Colly is a popular web scraping library for the Go programming language. It's designed to be fast and easy to use, and it provides a simple and flexible API for traversing and extracting information from websites.

Colly supports:

  • Concurrent scraping with a simple API
  • Automatic handling of cookies and sessions
  • Automatic handling of redirects
  • Support for parsing HTML and XML
  • Support for parsing JSON and binary data
  • Support for custom storage (e.g. scraping results to a database)
  • Simple JavaScript rendering with Colly's built-in rendering engine.

Colly also provides several optional features, such as support for user-agents, delay between requests, rate-limiting and proxy usage.

Colly's API is quite simple, and it is easy to get started with basic web scraping tasks. It's a good choice for scraping moderate to heavy sites, and it can be useful for a wide range of use cases, such as data mining, content extraction, and more.

Additionally, you can use it together with Goquery, a library that allow you to make jquery like queries on HTML documents and it is often used together with Colly to ease the way of parsing the HTML.

Highlights


popularcss-selectorsxpath-selectorscommunity-toolsoutput-pipelinesmiddlewaresasyncproductionlarge-scale

Example Use


package main

import (
  "fmt"

  "github.com/gocolly/colly/v2"
)

func main() {
  // Instantiate default collector
  c := colly.NewCollector(
    // Visit only domains: hackerspaces.org, wiki.hackerspaces.org
    colly.AllowedDomains("hackerspaces.org", "wiki.hackerspaces.org"),
  )

  // On every a element which has href attribute call callback
  c.OnHTML("a[href]", func(e *colly.HTMLElement) {
    link := e.Attr("href")
    // Print link
    fmt.Printf("Link found: %q -> %s\n", e.Text, link)
    // Visit link found on page
    // Only those links are visited which are in AllowedDomains
    c.Visit(e.Request.AbsoluteURL(link))
  })

  // Before making a request print "Visiting ..."
  c.OnRequest(func(r *colly.Request) {
    fmt.Println("Visiting", r.URL.String())
  })

  // Start scraping on https://hackerspaces.org
  c.Visit("https://hackerspaces.org/")
}

Alternatives / Similar


7,533 v1.3.4 (4 years ago) Feb 15 2020 compare
2,488 2024-05-09 (17 days ago) Jun 06 2019 compare
645 2024-05-11 (15 days ago) Feb 09 2017 compare
5,641 v0.18.0 (1 year, 1 month ago) Aug 06 2019 compare
2,025 (3 years ago) Nov 20 2016 compare

Other Languages

51,221 2.11.2 (12 days ago) Jul 26 2019 compare
1,479 1.0.4 (1 year, 9 months ago) Nov 22 2014 compare
6,630 1.5.0 (11 days ago) Sep 10 2012 compare
2,862 1.4.3 (8 months ago) Sep 04 2013 compare
2,892 v2.1.1 (5 months ago) Jul 17 2018 compare
5,994 1.1.14 (1 year, 10 months ago) Jul 26 2019 compare
793 0.7.1 (4 months ago) Jul 25 2009 compare
1,305 3.0.0 (1 year, 9 months ago) Dec 27 2011 compare
3,015 1.5.0 (3 months ago) Sep 30 2018 compare
152 2.2.4 (3 years ago) Dec 22 2019 compare
239 1.33.0 (3 months ago) Feb 05 2023 compare
3,229 0.9.13 (10 months ago) Jul 04 2017 compare
1,734 0.8.5 (1 year, 8 months ago) Oct 17 2018 compare
10,575 1.1.9 (5 years ago) Aug 24 2018 compare
1,327 v3.2.0 (a month ago) Dec 27 2021 compare
198 1.0.0-beta8.4 (10 months ago) Apr 18 2019 compare
498 3.0.0 (a month ago) May 04 2020 compare
1,325 v0.7.2 (5 months ago) Mar 16 2013 compare
412 0.1.3 (9 months ago) Feb 20 2022 compare
300 v1.7.2 (2 months ago) Apr 18 2022 compare
Was this page helpful?