pholcus
Pholcus is a minimalistic web crawler library written in the Go programming language. It is designed to be flexible and easy to use, and it supports concurrent, distributed, and modular crawling.
Note that Pholcus is documented and maintained in the Chinese language and has no english resources other than the code source itself.
Example Use
```go package main
import ( "github.com/henrylee2cn/pholcus/exec" _ "github.com/henrylee2cn/pholcus/spider/standard" // standard spider )
func main() { // create spider object spider := exec.NewSpider(exec.NewTask("demo", "https://www.example.com")) // add a callback for URL route by regex pattern. In this case it's any route: spider.AddRule(".*", "Parse") // Start spider spider.Start() }
// define callback here func Parse(self exec.Spider, doc goquery.Document) { // callbacks receive HTMl document reference and } ```