Skip to content

pholcusvscrawl4ai

Apache-2.0 5 1 7,584
Feb 15 2020 v1.4.0(2026-03-03 03:58:32 ago)
87,360 7 347 Apache-2.0
May 01 2024 1.6 million (month) 0.9.5(2026-10-14 10:25:32 ago)

Pholcus is a minimalistic web crawler library written in the Go programming language. It is designed to be flexible and easy to use, and it supports concurrent, distributed, and modular crawling.

Note that Pholcus is documented and maintained in the Chinese language and has no english resources other than the code source itself.

Crawl4AI is an open-source AI-powered web crawling and data extraction library for Python. It uses large language models (LLMs) to intelligently extract structured data from web pages with minimal code. Unlike traditional scraping frameworks that rely on CSS selectors or XPath, Crawl4AI can understand page content semantically and extract data based on natural language descriptions of what you want.

Key features include:

  • LLM-based extraction Define what data you want in plain English and Crawl4AI uses LLMs to find and extract it from the page content. Supports multiple LLM providers including OpenAI, Anthropic, and local models.
  • Automatic crawling Built-in crawler with support for JavaScript rendering, parallel crawling, and session management.
  • Structured output Returns data in structured formats (JSON, Pydantic models) making it easy to integrate into data pipelines.
  • Markdown conversion Can convert web pages to clean markdown format, useful for feeding content to LLMs.
  • Chunking strategies Multiple strategies for breaking down large pages into processable chunks for LLM extraction.
  • Async support Built on async Python for efficient concurrent crawling and extraction.

Crawl4AI is particularly useful for scraping unstructured content where writing traditional CSS/XPath selectors would be tedious or fragile. It excels at content extraction, article parsing, and data mining from diverse page layouts.

Highlights


ai-poweredasyncpopular

Example Use


package main

import (
    "github.com/henrylee2cn/pholcus/exec"
    _ "github.com/henrylee2cn/pholcus/spider/standard" // standard spider
)

func main() {
    // create spider object
    spider := exec.NewSpider(exec.NewTask("demo", "https://www.example.com"))
    // add a callback for URL route by regex pattern. In this case it's any route:
    spider.AddRule(".*", "Parse")
    // Start spider
    spider.Start()
}

// define callback here
func Parse(self *exec.Spider, doc *goquery.Document) {
    // callbacks receive HTMl document reference and 
}
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction_strategy import LLMExtractionStrategy
import asyncio

async def main():
    # Basic crawling - get page as markdown
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com")
        print(result.markdown)  # clean markdown content

    # AI-powered extraction with structured output
    strategy = LLMExtractionStrategy(
        instruction="Extract all product names and prices from this page",
    )
    config = CrawlerRunConfig(extraction_strategy=strategy)

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com/products",
            config=config,
        )
        print(result.extracted_content)  # structured JSON output

asyncio.run(main())

Alternatives / Similar


Was this page helpful?