chromedpvssplash
ChromeDP is an open-source library for driving browsers using the Chrome DevTools Protocol (CDP) in the Go programming language. It is a high-level library that abstracts away the low-level details of interacting with the CDP and provides a simple, intuitive API for performing common browser automation tasks such as clicking elements, filling out forms, and taking screenshots.
ChromeDP also supports parallel execution of browser tasks, making it well-suited for large-scale web scraping and testing applications. It is considered as one of the most popular Go package for automation and scraping tasks.
Splash is a javascript rendering service with an HTTP API. It's a lightweight browser with an HTTP API, implemented in Python 3 using Twisted and QT5.
It is built on top of the QtWebkit library and allows developers to interact with web pages in a headless mode, which means that the web pages are rendered in the background, without displaying them on the screen.
splash is particularly useful for web scraping and web testing tasks, as it allows developers to interact with web pages in a way that is very similar to how a human user would interact with the browser.
It also allows you to execute javascript and interact with web pages even if they use heavy javascript.
Unlike Selenium or Playwright, splash is powered by webkit embedded browser instead of a real browser like Chrome or Firefox. As a down-side splash requests are easy to detect and block when scraping websites with anti-scraping features.
One benefit of splash is that it seemlesly integrates with Scrapy.
Example Use
package main
import (
"context"
"fmt"
"github.com/chromedp/chromedp"
)
func main() {
var title, firstParagraph string
// create context
ctx, cancel := chromedp.NewContext(context.Background())
defer cancel()
// run task list (a scraping scenario)
err := chromedp.Run(ctx,
// go to page
chromedp.Navigate("https://www.example.com"),
// wait for element to load
chromedp.WaitVisible("body"),
// extract text from an element (css selector)
chromedp.Text("title", &title),
// extract first paragraph element
chromedp.First(chromedp.ByTagName("p"), &firstParagraph),
)
if err != nil {
fmt.Println("error:", err)
return
}
fmt.Printf("Title: %s\n", title)
fmt.Printf("First paragraph: %s\n", firstParagraph)
}
# once splash server is started it can be requested to render pages through
# HTTP requests:
import requests
url = "http://localhost:8050/render.html"
payload = {
'url': 'https://www.example.com',
'timeout': 30,
'wait': 2
}
response = requests.get(url, params=payload)
# Get the page HTML
print(response.text)