Skip to content

chromedpvsrequestium

MIT 134 6 11,155
58.1 thousand (month) Nov 27 2019 chromedp(a month ago)
1,835 2 6 BSD-3-Clause
Dec 28 2012 52.3 thousand (month) 0.4.0(10 months ago)

ChromeDP is an open-source library for driving browsers using the Chrome DevTools Protocol (CDP) in the Go programming language. It is a high-level library that abstracts away the low-level details of interacting with the CDP and provides a simple, intuitive API for performing common browser automation tasks such as clicking elements, filling out forms, and taking screenshots.

ChromeDP also supports parallel execution of browser tasks, making it well-suited for large-scale web scraping and testing applications. It is considered as one of the most popular Go package for automation and scraping tasks.

Requestium is a Python library that merges the power of Requests, Selenium, and Parsel into a single integrated tool for automatizing web actions.

The library was created for writing web automation scripts that are written using mostly Requests but that are able to seamlessly switch to Selenium for the JavaScript heavy parts of the website, while maintaining the session.

Requestium adds independent improvements to both Requests and Selenium, and every new feature is lazily evaluated, so its useful even if writing scripts that use only Requests or Selenium.

Example Use


package main

import (
  "context"
  "fmt"

  "github.com/chromedp/chromedp"
)

func main() {
  var title, firstParagraph string

  // create context
  ctx, cancel := chromedp.NewContext(context.Background())
  defer cancel()

  // run task list (a scraping scenario)
  err := chromedp.Run(ctx,
    // go to page
    chromedp.Navigate("https://www.example.com"),
    // wait for element to load
    chromedp.WaitVisible("body"),
    // extract text from an element (css selector)
    chromedp.Text("title", &title),
    // extract first paragraph element
    chromedp.First(chromedp.ByTagName("p"), &firstParagraph),
  )
  if err != nil {
    fmt.Println("error:", err)
    return
  }

  fmt.Printf("Title: %s\n", title)
  fmt.Printf("First paragraph: %s\n", firstParagraph)
}
from requestium import Session, Keys

session = Session(webdriver_path='./chromedriver',
            browser='chrome-headless',
            default_timeout=15)

# then session object can be used like requests and parsel:
title = session.get('http://samplesite.com').xpath('//title/text()').extract_first(default='Default Title')

# other advance functions like POST requests and proxy settings are also available:
s.post('http://www.samplesite.com/sample', data={'field1': 'data1'})
s.proxies.update({'http': 'http://10.11.4.254:3128', 'https': 'https://10.11.4.252:3128'})

# session can also be used like selenium as it exposes all selenium functions.
# like typing keys:
s.driver.find_element_by_xpath("//input[@class='user_name']").send_keys('James Bond', Keys.ENTER)

Alternatives / Similar


Was this page helpful?