puppeteer-extravscurl-cffi

javascript http-clientstealth

python http-clientstealth

MIT 268 2 7,293

2.0 million (month) May 15 2018 3.3.6(2023-03-01 12:19:02 ago)

1,751 2 34 MIT

Feb 23 2022 594.9 thousand (month) 0.7.1(2024-07-13 09:07:25 ago)

---

Puppeteer-extra is a modular plugin framework that wraps Puppeteer (and Playwright) to add extra functionality through a plugin system. It acts as a drop-in replacement for Puppeteer while enabling powerful extensions for stealth, captcha solving, ad blocking, and more.

The most popular plugins include:

• puppeteer-extra-plugin-stealth Applies various evasion techniques to make the automated browser harder to detect. Patches common detection vectors like navigator.webdriver, Chrome.runtime, WebGL renderer strings, and more. This is the most widely used Puppeteer stealth solution.
• puppeteer-extra-plugin-recaptcha Automatically detects and solves reCAPTCHA and hCaptcha challenges using third-party solving services (2captcha, anti-captcha).
• puppeteer-extra-plugin-adblocker Blocks ads and trackers to speed up page loading and reduce bandwidth usage during scraping.
• puppeteer-extra-plugin-anonymize-ua Randomizes the User-Agent string to avoid fingerprinting.

Key features of the framework:

• Drop-in replacement Use puppeteer-extra instead of puppeteer in your imports - existing code works without changes.
• Plugin composition Multiple plugins can be stacked and they work together without conflicts.
• Playwright support The same plugin system works with Playwright via playwright-extra.
• Community plugins Active community creating and maintaining plugins for various use cases.

Puppeteer-extra is the go-to solution for adding stealth capabilities to Puppeteer-based scrapers without rewriting existing code.

Curl-cffi is a Python library for implementing curl-impersonate which is a HTTP client that appears as one of popular web browsers like: - Google Chrome - Microsoft Edge - Safari - Firefox Unlike requests and httpx which are native Python libraries, curl-cffi uses cURL and inherits it's powerful features like extensive HTTP protocol support and detection patches for TLS and HTTP fingerprinting.

Using curl-cffi web scrapers can bypass TLS and HTTP fingerprinting.

Highlights

---

plugin-systemextendiblecommunity-toolsstealth

bypasshttp2tls-fingerprinthttp-fingerprintsyncasync

Example Use

---

```javascript const puppeteer = require('puppeteer-extra'); // Add stealth plugin to avoid bot detection const StealthPlugin = require('puppeteer-extra-plugin-stealth'); puppeteer.use(StealthPlugin()); // Add recaptcha solving plugin const RecaptchaPlugin = require('puppeteer-extra-plugin-recaptcha'); puppeteer.use(RecaptchaPlugin({ provider: { id: '2captcha', token: 'YOUR_API_KEY' }, })); (async () => { // Launch browser - stealth is applied automatically const browser = await puppeteer.launch({ headless: true }); const page = await browser.newPage(); await page.goto('https://example.com/login'); // If there's a captcha, it will be solved automatically const { solved } = await page.solveRecaptchas(); console.log(`Solved ${solved.length} captchas`); // Regular Puppeteer API works as normal await page.type('#username', 'user@example.com'); await page.type('#password', 'password'); await page.click('#login-button'); await page.waitForNavigation(); console.log('Logged in:', page.url()); await browser.close(); })(); ```

curl-cffi can be accessed as low-level curl client as well as an easy high-level HTTP client: ```python from curl_cffi import requests response = requests.get('https://httpbin.org/json') print(response.json()) # or using sessions session = requests.Session() response = session.get('https://httpbin.org/json') # also supports async requests using asyncio import asyncio from curl_cffi.requests import AsyncSession urls = [ "http://httpbin.org/html", "http://httpbin.org/html", "http://httpbin.org/html", ] async with AsyncSession() as s: tasks = [] for url in urls: task = s.get(url) tasks.append(task) # scrape concurrently: responses = await asyncio.gather(*tasks) # also supports websocket connections from curl_cffi.requests import Session, WebSocket def on_message(ws: WebSocket, message): print(message) with Session() as s: ws = s.ws_connect( "wss://api.gemini.com/v1/marketdata/BTCUSD", on_message=on_message, ) ws.run_forever() ```

Alternatives / Similar

---

puppeteer-stealth

94,086 compare

curl-impersonate

5,944 compare

puppeteer

94,086 compare

primp new

504 compare

undetected-chromedriver

12,519 compare

camoufox new

7,089 compare

curl-cffi

1,751 compare

nodriver new

4,003 compare

botasaurus new

4,321 compare

hrequests

1,001 compare

selenium-driverless

718 compare

ayakashi

217 compare

pydoll new

- compare

curl-impersonate

5,944 compare

primp new

504 compare

hrequests

1,001 compare

requests

53,883 compare

node-fetch

8,860 compare

httpx

15,183 compare

aiohttp

16,395 compare

axios

108,987 compare

got

14,897 compare

needle

1,631 compare

superagent

16,610 compare

faraday

5,927 compare

pycurl

1,147 compare

puppeteer-stealth

94,086 compare

excon

1,172 compare

httpclient

707 compare

httparty

5,889 compare

puppeteer-extra new

7,293 compare

typhoeus

4,131 compare

httr

985 compare

undetected-chromedriver

12,519 compare

camoufox new

7,089 compare

guzzle

23,447 compare

rvest

1,517 compare

nodriver new

4,003 compare

wreck

378 compare

em-http-request

1,219 compare

symfony-http

2,033 compare

mechanize new

4,440 compare

treq

605 compare

http-2

908 compare

req

4,781 compare

resty

11,632 compare

nestful

505 compare

botasaurus new

4,321 compare

crul

107 compare

requests

3,577 compare

buzz

1,924 compare

selenium-driverless

718 compare

httpful

1,803 compare

ralger

165 compare

http.rb

3,104 compare

pydoll new

- compare