puppeteer-extravshrequests

MIT 268 2 7,293

2.0 million (month) May 15 2018 3.3.6(2023-03-01 12:19:02 ago)

1,001 1 51 MIT

Feb 23 2022 33.3 thousand (month) 0.9.2(2024-12-01 02:55:27 ago)

Puppeteer-extra is a modular plugin framework that wraps Puppeteer (and Playwright) to add extra functionality through a plugin system. It acts as a drop-in replacement for Puppeteer while enabling powerful extensions for stealth, captcha solving, ad blocking, and more.

The most popular plugins include:

puppeteer-extra-plugin-stealth Applies various evasion techniques to make the automated browser harder to detect. Patches common detection vectors like navigator.webdriver, Chrome.runtime, WebGL renderer strings, and more. This is the most widely used Puppeteer stealth solution.
puppeteer-extra-plugin-recaptcha Automatically detects and solves reCAPTCHA and hCaptcha challenges using third-party solving services (2captcha, anti-captcha).
puppeteer-extra-plugin-adblocker Blocks ads and trackers to speed up page loading and reduce bandwidth usage during scraping.
puppeteer-extra-plugin-anonymize-ua Randomizes the User-Agent string to avoid fingerprinting.

Key features of the framework:

Drop-in replacement Use puppeteer-extra instead of puppeteer in your imports - existing code works without changes.
Plugin composition Multiple plugins can be stacked and they work together without conflicts.
Playwright support The same plugin system works with Playwright via playwright-extra.
Community plugins Active community creating and maintaining plugins for various use cases.

Puppeteer-extra is the go-to solution for adding stealth capabilities to Puppeteer-based scrapers without rewriting existing code.

hrequests is a feature rich modern replacement for a famous requests library for Python. It provides a feature rich HTTP client capable of resisting popular scraper identification techniques: - Seamless transition between headless browser and http client based requests - Integrated HTML parser - Mimicking of real browser TLS fingerprints - Javascript rendering - HTTP2 support - Realistic browser headers

Highlights

plugin-systemextendiblecommunity-toolsstealth

bypasshttp2tls-fingerprinthttp-fingerprintsyncasync

Example Use

```javascript const puppeteer = require('puppeteer-extra'); // Add stealth plugin to avoid bot detection const StealthPlugin = require('puppeteer-extra-plugin-stealth'); puppeteer.use(StealthPlugin()); // Add recaptcha solving plugin const RecaptchaPlugin = require('puppeteer-extra-plugin-recaptcha'); puppeteer.use(RecaptchaPlugin({ provider: { id: '2captcha', token: 'YOUR_API_KEY' }, })); (async () => { // Launch browser - stealth is applied automatically const browser = await puppeteer.launch({ headless: true }); const page = await browser.newPage(); await page.goto('https://example.com/login'); // If there's a captcha, it will be solved automatically const { solved } = await page.solveRecaptchas(); console.log(`Solved ${solved.length} captchas`); // Regular Puppeteer API works as normal await page.type('#username', 'user@example.com'); await page.type('#password', 'password'); await page.click('#login-button'); await page.waitForNavigation(); console.log('Logged in:', page.url()); await browser.close(); })(); ```

hrequests has almost identical API and UX as requests and here's a quick overview: ```python import hrequests # perform HTTP client requests resp = hrequests.get('https://httpbin.org/html') print(resp.status_code) # 200 # use headless browsers and sessions: session = hrequests.Session('chrome', version=122, os="mac") # supports asyncio and easy concurrency requests = [ hrequests.async_get('https://www.google.com/', browser='firefox'), hrequests.async_get('https://www.duckduckgo.com/'), hrequests.async_get('https://www.yahoo.com/'), hrequests.async_get('https://www.httpbin.org/'), ] responses = hrequests.map(requests, size=3) # max 3 conccurency ```

Alternatives / Similar

puppeteer-stealth

94,086 compare

curl-impersonate

5,944 compare

puppeteer

94,086 compare

primp new

504 compare

undetected-chromedriver

12,519 compare

camoufox new

7,089 compare

curl-cffi

1,751 compare

nodriver new

4,003 compare

botasaurus new

4,321 compare

hrequests

1,001 compare

selenium-driverless

718 compare

ayakashi

217 compare

pydoll new

- compare

curl-impersonate

5,944 compare

primp new

504 compare

curl-cffi

1,751 compare

requests

53,883 compare

node-fetch

8,860 compare

httpx

15,183 compare

aiohttp

16,395 compare

axios

108,987 compare

got

14,897 compare

needle

1,631 compare

superagent

16,610 compare

faraday

5,927 compare

pycurl

1,147 compare

puppeteer-stealth

94,086 compare

excon

1,172 compare

httpclient

707 compare

httparty

5,889 compare

puppeteer-extra new

7,293 compare

typhoeus

4,131 compare

httr

985 compare

undetected-chromedriver

12,519 compare

camoufox new

7,089 compare

guzzle

23,447 compare

rvest

1,517 compare

nodriver new

4,003 compare

wreck

378 compare

em-http-request

1,219 compare

symfony-http

2,033 compare

mechanize new

4,440 compare

treq

605 compare

http-2

908 compare

req

4,781 compare

resty

11,632 compare

nestful

505 compare

botasaurus new

4,321 compare

crul

107 compare

requests

3,577 compare

buzz

1,924 compare

selenium-driverless

718 compare

httpful

1,803 compare

ralger

165 compare

http.rb

3,104 compare

pydoll new

- compare