puppeteer-extravspydoll

MIT 268 2 7,293

2.0 million (month) May 15 2018 3.3.6(2023-03-01 12:19:02 ago)

- - - None

Jun 01 2024 0.0.0(2025-02-01 00:00:00 ago)

Puppeteer-extra is a modular plugin framework that wraps Puppeteer (and Playwright) to add extra functionality through a plugin system. It acts as a drop-in replacement for Puppeteer while enabling powerful extensions for stealth, captcha solving, ad blocking, and more.

The most popular plugins include:

puppeteer-extra-plugin-stealth Applies various evasion techniques to make the automated browser harder to detect. Patches common detection vectors like navigator.webdriver, Chrome.runtime, WebGL renderer strings, and more. This is the most widely used Puppeteer stealth solution.
puppeteer-extra-plugin-recaptcha Automatically detects and solves reCAPTCHA and hCaptcha challenges using third-party solving services (2captcha, anti-captcha).
puppeteer-extra-plugin-adblocker Blocks ads and trackers to speed up page loading and reduce bandwidth usage during scraping.
puppeteer-extra-plugin-anonymize-ua Randomizes the User-Agent string to avoid fingerprinting.

Key features of the framework:

Drop-in replacement Use puppeteer-extra instead of puppeteer in your imports - existing code works without changes.
Plugin composition Multiple plugins can be stacked and they work together without conflicts.
Playwright support The same plugin system works with Playwright via playwright-extra.
Community plugins Active community creating and maintaining plugins for various use cases.

Puppeteer-extra is the go-to solution for adding stealth capabilities to Puppeteer-based scrapers without rewriting existing code.

Pydoll is a Python library for browser automation that uses the Chrome DevTools Protocol (CDP) directly, designed to be undetectable by anti-bot systems. Unlike Selenium-based tools, Pydoll does not use WebDriver and avoids the common detection vectors that anti-bot systems look for.

Key features include:

Native CDP communication Connects directly to Chrome/Chromium via CDP websocket without intermediary drivers, avoiding the automation flags and fingerprints that WebDriver-based tools leave behind.
Event-driven architecture Built around an async event system that can listen for and react to browser events like network requests, console messages, and DOM changes.
Network interception Can intercept, modify, and mock network requests and responses, useful for blocking unnecessary resources or modifying API responses during scraping.
Async-first design Fully asynchronous API built on Python's asyncio for efficient concurrent automation.
Clean API Provides a high-level, Pythonic API for common browser automation tasks while still allowing direct CDP command execution for advanced use cases.
Multi-browser support Can manage multiple browser instances and pages concurrently.

Pydoll fills a similar niche to nodriver and camoufox — browser automation with a focus on avoiding detection — but takes a different approach by providing more granular control over CDP communication and network interception.

Highlights

plugin-systemextendiblecommunity-toolsstealth

anti-detectcdpasync

Example Use

```javascript const puppeteer = require('puppeteer-extra'); // Add stealth plugin to avoid bot detection const StealthPlugin = require('puppeteer-extra-plugin-stealth'); puppeteer.use(StealthPlugin()); // Add recaptcha solving plugin const RecaptchaPlugin = require('puppeteer-extra-plugin-recaptcha'); puppeteer.use(RecaptchaPlugin({ provider: { id: '2captcha', token: 'YOUR_API_KEY' }, })); (async () => { // Launch browser - stealth is applied automatically const browser = await puppeteer.launch({ headless: true }); const page = await browser.newPage(); await page.goto('https://example.com/login'); // If there's a captcha, it will be solved automatically const { solved } = await page.solveRecaptchas(); console.log(`Solved ${solved.length} captchas`); // Regular Puppeteer API works as normal await page.type('#username', 'user@example.com'); await page.type('#password', 'password'); await page.click('#login-button'); await page.waitForNavigation(); console.log('Logged in:', page.url()); await browser.close(); })(); ```

```python import asyncio from pydoll.browser import Chrome from pydoll.constants import By async def main(): async with Chrome() as browser: # Open a new page page = await browser.new_page() await page.go_to("https://example.com") # Find and interact with elements search_input = await page.find_element(By.CSS, "input[name='q']") await search_input.type_text("web scraping") submit_btn = await page.find_element(By.CSS, "button[type='submit']") await submit_btn.click() # Wait for results and extract content await page.wait_element(By.CSS, ".results") results = await page.find_elements(By.CSS, ".result-item") for result in results: title = await result.get_text() print(title) # Network interception example await page.enable_network_interception() # intercept and analyze API calls made by the page asyncio.run(main()) ```