puppeteer-stealthvscurl-impersonate
Puppeteer Stealth is puppeteer plugin that fortifies headles browser for web scraping. This makes detection of puppeteer scrapers more difficult allowing to scrape targets which use headless browser detection techniques.
Puppeteer-stealth does this by applying various javascript patches to cover up traces of headless browser presence in the web scraping browser's environment.
Curl-impersonate is a special build of libcurl and cURL HTTP client that impersonates the four major browsers:
- Google Chrome
- Microsoft Edge
- Safari
- Firefox
Curl-impersonate achieves this by patching TLS and HTTP fingerprints to be identical to that of one of these real browsers.
Unlike other HTTP clients curl-impersonate can bypass TSL and HTTP fingerprinting and detection techniques though it does not implement anything for Javascript fingerprint or bypass.
Highlights
Example Use
const puppeteer = require('puppeteer-extra')
// add stealth plugin and use defaults (all evasion techniques)
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())
// puppeteer usage as normal
puppeteer.launch({ headless: true }).then(async browser => {
console.log('Running tests..')
const page = await browser.newPage()
await page.goto('https://bot.sannysoft.com')
await page.waitForTimeout(5000)
await page.screenshot({ path: 'result.png', fullPage: true })
await browser.close()
console.log("success - check the result.png screenshot")
})
$ curl_chrome116 https://www.wikipedia.org