Skip to content

gotvscurl-impersonate

MIT 116 10 14,048
86.5 million (month) Mar 27 2014 14.4.1(30 days ago)
3,452 1 60 MIT
Feb 23 2022 v0.6.1(4 months ago)

Got is a lightweight and powerful HTTP client for Node.js. It is built on top of the http and https modules and provides a simple, consistent API for making HTTP requests.

Got is one of the most feature-rich http clients in NodeJS ecosystem offering http2, proxy and asynchronous support making it ideal for web scraping.

Got also supports many specific domain integrations like AWS, plugins for various public APIs like github.

Note that Got has some inconsistent behaviors when it comes to web scraping use.
For example, it normalizes http headers which is undesired functionality in scraping and should be disabled.

Curl-impersonate is a special build of libcurl and cURL HTTP client that impersonates the four major browsers: - Google Chrome - Microsoft Edge - Safari - Firefox Curl-impersonate achieves this by patching TLS and HTTP fingerprints to be identical to that of one of these real browsers.

Unlike other HTTP clients curl-impersonate can bypass TSL and HTTP fingerprinting and detection techniques though it does not implement anything for Javascript fingerprint or bypass.

Highlights


http2asyncpopularextendibletypescriptproxy
bypasshttp2tls-fingerprinthttp-fingerprintlow-level

Example Use


const got = require('got');

// GET requests are default and can be made calling the module as is:
const response = await got('https://api.example.com');
console.log(response.body);

// POST requests can send 
const response = await got.post('https://api.example.com', {
    json: { name: 'John Doe' },
});
console.log(response.body);

// handling cookies
import {CookieJar} from 'tough-cookie';

const cookieJar = new CookieJar();

await cookieJar.setCookie('foo=bar', 'https://httpbin.org');
await got('https://httpbin.org/anything', {cookieJar});

// using proxy
import got from 'got';
import {HttpsProxyAgent} from 'hpagent';

await got('https://httpbin.org/ip', {
  agent: {
    https: new HttpsProxyAgent({
      keepAlive: true,
      keepAliveMsecs: 1000,
      maxSockets: 256,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      proxy: 'https://localhost:8080'
    })
  }
});
curl-impersonate installs itself under `curl_` terminal commands like `curl_chrome116`:
$ curl_chrome116 https://www.wikipedia.org
To use it in HTTP client libraries that use `libcurl` replace curl path with one of these. To use it in python directly see curl-cffi Python package

Alternatives / Similar


Was this page helpful?