Skip to content

hrequestsvscurl-impersonate

MIT 35 1 780
10.7 thousand (month) Feb 23 2022 0.9.2(8 months ago)
4,221 1 68 MIT
Feb 23 2022 v0.6.1(1 year, 5 months ago)

hrequests is a feature rich modern replacement for a famous requests library for Python. It provides a feature rich HTTP client capable of resisting popular scraper identification techniques: - Seamless transition between headless browser and http client based requests - Integrated HTML parser - Mimicking of real browser TLS fingerprints - Javascript rendering - HTTP2 support - Realistic browser headers

Curl-impersonate is a special build of libcurl and cURL HTTP client that impersonates the four major browsers: - Google Chrome - Microsoft Edge - Safari - Firefox Curl-impersonate achieves this by patching TLS and HTTP fingerprints to be identical to that of one of these real browsers.

Unlike other HTTP clients curl-impersonate can bypass TSL and HTTP fingerprinting and detection techniques though it does not implement anything for Javascript fingerprint or bypass.

Highlights


bypasshttp2tls-fingerprinthttp-fingerprintsyncasync
bypasshttp2tls-fingerprinthttp-fingerprintlow-level

Example Use


hrequests has almost identical API and UX as requests and here's a quick overview:
import hrequests

# perform HTTP client requests
resp = hrequests.get('https://httpbin.org/html')
print(resp.status_code)
# 200

# use headless browsers and sessions:
session = hrequests.Session('chrome', version=122, os="mac")

# supports asyncio and easy concurrency
requests = [
    hrequests.async_get('https://www.google.com/', browser='firefox'),
    hrequests.async_get('https://www.duckduckgo.com/'),
    hrequests.async_get('https://www.yahoo.com/'),
    hrequests.async_get('https://www.httpbin.org/'),
]
responses = hrequests.map(requests, size=3)  # max 3 conccurency
curl-impersonate installs itself under `curl_` terminal commands like `curl_chrome116`:
$ curl_chrome116 https://www.wikipedia.org
To use it in HTTP client libraries that use `libcurl` replace curl path with one of these. To use it in python directly see curl-cffi Python package

Alternatives / Similar


Was this page helpful?