Skip to content

hrequestsvshttpclient

Apache-2.0 13 1 541
2.5 thousand (month) Feb 23 2022 0.8.2(3 months ago)
699 4 94 ruby
Jul 25 2009 3.8 million (month) 2.8.3(7 years ago)

hrequests is a feature rich modern replacement for a famous requests library for Python. It provides a feature rich HTTP client capable of resisting popular scraper identification techniques: - Seamless transition between headless browser and http client based requests - Integrated HTML parser - Mimicking of real browser TLS fingerprints - Javascript rendering - HTTP2 support - Realistic browser headers

HTTPClient is a Ruby gem that provides a simple and flexible interface for making HTTP requests. It's a full-featured HTTP client library with support for cookies, redirects, proxy, and more. It's built on top of the libwww-perl library, which is a widely-used, robust and well-documented library.

Features:

  • methods like GET/HEAD/POST/* via HTTP/1.1.
  • HTTPS(SSL), Cookies, proxy, authentication(Digest, NTLM, Basic), etc.
  • asynchronous HTTP request, streaming HTTP request.
  • debug mode CLI.
  • by contrast with net/http in standard distribution;
  • Cookies support
  • MT-safe
  • streaming POST (POST with File/IO)
  • Digest auth
  • Negotiate/NTLM auth for WWW-Authenticate (requires net/ntlm module; rubyntlm gem)
  • NTLM auth for Proxy-Authenticate (requires 'win32/sspi' module; rubysspi gem)
  • extensible with filter interface
  • you don't have to care HTTP/1.1 persistent connection (httpclient cares instead of you)

Highlights


bypasshttp2tls-fingerprinthttp-fingerprintsyncasync

Example Use


hrequests has almost identical API and UX as requests and here's a quick overview:
import hrequests

# perform HTTP client requests
resp = hrequests.get('https://httpbin.org/html')
print(resp.status_code)
# 200

# use headless browsers and sessions:
session = hrequests.Session('chrome', version=122, os="mac")

# supports asyncio and easy concurrency
requests = [
    hrequests.async_get('https://www.google.com/', browser='firefox'),
    hrequests.async_get('https://www.duckduckgo.com/'),
    hrequests.async_get('https://www.yahoo.com/'),
    hrequests.async_get('https://www.httpbin.org/'),
]
responses = hrequests.map(requests, size=3)  # max 3 conccurency
require 'httpclient'

client = HTTPClient.new
# GET requests
response = client.get("http://httpbin.org/get")
puts response.content

# POST requests
data = { name: "value" }
response = client.post("http://httpbin.org/post", data)
puts response.content

Alternatives / Similar


Was this page helpful?