curl-cffivsexcon
Curl-cffi is a Python library for implementing curl-impersonate which is a
HTTP client that appears as one of popular web browsers like:
- Google Chrome
- Microsoft Edge
- Safari
- Firefox
Unlike requests and httpx which are native Python libraries, curl-cffi uses cURL and inherits it's powerful features
like extensive HTTP protocol support and detection patches for TLS and HTTP fingerprinting.
Using curl-cffi web scrapers can bypass TLS and HTTP fingerprinting.
Excon is a Ruby library for making HTTP requests. It is designed to be fast and efficient, and is often used as a building block for other Ruby libraries and frameworks.
One of the main features of Excon is its support for persistent connections, which allows it to reuse the same connection for multiple requests, reducing the overhead of establishing a new connection for each request.
Excon also supports streaming requests and responses, which allows you to read or write data to the server incrementally, without having to load the entire response into memory at once.
Highlights
Example Use
from curl_cffi import requests
response = requests.get('https://httpbin.org/json')
print(response.json())
# or using sessions
session = requests.Session()
response = session.get('https://httpbin.org/json')
# also supports async requests using asyncio
import asyncio
from curl_cffi.requests import AsyncSession
urls = [
"http://httpbin.org/html",
"http://httpbin.org/html",
"http://httpbin.org/html",
]
async with AsyncSession() as s:
tasks = []
for url in urls:
task = s.get(url)
tasks.append(task)
# scrape concurrently:
responses = await asyncio.gather(*tasks)
# also supports websocket connections
from curl_cffi.requests import Session, WebSocket
def on_message(ws: WebSocket, message):
print(message)
with Session() as s:
ws = s.ws_connect(
"wss://api.gemini.com/v1/marketdata/BTCUSD",
on_message=on_message,
)
ws.run_forever()
require 'excon'
# GET requests
response = Excon.get('https://jsonplaceholder.typicode.com/posts/1')
puts response.body
puts response.status
puts response.headers
# POST requests
response = Excon.post('https://jsonplaceholder.typicode.com/posts',
:body => { :title => 'foo', :body => 'bar', :userId => 1 }.to_json,
:headers => { 'Content-Type' => 'application/json' } )
puts response.body