Skip to content

node-fetchvscurl-cffi

MIT 206 10 8,753
228.5 million (month) Dec 28 2012 3.3.2(10 months ago)
1,751 2 34 MIT
Feb 23 2022 594.9 thousand (month) 0.7.1(3 months ago)

node-fetch is a lightweight library that provides a fetch()-like API for making HTTP requests in Node.js. It is a light-weight implementation of the Fetch API, which is mostly compatible with the browser's version.

node-fetch is primarily known as almost identical package fetch() is included in web browsers so it shares the same use common API. It's great starting point for people coming from front-end environment.

Curl-cffi is a Python library for implementing curl-impersonate which is a HTTP client that appears as one of popular web browsers like: - Google Chrome - Microsoft Edge - Safari - Firefox Unlike requests and httpx which are native Python libraries, curl-cffi uses cURL and inherits it's powerful features like extensive HTTP protocol support and detection patches for TLS and HTTP fingerprinting.

Using curl-cffi web scrapers can bypass TLS and HTTP fingerprinting.

Highlights


popular
bypasshttp2tls-fingerprinthttp-fingerprintsyncasync

Example Use


const fetch = require('node-fetch');

// fetch supports both Promises and async/await
fetch('http://httpbin.org/get')
    .then(res => res.text())
    .then(body => console.log(body))
    .catch(err => console.error(err));
const response = await fetch('http://httpbin.org/get');

// for concurrent scraping Promise.all can be used
const results = await Promise.all([
  fetch('http://httpbin.org/html'),
  fetch('http://httpbin.org/html'),
  fetch('http://httpbin.org/html'),
])

// POST requests
await fetch('http://httpbin.org/post', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ name: 'John Doe' }),
})

// Proxy use:
const agent = new https.Agent({
    rejectUnauthorized: false,
    proxy: {
        host: 'proxy.example.com',
        port: 8080
    }
});

await fetch('https://httpbin.org/ip', { agent })

// setting headers and cookies
const headers = new fetch.Headers();
headers.append('Cookie', 'myCookie=123');
headers.append('X-My-Header', 'myValue');

await fetch('https://httpbin.org/headers', { headers })
curl-cffi can be accessed as low-level curl client as well as an easy high-level HTTP client:
from curl_cffi import requests

response = requests.get('https://httpbin.org/json')
print(response.json())

# or using sessions
session = requests.Session()
response = session.get('https://httpbin.org/json')

# also supports async requests using asyncio
import asyncio
from curl_cffi.requests import AsyncSession

urls = [
  "http://httpbin.org/html",
  "http://httpbin.org/html",
  "http://httpbin.org/html",
]

async with AsyncSession() as s:
    tasks = []
    for url in urls:
        task = s.get(url)
        tasks.append(task)
    # scrape concurrently:
    responses = await asyncio.gather(*tasks)

# also supports websocket connections
from curl_cffi.requests import Session, WebSocket

def on_message(ws: WebSocket, message):
    print(message)

with Session() as s:
    ws = s.ws_connect(
        "wss://api.gemini.com/v1/marketdata/BTCUSD",
        on_message=on_message,
    )
    ws.run_forever()

Alternatives / Similar


Was this page helpful?