node-fetchvsfaraday
node-fetch is a lightweight library that provides a fetch()-like API for making HTTP requests in Node.js. It is a light-weight implementation of the Fetch API, which is mostly compatible with the browser's version.
node-fetch is primarily known as almost identical package fetch() is included in web browsers so it shares the same use common API. It's great starting point for people coming from front-end environment.
Faraday is a Ruby gem that provides a simple and flexible interface for making HTTP requests. It allows you to create a Faraday connection object, which you can use to send requests and receive responses.
Faraday abstracts away the details of the underlying HTTP client library, so you can use it with different libraries such as Net::HTTP, HTTPClient, typhoeus and others.
Since Faraday can adapt many other HTTP clients it's very popular choice in web scraping.
Highlights
Example Use
const fetch = require('node-fetch');
// fetch supports both Promises and async/await
fetch('http://httpbin.org/get')
.then(res => res.text())
.then(body => console.log(body))
.catch(err => console.error(err));
const response = await fetch('http://httpbin.org/get');
// for concurrent scraping Promise.all can be used
const results = await Promise.all([
fetch('http://httpbin.org/html'),
fetch('http://httpbin.org/html'),
fetch('http://httpbin.org/html'),
])
// POST requests
await fetch('http://httpbin.org/post', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ name: 'John Doe' }),
})
// Proxy use:
const agent = new https.Agent({
rejectUnauthorized: false,
proxy: {
host: 'proxy.example.com',
port: 8080
}
});
await fetch('https://httpbin.org/ip', { agent })
// setting headers and cookies
const headers = new fetch.Headers();
headers.append('Cookie', 'myCookie=123');
headers.append('X-My-Header', 'myValue');
await fetch('https://httpbin.org/headers', { headers })
# GET requests
response = Faraday.get('http://httpbingo.org')
put response.status
put response.headers
put response.body
# or use a persistent client session:
conn = Faraday.new(
url: 'http://httpbin.org/get',
params: {param: '1'},
headers: {'Content-Type' => 'application/json'}
)
# POST requests
response = conn.post('/post') do |req|
req.params['limit'] = 100
req.body = {query: 'chunky bacon'}.to_json
end