axiosvsneedle
axios is a popular JavaScript library that allows you to make HTTP requests from a Node.js environment. It is a promise-based library that works in both the browser and Node.js. It is similar to the Fetch API, but with a more powerful feature set and better browser compatibility.
One of the main benefits of using axios is that it automatically transforms the response data into a JSON object, making it easy to work with.
Axios is known for user-friendly API and support for asynchronous async/await syntax making it very accessible in web scraping.
needle is an HTTP client library for Node.js that provides a simple, flexible, and powerful API for making HTTP requests. It supports all major HTTP methods and has a clean and easy-to-use interface for handling responses and errors.
Example Use
// axios can be used with promises:
axios.get('http://httpbin.org/json')
.then(response => {
console.log(response.data);
})
.catch(error => {
console.log(error);
});
// or async await syntax:
var resp = await axios.get('http://httpbin.org/json');
console.log(resp.data);
// to make requests concurrently Promise.all function can be used:
const results = await Promise.all([
axios.get('http://httpbin.org/html'),
axios.get('http://httpbin.org/html'),
axios.get('http://httpbin.org/html'),
])
// axios also supports other type of requests like POST and even automatically serialize them:
await axios.post('http://httpbin.org/post', {'query': 'hello world'});
// or formdata
const data = {name: 'John Doe', email: 'johndoe@example.com'};
await axios.post('https://jsonplaceholder.typicode.com/users',
querystring.stringify(data),
{
headers: {
'Content-Type': 'application/x-www-form-urlencoded'
}
}
);
// default values like headers can be configured globally
axios.defaults.headers.common['User-Agent'] = 'webscraping.fyi';
// or for session instance:
const instance = axios.create({
headers: {"User-Agent": "webscraping.fyi"},
})
const needle = require('needle');
// needle supports both Promises and async/await
needle.get('https://httpbin.org/get', (err, res) => {
if (err) {
console.error(err);
return;
}
console.log(res.body);
});
const response = await needle.get('https://httpbin.org/get')
// concurrent requests can be sent using Promise.all
const results = await Promise.all([
needle.get('http://httpbin.org/html'),
needle.get('http://httpbin.org/html'),
needle.get('http://httpbin.org/html'),
])
// POST requests
const data = { name: 'John Doe' };
await needle.post('https://api.example.com', data)
// proxy
const options = {
proxy: 'http://proxy.example.com:8080'
};
await needle.get('https://httpbin.org/ip', options)
// headers and cookies
const options = {
headers: {
'Cookie': 'myCookie=123',
'X-My-Header': 'myValue'
}
};
await needle.get('https://httpbin.org/headers', options)