Skip to content

gotvswreck

MIT 116 10 14,048
86.5 million (month) Mar 27 2014 14.4.1(30 days ago)
383 7 3 BSD-3-Clause
Aug 06 2011 226.6 thousand (month) 18.1.0(5 months ago)

Got is a lightweight and powerful HTTP client for Node.js. It is built on top of the http and https modules and provides a simple, consistent API for making HTTP requests.

Got is one of the most feature-rich http clients in NodeJS ecosystem offering http2, proxy and asynchronous support making it ideal for web scraping.

Got also supports many specific domain integrations like AWS, plugins for various public APIs like github.

Note that Got has some inconsistent behaviors when it comes to web scraping use.
For example, it normalizes http headers which is undesired functionality in scraping and should be disabled.

Wreck is an HTTP client library for Node.js. It provides a simple, consistent API for making HTTP requests, including support for both the client and server side of an HTTP transaction.

Wreck is a very minimal but stable as it's part of Hapi web framework project. For web scraping, it doesn't offer required features like proxy configuration or http2 support so it's not recommended.

Highlights


http2asyncpopularextendibletypescriptproxy

Example Use


const got = require('got');

// GET requests are default and can be made calling the module as is:
const response = await got('https://api.example.com');
console.log(response.body);

// POST requests can send 
const response = await got.post('https://api.example.com', {
    json: { name: 'John Doe' },
});
console.log(response.body);

// handling cookies
import {CookieJar} from 'tough-cookie';

const cookieJar = new CookieJar();

await cookieJar.setCookie('foo=bar', 'https://httpbin.org');
await got('https://httpbin.org/anything', {cookieJar});

// using proxy
import got from 'got';
import {HttpsProxyAgent} from 'hpagent';

await got('https://httpbin.org/ip', {
  agent: {
    https: new HttpsProxyAgent({
      keepAlive: true,
      keepAliveMsecs: 1000,
      maxSockets: 256,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      proxy: 'https://localhost:8080'
    })
  }
});
const Wreck = require('wreck');

// get request
Wreck.get('http://example.com', (err, res, payload) => {
    if (err) {
        throw err;
    }
    console.log(payload.toString());
});

// post request
const options = {
    headers: { 'content-type': 'application/json' },
    payload: JSON.stringify({ name: 'John Doe' })
};

Wreck.post('http://example.com', options, (err, res, payload) => {
    if (err) {
        throw err;
    }
    console.log(payload.toString());
});

Alternatives / Similar


Was this page helpful?