Skip to content

node-crawler

6,623 5 34 MIT
1.5.0 (11 May 2024) Sep 10 2012 36.3 thousand (month)

node-crawler is a popular web scraping library for Node.js that allows you to easily navigate and extract data from websites. It has a simple API and supports concurrency, making it efficient for scraping large numbers of pages.

Features:

  • Server-side DOM & automatic jQuery insertion with Cheerio (default) or JSDOM,
  • Configurable pool size and retries,
  • Control rate limit,
  • Priority queue of requests,
  • forceUTF8 mode to let crawler deal for you with charset detection and conversion,
  • Compatible with 4.x or newer version.
  • Http2 support
  • Proxy support

Example Use


const Crawler = require('crawler');

const c = new Crawler({
    maxConnections: 10,
    // This will be called for each crawled page
    callback: (error, res, done) => {
        if (error) {
            console.log(error);
        } else {
            const $ = res.$;
            // $ is Cheerio by default
            //a lean implementation of core jQuery designed specifically for the server
            console.log($('title').text());
        }
        done();
    }
});

// Queue just one URL, with default callback
c.queue('http://www.amazon.com');

// Queue a list of URLs
c.queue(['http://www.google.com/','http://www.yahoo.com']);

// Queue URLs with custom callbacks & parameters
c.queue([{
    uri: 'http://parishackers.org/',
    jQuery: false,

    // The global callback won't be called
    callback: (error, res, done) => {
        if (error) {
            console.log(error);
        } else {
            console.log('Grabbed', res.body.length, 'bytes');
        }
        done();
    }
}]);

// Queue some HTML code directly without grabbing (mostly for tests)
c.queue([{
    html: '<p>This is a <strong>test</strong></p>'
}]);

Alternatives / Similar


197 1.0.0-beta8.4 (10 months ago) Apr 18 2019 compare

Other Languages

22,276 v2.1.0 (3 years ago) May 14 2018 compare
7,533 v1.3.4 (4 years ago) Feb 15 2020 compare
2,488 2024-05-09 (3 days ago) Jun 06 2019 compare
645 2024-05-11 (16 hours ago) Feb 09 2017 compare
51,043 2.11.1 (2 months ago) Jul 26 2019 compare
1,476 1.0.4 (1 year, 8 months ago) Nov 22 2014 compare
2,857 1.4.3 (7 months ago) Sep 04 2013 compare
5,630 v0.18.0 (1 year, 1 month ago) Aug 06 2019 compare
2,026 (2 years ago) Nov 20 2016 compare
2,891 v2.1.1 (5 months ago) Jul 17 2018 compare
5,964 1.1.14 (1 year, 9 months ago) Jul 26 2019 compare
3,012 1.5.0 (2 months ago) Sep 30 2018 compare
1,304 3.0.0 (1 year, 8 months ago) Dec 27 2011 compare
792 0.7.1 (3 months ago) Jul 25 2009 compare
152 2.2.4 (3 years ago) Dec 22 2019 compare
3,219 0.9.13 (9 months ago) Jul 04 2017 compare
235 1.33.0 (2 months ago) Feb 05 2023 compare
1,733 0.8.5 (1 year, 8 months ago) Oct 17 2018 compare
10,534 1.1.9 (5 years ago) Aug 24 2018 compare
1,321 v3.2.0 (a month ago) Dec 27 2021 compare
493 3.0.0 (a month ago) May 04 2020 compare
1,324 v0.7.2 (5 months ago) Mar 16 2013 compare
412 0.1.3 (9 months ago) Feb 20 2022 compare
298 v1.7.2 (a month ago) Apr 18 2022 compare