Skip to content

dataflowkit

645 3 4 BSD-3-Clause
2024-05-11 (11 May 2024) Feb 09 2017

Dataflow kit ("DFK") is a Web Scraping framework for Gophers. It extracts data from web pages, following the specified CSS Selectors. You can use it in many ways for data mining, data processing or archiving.

Web-scraping pipeline consists of 3 general components:

  • Downloading an HTML web-page. (Fetch Service)
  • Parsing an HTML page and retrieving data we're interested in (Parse Service)
  • Encoding parsed data to CSV, MS Excel, JSON, JSON Lines or XML format.

For fetching dataflowkit has several types of page fetchers:

  • Base fetcher uses standard golang http client to fetch pages as is. It works faster than Chrome fetcher. But Base fetcher cannot render dynamic javascript driven web pages.
  • Chrome fetcher is intended for rendering dynamic javascript based content. It sends requests to Chrome running in headless mode.

For parsing dataflowkit extracts data from downloaded web page following the rules listed in configuration JSON file. Extracted data is returned in CSV, MS Excel, JSON or XML format.

Some dataflowkit features:

  • Scraping of JavaScript generated pages;
  • Data extraction from paginated websites;
  • Processing infinite scrolled pages.
  • S—Āraping of websites behind login form;
  • Cookies and sessions handling;
  • Following links and detailed pages processing;
  • Managing delays between requests per domain;
  • Following robots.txt directives;
  • Saving intermediate data in Diskv or Mongodb. Storage interface is flexible enough to add more storage types easily;
  • Encode results to CSV, MS Excel, JSON(Lines), XML formats;
  • Dataflow kit is fast. It takes about 4-6 seconds to fetch and then parse 50 pages.
  • Dataflow kit is suitable to process quite large volumes of data. Our tests show the time needed to parse appr. 4 millions of pages is about 7 hours.

Example Use


Dataflowkit uses JSON configuration like:

{
  "name": "collection",
  "request": {
      "url": "https://example.com"
  },
  "fields": [
      {
          "name": "Title",
          "selector": ".product-container a",
          "extractor": {
              "types": [
                  "text",
                  "href"
              ],
              "filters": [
                  "trim",
                  "lowerCase"
              ],
              "params": {
                  "includeIfEmpty": false
              }
          }
      },
      {
          "name": "Image",
          "selector": "#product-container img",
          "extractor": {
              "types": [
                  "alt",
                  "src",
                  "width",
                  "height"
              ],
              "filters": [
                  "trim",
                  "upperCase"
              ]
          }
      },
      {
          "name": "Buyinfo",
          "selector": ".buy-info",
          "extractor": {
              "types": [
                  "text"
              ],
              "params": {
                  "includeIfEmpty": false
              }
          }
      }
  ],
  "paginator": {
      "selector": ".next",
      "attr": "href",
      "maxPages": 3
  },
  "format": "json",
  "fetcherType": "chrome",
  "paginateResults": false
}
which is then ingested through CLI command.

Alternatives / Similar


22,276 v2.1.0 (3 years ago) May 14 2018 compare
7,533 v1.3.4 (4 years ago) Feb 15 2020 compare
2,488 2024-05-09 (3 days ago) Jun 06 2019 compare
5,630 v0.18.0 (1 year, 1 month ago) Aug 06 2019 compare
2,026 (2 years ago) Nov 20 2016 compare

Other Languages

51,043 2.11.1 (2 months ago) Jul 26 2019 compare
1,476 1.0.4 (1 year, 8 months ago) Nov 22 2014 compare
2,857 1.4.3 (7 months ago) Sep 04 2013 compare
6,623 1.5.0 (23 hours ago) Sep 10 2012 compare
2,891 v2.1.1 (5 months ago) Jul 17 2018 compare
5,964 1.1.14 (1 year, 9 months ago) Jul 26 2019 compare
3,012 1.5.0 (2 months ago) Sep 30 2018 compare
1,304 3.0.0 (1 year, 8 months ago) Dec 27 2011 compare
792 0.7.1 (3 months ago) Jul 25 2009 compare
152 2.2.4 (3 years ago) Dec 22 2019 compare
3,219 0.9.13 (9 months ago) Jul 04 2017 compare
235 1.33.0 (2 months ago) Feb 05 2023 compare
1,733 0.8.5 (1 year, 8 months ago) Oct 17 2018 compare
10,534 1.1.9 (5 years ago) Aug 24 2018 compare
1,321 v3.2.0 (a month ago) Dec 27 2021 compare
493 3.0.0 (a month ago) May 04 2020 compare
197 1.0.0-beta8.4 (10 months ago) Apr 18 2019 compare
1,324 v0.7.2 (5 months ago) Mar 16 2013 compare
412 0.1.3 (9 months ago) Feb 20 2022 compare
298 v1.7.2 (a month ago) Apr 18 2022 compare