Skip to content

ralgervsfaraday

MIT 3 1 153
1.2 thousand (month) Dec 22 2019 2.2.4(3 years ago)
5,646 7 37 MIT
2.9.0(2 months ago) Dec 19 2009 4.5 million (month)

ralger is a small web scraping framework for R based on rvest and xml2.

It's goal to simplify basic web scraping and it provides a convenient and easy to use API.

It offers functions for retrieving pages, parsing HTML using CSS selectors, automatic table parsing and auto link, title, image and paragraph extraction.

Faraday is a Ruby gem that provides a simple and flexible interface for making HTTP requests. It allows you to create a Faraday connection object, which you can use to send requests and receive responses.

Faraday abstracts away the details of the underlying HTTP client library, so you can use it with different libraries such as Net::HTTP, HTTPClient, typhoeus and others.

Since Faraday can adapt many other HTTP clients it's very popular choice in web scraping.

Example Use


library("ralger")

url <- "http://www.shanghairanking.com/rankings/arwu/2021"

# retrieve HTML and select elements using CSS selectors:
best_uni <- scrap(link = url, node = "a span", clean = TRUE)
head(best_uni, 5)
#>  [1] "Harvard University"
#>  [2] "Stanford University"
#>  [3] "University of Cambridge"
#>  [4] "Massachusetts Institute of Technology (MIT)"
#>  [5] "University of California, Berkeley"

# ralger can also parse HTML attributes
attributes <- attribute_scrap(
  link = "https://ropensci.org/",
  node = "a", # the a tag
  attr = "class" # getting the class attribute
)

head(attributes, 10) # NA values are a tags without a class attribute
#>  [1] "navbar-brand logo" "nav-link"          NA
#>  [4] NA                  NA                  "nav-link"
#>  [7] NA                  "nav-link"          NA
#> [10] NA
#

# ralger can automatically scrape tables:
data <- table_scrap(link ="https://www.boxofficemojo.com/chart/top_lifetime_gross/?area=XWW")

head(data)
#> # A tibble: 6 × 4
#>    Rank Title                                      `Lifetime Gross`  Year
#>   <int> <chr>                                      <chr>            <int>
#> 1     1 Avatar                                     $2,847,397,339    2009
#> 2     2 Avengers: Endgame                          $2,797,501,328    2019
#> 3     3 Titanic                                    $2,201,647,264    1997
#> 4     4 Star Wars: Episode VII - The Force Awakens $2,069,521,700    2015
#> 5     5 Avengers: Infinity War                     $2,048,359,754    2018
#> 6     6 Spider-Man: No Way Home                    $1,901,216,740    2021
# GET requests
response = Faraday.get('http://httpbingo.org')
put response.status
put response.headers
put response.body

# or use a persistent client session:
conn = Faraday.new(
  url: 'http://httpbin.org/get',
  params: {param: '1'},
  headers: {'Content-Type' => 'application/json'}
)

# POST requests
response = conn.post('/post') do |req|
  req.params['limit'] = 100
  req.body = {query: 'chunky bacon'}.to_json
end

Alternatives / Similar