Skip to content

ralgervshttpclient

MIT 3 1 153
1.2 thousand (month) Dec 22 2019 2.2.4(3 years ago)
697 4 96 ruby
2.8.3(7 years ago) Jul 25 2009 3.6 million (month)

ralger is a small web scraping framework for R based on rvest and xml2.

It's goal to simplify basic web scraping and it provides a convenient and easy to use API.

It offers functions for retrieving pages, parsing HTML using CSS selectors, automatic table parsing and auto link, title, image and paragraph extraction.

HTTPClient is a Ruby gem that provides a simple and flexible interface for making HTTP requests. It's a full-featured HTTP client library with support for cookies, redirects, proxy, and more. It's built on top of the libwww-perl library, which is a widely-used, robust and well-documented library.

Features:

  • methods like GET/HEAD/POST/* via HTTP/1.1.
  • HTTPS(SSL), Cookies, proxy, authentication(Digest, NTLM, Basic), etc.
  • asynchronous HTTP request, streaming HTTP request.
  • debug mode CLI.
  • by contrast with net/http in standard distribution;
  • Cookies support
  • MT-safe
  • streaming POST (POST with File/IO)
  • Digest auth
  • Negotiate/NTLM auth for WWW-Authenticate (requires net/ntlm module; rubyntlm gem)
  • NTLM auth for Proxy-Authenticate (requires 'win32/sspi' module; rubysspi gem)
  • extensible with filter interface
  • you don't have to care HTTP/1.1 persistent connection (httpclient cares instead of you)

Example Use


library("ralger")

url <- "http://www.shanghairanking.com/rankings/arwu/2021"

# retrieve HTML and select elements using CSS selectors:
best_uni <- scrap(link = url, node = "a span", clean = TRUE)
head(best_uni, 5)
#>  [1] "Harvard University"
#>  [2] "Stanford University"
#>  [3] "University of Cambridge"
#>  [4] "Massachusetts Institute of Technology (MIT)"
#>  [5] "University of California, Berkeley"

# ralger can also parse HTML attributes
attributes <- attribute_scrap(
  link = "https://ropensci.org/",
  node = "a", # the a tag
  attr = "class" # getting the class attribute
)

head(attributes, 10) # NA values are a tags without a class attribute
#>  [1] "navbar-brand logo" "nav-link"          NA
#>  [4] NA                  NA                  "nav-link"
#>  [7] NA                  "nav-link"          NA
#> [10] NA
#

# ralger can automatically scrape tables:
data <- table_scrap(link ="https://www.boxofficemojo.com/chart/top_lifetime_gross/?area=XWW")

head(data)
#> # A tibble: 6 × 4
#>    Rank Title                                      `Lifetime Gross`  Year
#>   <int> <chr>                                      <chr>            <int>
#> 1     1 Avatar                                     $2,847,397,339    2009
#> 2     2 Avengers: Endgame                          $2,797,501,328    2019
#> 3     3 Titanic                                    $2,201,647,264    1997
#> 4     4 Star Wars: Episode VII - The Force Awakens $2,069,521,700    2015
#> 5     5 Avengers: Infinity War                     $2,048,359,754    2018
#> 6     6 Spider-Man: No Way Home                    $1,901,216,740    2021
require 'httpclient'

client = HTTPClient.new
# GET requests
response = client.get("http://httpbin.org/get")
puts response.content

# POST requests
data = { name: "value" }
response = client.post("http://httpbin.org/post", data)
puts response.content

Alternatives / Similar