choppervsralger

python html-parserframeworkhttp-client

r html-parserframeworkhttp-client

MIT 1 3 23

1.7 thousand (month) Jul 24 2014 0.6.0(2023-04-26 10:16:25 ago)

165 1 3 MIT

Dec 22 2019 327 (month) 2.3.0(2021-03-18 00:10:00 ago)

---

Chopper is a tool to extract elements from HTML by preserving ancestors and CSS rules.

Compared to other HTML parsers Chopper is designed to retain original HTML tree but eliminate elements that do not match parsing rules. Meaning, we can parse HTML elements and keep thei structure for machine learning or other tasks where data structure is needed as well as the data value.

ralger is a small web scraping framework for R based on rvest and xml2.

It's goal to simplify basic web scraping and it provides a convenient and easy to use API.

It offers functions for retrieving pages, parsing HTML using CSS selectors, automatic table parsing and auto link, title, image and paragraph extraction.

Example Use

---

```python HTML = """

HELLO WORLD Do not want

<div id="footer"></div>

"""

CSS = """ div { border: 1px solid black; } div#main { color: blue; } div.iwantthis { background-color: red; } a { color: green; } div#footer { border-top: 2px solid red; } """

extractor = Extractor.keep('//div[@class="iwantthis"]').discard('//a') html, css = extractor.extract(HTML, CSS)

will result in:

html """

HELLO WORLD

"""

css """ div{border:1px solid black;} div#main{color:blue;} div.iwantthis{background-color:red;} """ ```

```r library("ralger")

url <- "http://www.shanghairanking.com/rankings/arwu/2021"

retrieve HTML and select elements using CSS selectors:

best_uni <- scrap(link = url, node = "a span", clean = TRUE) head(best_uni, 5)

> [1] "Harvard University"

> [2] "Stanford University"

> [3] "University of Cambridge"

> [4] "Massachusetts Institute of Technology (MIT)"

> [5] "University of California, Berkeley"

ralger can also parse HTML attributes

attributes <- attribute_scrap( link = "https://ropensci.org/", node = "a", # the a tag attr = "class" # getting the class attribute )

head(attributes, 10) # NA values are a tags without a class attribute

> [1] "navbar-brand logo" "nav-link" NA

> [4] NA NA "nav-link"

> [7] NA "nav-link" NA

> [10] NA

ralger can automatically scrape tables:

data <- table_scrap(link ="https://www.boxofficemojo.com/chart/top_lifetime_gross/?area=XWW")

head(data)

> # A tibble: 6 × 4

> Rank Title Lifetime Gross Year

>

> 1 1 Avatar $2,847,397,339 2009

> 2 2 Avengers: Endgame $2,797,501,328 2019

> 3 3 Titanic $2,201,647,264 1997

> 4 4 Star Wars: Episode VII - The Force Awakens $2,069,521,700 2015

> 5 5 Avengers: Infinity War $2,048,359,754 2018

> 6 6 Spider-Man: No Way Home $1,901,216,740 2021

```

Alternatives / Similar

---

parse5

3,886 compare

sax-js

1,153 compare

htmlparser2

4,789 compare

lxml

3,010 compare

beautifulsoup

- compare

jsdom new

21,552 compare

xmltodict

5,734 compare

cheerio

30,265 compare

html5lib

1,220 compare

cssselect

309 compare

feedparser

2,351 compare

nokogiri

6,248 compare

parsel

1,324 compare

selectolax

1,607 compare

pyquery

2,381 compare

xml2

223 compare

requests-html

13,863 compare

rvest

1,517 compare

untangle

632 compare

scrapling new

36,206 compare

html5-php

1,772 compare

domcrawler

4,038 compare

goquery

14,926 compare

cascadia

754 compare

htmlquery

781 compare

xpath

739 compare

soup

2,227 compare

chompjs

218 compare

html5-parser

700 compare

gazpacho

768 compare

embed

2,103 compare

simple-html-dom new

- compare

ralger

165 compare

rvest

1,517 compare

scrapling new

36,206 compare

mechanize new

4,440 compare

requests

53,883 compare

node-fetch

8,860 compare

httpx

15,183 compare

aiohttp

16,395 compare

axios

108,987 compare

parse5

3,886 compare

sax-js

1,153 compare

htmlparser2

4,789 compare

lxml

3,010 compare

beautifulsoup

- compare

jsdom new

21,552 compare

colly

25,231 compare

katana new

16,499 compare

got

14,897 compare

xmltodict

5,734 compare

cheerio

30,265 compare

pholcus

7,594 compare

curl-impersonate

5,944 compare

needle

1,631 compare

superagent

16,610 compare

html5lib

1,220 compare

geziyor

2,772 compare

cssselect

309 compare

feedparser

2,351 compare

dataflowkit

711 compare

primp new

504 compare

faraday

5,927 compare

nokogiri

6,248 compare

pycurl

1,147 compare

parsel

1,324 compare

selectolax

1,607 compare

excon

1,172 compare

scrapy

61,276 compare

httpclient

707 compare

httparty

5,889 compare

pyquery

2,381 compare

crawl4ai new

63,373 compare

typhoeus

4,131 compare

httr

985 compare

xml2

223 compare

requests-html

13,863 compare

curl-cffi

1,751 compare

guzzle

23,447 compare

untangle

632 compare

crawlee new

22,720 compare

wreck

378 compare

em-http-request

1,219 compare

html5-php

1,772 compare

symfony-http

2,033 compare

treq

605 compare

domcrawler

4,038 compare

http-2

908 compare

scrapegraphai new

23,278 compare

req

4,781 compare

goquery

14,926 compare

cascadia

754 compare

resty

11,632 compare

htmlquery

781 compare

ferret

5,964 compare

xpath

739 compare

gocrawl

2,053 compare

soup

2,227 compare

nestful

505 compare

chompjs

218 compare

scrapyd

3,087 compare

botasaurus new

4,321 compare

hrequests

1,001 compare

crul

107 compare

requests

3,577 compare

html5-parser

700 compare

node-crawler

6,790 compare

panther

3,062 compare

goutte new

9,215 compare

buzz

1,924 compare

gazpacho

768 compare

gracy

248 compare

httpful

1,803 compare

embed

2,103 compare

spidr

835 compare

kimurai new

1,098 compare

scrapydweb

3,400 compare

chopper

23 compare

photon

12,807 compare

wombat

1,360 compare

autoscraper

7,136 compare

simple-html-dom new

- compare

roach

1,454 compare

gerapy

3,495 compare

ruia

1,743 compare

ayakashi

217 compare

http.rb

3,104 compare

phpscraper

583 compare

dude

425 compare

php-spider

1,341 compare

crwlr-crawler

369 compare

firecrawl new

- compare