requests-htmlvsnokogiri
requests-html is a Python package that allows you to easily make HTTP requests and parse the HTML content of web pages. It is built on top of the popular requests package and uses the html parser from the lxml library, which makes it fast and efficient. This package is designed to provide a simple and convenient API for web scraping, and it supports features such as JavaScript rendering, CSS selectors, and form submissions.
It also offers a lot of functionalities such as cookie, session, and proxy support, which makes it an easy-to-use package for web scraping and web automation tasks.
In short requests-html offers:
- Full JavaScript support!
- CSS Selectors (a.k.a jQuery-style, thanks to PyQuery).
- XPath Selectors, for the faint of heart.
- Mocked user-agent (like a real web browser).
- Automatic following of redirects.
- Connection–pooling and cookie persistence.
- The Requests experience you know and love, with magical parsing abilities.
- Async Support
Nokogiri is a Ruby gem that provides a simple and powerful way to parse and search XML and HTML documents. It is built on top of the underlying C library libxml2, which is known for its speed and reliability.
Nokogiri provides a simple and intuitive API for parsing and searching XML and HTML documents, and it is widely used in the Ruby ecosystem for web scraping and data extraction.
One of the main features of Nokogiri is its ability to search and navigate through XML and HTML documents using a CSS or XPath selectors.
Nokogiri also provides a variety of other features that can simplify the process of working with XML and HTML documents. It can automatically handle character encodings and normalize documents, it can parse and search large documents with low memory usage, and it can validate documents against a DTD or schema.
Highlights
Example Use
from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://www.example.com')
# print the HTML content of the page
print(r.html.html)
# use CSS selectors to find specific elements on the page
title = r.html.find('title', first=True)
print(title.text)
require 'nokogiri'
html_string = '<html><head><title>Page Title</title></head><body><h1 class="header-class">Hello World!</h1><p>This is a sample webpage.</p></body></html>'
# Parse the HTML string
doc = Nokogiri::HTML(html_string)
# Extract the class attribute of h1 tag using CSS selector
h1_class = doc.css("h1")[0]['class']
# or XPath
h1_class = doc.xpath("//h1")[0]['class']
puts "H1 class: #{h1_class}"