cssselectvspyquery
cssselect is a BSD-licensed Python library to parse CSS3 selectors and translate them to XPath 1.0 expressions.
XPath 1.0 expressions can be used in lxml or another XPath engine to find the matching elements in an XML or HTML document.
cssselect is used by other popular Python packages like parsel and scrapy but can also be used on it's own to generate
valid XPath 1.0 expressions for parsing HTML and XML documents in other tools.
Note that because XPath selectors are more powerful than CSS selectors this translation is only possible one way. Converting XPath to CSS selectors is impractical and not supported by cssselect.
PyQuery is a Python library for working with XML and HTML documents. It is similar to BeautifulSoup and is often used as a drop-in replacement for it.
PyQuery is inspired by javascript's jQuery and uses similar API allowing selecting of HTML nodes through CSS selectors. This makes it easy for developers who are already familiar with jQuery to use PyQuery in Python.
Unlike jQuery, PyQuery doesn't support XPath selectors and relies entirely on CSS selectors though offers similar HTML parsing features like selection of HTML elements, their attributes and text as well as html tree modification.
PyQuery also comes with a http client (through requests) so it can load and parse web URLs by itself.
Highlights
Example Use
from cssselect import GenericTranslator, SelectorError
translator = GenericTranslator()
try:
expression = translator.css_to_xpath('div.content')
print(expression)
'descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' content ')]'
except SelectorError as e:
print(f'Invalid selector {e}')
from pyquery import PyQuery as pq
# this is our HTML page:
html = """
<head>
<title>Hello World!</title>
</head>
<body>
<div id="product">
<h1>Product Title</h1>
<p>paragraph 1</p>
<p>paragraph2</p>
<span class="price">$10</span>
</div>
</body>
"""
doc = pq(html)
# we can use CSS selectors:
print(doc('#product .price').text())
"$10"
# it's also possible to modify HTML tree in various ways:
# insert text into selected element:
print(doc('h1').append('<span>discounted</span>'))
"<h1>Product Title<span>discounted</span></h1>"
# or remove elements
doc('p').remove()
print(doc('#product').html())
"""
<h1>Product Title<span>discounted</span></h1>
<span class="price">$10</span>
"""
# pyquery can also retrieve web documents using requests:
doc = pq(url='http://httpbin.org/html', headers={"User-Agent": "webscraping.fyi"})
print(doc('h1').html())