embedvspyquery
PHP library to get information from any web page (using oembed, opengraph, twitter-cards, scrapping the html, etc). It's compatible with any web service (youtube, vimeo, flickr, instagram, etc) and has adapters to some sites like (archive.org, github, facebook, etc).
PyQuery is a Python library for working with XML and HTML documents. It is similar to BeautifulSoup and is often used as a drop-in replacement for it.
PyQuery is inspired by javascript's jQuery and uses similar API allowing selecting of HTML nodes through CSS selectors. This makes it easy for developers who are already familiar with jQuery to use PyQuery in Python.
Unlike jQuery, PyQuery doesn't support XPath selectors and relies entirely on CSS selectors though offers similar HTML parsing features like selection of HTML elements, their attributes and text as well as html tree modification.
PyQuery also comes with a http client (through requests) so it can load and parse web URLs by itself.
Highlights
Example Use
use Embed\Embed;
$embed = new Embed();
//Load any url:
$info = $embed->get('https://www.youtube.com/watch?v=PP1xn5wHtxE');
//Get content info
$info->title; //The page title
$info->description; //The page description
$info->url; //The canonical url
$info->keywords; //The page keywords
$info->image; //The thumbnail or main image
$info->code->html; //The code to embed the image, video, etc
$info->code->width; //The exact width of the embed code (if exists)
$info->code->height; //The exact height of the embed code (if exists)
$info->code->ratio; //The aspect ratio (width/height)
$info->authorName; //The resource author
$info->authorUrl; //The author url
$info->cms; //The cms used
$info->language; //The language of the page
$info->languages; //The alternative languages
$info->providerName; //The provider name of the page (Youtube, Twitter, Instagram, etc)
$info->providerUrl; //The provider url
$info->icon; //The big icon of the site
$info->favicon; //The favicon of the site (an .ico file or a png with up to 32x32px)
$info->publishedTime; //The published time of the resource
$info->license; //The license url of the resource
$info->feeds; //The RSS/Atom feeds
from pyquery import PyQuery as pq
# this is our HTML page:
html = """
<head>
<title>Hello World!</title>
</head>
<body>
<div id="product">
<h1>Product Title</h1>
<p>paragraph 1</p>
<p>paragraph2</p>
<span class="price">$10</span>
</div>
</body>
"""
doc = pq(html)
# we can use CSS selectors:
print(doc('#product .price').text())
"$10"
# it's also possible to modify HTML tree in various ways:
# insert text into selected element:
print(doc('h1').append('<span>discounted</span>'))
"<h1>Product Title<span>discounted</span></h1>"
# or remove elements
doc('p').remove()
print(doc('#product').html())
"""
<h1>Product Title<span>discounted</span></h1>
<span class="price">$10</span>
"""
# pyquery can also retrieve web documents using requests:
doc = pq(url='http://httpbin.org/html', headers={"User-Agent": "webscraping.fyi"})
print(doc('h1').html())