jmespathvsjsonpath-ng
JMESPath (pronounced “james path”) allows you to declaratively specify how to extract elements from a JSON document.
In web scraping, jmespath is a powerful tool for parsing and reshaping large JSON datasets. Jmespath is fast and easily extendible following it's own powerful query language.
For more see the Json parsing introduction section.
jsonpath-ng is a Python library for parsing and querying JSON data.
It is a powerful tool for extracting and manipulating data from JSON structures,
The library uses a syntax similar to XPath, which is a well-known language for querying and manipulating XML data, to query and extract data from JSON structures. This makes it familiar and intuitive for many developers who have worked with XML in the past. JSONPath implementation in Python.
JSONPath is a JSON query path language inspired by XPath (path language for querying XML/HTML). For more see the initial syntax proposal.
jsonpath-ng is implemented in pure Python and can be easily extended with additional python functions if needed.
Most commonly used JSONPath feature in web scraping is the recursive key lookup ($..key)
which is a convenient way to find specific data fields in large datasets.
Example Use
import jmespath
data = {
"data": {
"info": {
"products": [
{"price": {"usd": 1}, "_type": "product", "id": "123"},
{"price": {"usd": 2}, "_type": "product", "id": "345"}
]
}
}
}
# easily reshape nested dataset to flat structure:
jmespath.search("data.info.products[*].{id:id, price:price.usd}", data)
[{'id': '123', 'price': 1}, {'id': '345', 'price': 2}]
from jsonpath_ng import jsonpath, parse
# A robust parser, not just a regex. (Makes powerful extensions possible; see below)
jsonpath_expr = parse('foo[*].baz')
# Extracting values is easy
[match.value for match in jsonpath_expr.find({'foo': [{'baz': 1}, {'baz': 2}]})]
[1, 2]
# Matches remember where they came from
[str(match.full_path) for match in jsonpath_expr.find({'foo': [{'baz': 1}, {'baz': 2}]})]
['foo.[0].baz', 'foo.[1].baz']
# And this can be useful for automatically providing ids for bits of data that do not have them (currently a global switch)
jsonpath.auto_id_field = 'id'
[match.value for match in parse('foo[*].id').find({'foo': [{'id': 'bizzle'}, {'baz': 3}]})]
['foo.bizzle', 'foo.[1]']
# A handy extension: named operators like `parent`
[match.value for match in parse('a.*.b.`parent`.c').find({'a': {'x': {'b': 1, 'c': 'number one'}, 'y': {'b': 2, 'c': 'number two'}}})]
['number two', 'number one']
# You can also build expressions directly quite easily
from jsonpath_ng.jsonpath import Fields
from jsonpath_ng.jsonpath import Slice
jsonpath_expr_direct = Fields('foo').child(Slice('*')).child(Fields('baz')) # This is equivalent