jsonpath-ngvskiba
jsonpath-ng is a Python library for parsing and querying JSON data.
It is a powerful tool for extracting and manipulating data from JSON structures,
The library uses a syntax similar to XPath, which is a well-known language for querying and manipulating XML data, to query and extract data from JSON structures. This makes it familiar and intuitive for many developers who have worked with XML in the past. JSONPath implementation in Python.
JSONPath is a JSON query path language inspired by XPath (path language for querying XML/HTML). For more see the initial syntax proposal.
jsonpath-ng is implemented in pure Python and can be easily extended with additional python functions if needed.
Most commonly used JSONPath feature in web scraping is the recursive key lookup ($..key)
which is a convenient way to find specific data fields in large datasets.
Kiba is a lightweight Ruby gem that provides a simple and powerful way to process and transform data in an ETL (Extract, Transform, Load) pipeline. It allows you to define a set of operations to perform on the data, and then automatically applies those operations to the data, making it easy to extract, transform, and load data from various sources and formats.
Kiba provides a simple and intuitive API for defining the pipeline, and it is built on top of the Enumerator API, which allows for easy manipulation of large datasets with low memory usage.
Example Use
from jsonpath_ng import jsonpath, parse
# A robust parser, not just a regex. (Makes powerful extensions possible; see below)
jsonpath_expr = parse('foo[*].baz')
# Extracting values is easy
[match.value for match in jsonpath_expr.find({'foo': [{'baz': 1}, {'baz': 2}]})]
[1, 2]
# Matches remember where they came from
[str(match.full_path) for match in jsonpath_expr.find({'foo': [{'baz': 1}, {'baz': 2}]})]
['foo.[0].baz', 'foo.[1].baz']
# And this can be useful for automatically providing ids for bits of data that do not have them (currently a global switch)
jsonpath.auto_id_field = 'id'
[match.value for match in parse('foo[*].id').find({'foo': [{'id': 'bizzle'}, {'baz': 3}]})]
['foo.bizzle', 'foo.[1]']
# A handy extension: named operators like `parent`
[match.value for match in parse('a.*.b.`parent`.c').find({'a': {'x': {'b': 1, 'c': 'number one'}, 'y': {'b': 2, 'c': 'number two'}}})]
['number two', 'number one']
# You can also build expressions directly quite easily
from jsonpath_ng.jsonpath import Fields
from jsonpath_ng.jsonpath import Slice
jsonpath_expr_direct = Fields('foo').child(Slice('*')).child(Fields('baz')) # This is equivalent
require 'kiba'
data = [{ name: 'Alice', age: 25 }, { name: 'Bob', age: 30 }]
Kiba.parse do
source Kiba::Common::EnumerableSource, data
transform { |row| row[:age] += 1 }
destination Kiba::Common::EnumerableDestination
end.run
# Output: [{ name: 'Alice', age: 26 }, { name: 'Bob', age: 31 }]