chompjsvsxpath

MIT 5 1 218

47.0 thousand (month) Jul 30 2007 1.4.0(2025-08-04 21:07:54 ago)

739 2 18 MIT

Jun 08 2019 58.1 thousand (month) v1.3.6(2026-02-23 07:10:29 ago)

chompjs can be used in web scrapping for turning JavaScript objects embedded in pages into valid Python dictionaries.

In web scraping this is particularly useful for parsing Javascript variables like: python import chompjs js = """ var myObj = { myMethod: function(params) { // ... }, myValue: 100 } """ chompjs.parse_js_object(js, json_params={'strict': False}) {'myMethod': 'function(params) {\n // ...\n }', 'myValue': 100}

In practice this can be used to extract hidden JSON data like data from <script id=__NEXT_DATA__> elements from nextjs (and similar) websites. Unlike json.loads command chompjs can ingest json documents that contain javascript natives like functions making it a super easy way to scrape hidden web data objects.

xpath is a library for Go that allows you to use XPath expressions to select elements from an HTML document. It is built on top of the html package in the Go standard library, and provides a way to select elements from an HTML document using XPath expressions, which are more powerful and expressive than CSS selectors.

Example Use

```python # basic use import chompjs js = """ var myObj = { myMethod: function(params) { // ... }, myValue: 100 } """ chompjs.parse_js_object(js, json_params={'strict': False}) {'myMethod': 'function(params) {\n // ...\n }', 'myValue': 100} # example how to use with hidden data parsing: import httpx import chompjs from parsel import Selector response = httpx.get("http://example.com") hidden_script = Selector(response.text).css("script#__NEXT_DATA__::text").get() data = chompjs.parse_js_object(hidden_script) print(data['props']) ```

```go package main import ( "fmt" "github.com/antchfx/xpath" "golang.org/x/net/html" "strings" ) func main() { // Create an HTML string html := `

Hello, World!

Example

` // Parse the HTML string into a node tree doc, err := html.Parse(strings.NewReader(html)) if err != nil { fmt.Println("Error:", err) return } // Compile the XPath expression expr, err := xpath.Compile("//p") if err != nil { fmt.Println("Error:", err) return } // Use the Evaluate method to select elements from the document nodes, err := expr.Evaluate(xpath.NodeNavigator(doc)) if err != nil { fmt.Println("Error:", err) return } if nodes.MoveNext() { fmt.Println(nodes.Current().Value()) // > Hello, World! } } ```