node-crawlervsscrapling
node-crawler is a popular web scraping library for Node.js that allows you to easily navigate and extract data from websites. It has a simple API and supports concurrency, making it efficient for scraping large numbers of pages.
Features:
- Server-side DOM & automatic jQuery insertion with Cheerio (default) or JSDOM,
- Configurable pool size and retries,
- Control rate limit,
- Priority queue of requests,
- forceUTF8 mode to let crawler deal for you with charset detection and conversion,
- Compatible with 4.x or newer version.
- Http2 support
- Proxy support
Scrapling is an adaptive web scraping framework for Python that introduces "self-healing" selectors — selectors that can track and find elements even when the website's DOM structure changes. This solves one of the biggest maintenance headaches in web scraping: broken selectors after website updates.
Key features include:
- Self-healing selectors Scrapling uses smart element matching that can identify target elements even after the page structure changes. It builds a fingerprint of the element based on multiple attributes (text, position, siblings, attributes) and uses fuzzy matching to relocate it.
- Multiple parsing backends Supports different parsing engines including lxml (fast) and a custom engine, allowing you to choose the right balance of speed and features.
- Scrapy-like Spider API Provides a familiar Spider class pattern for organizing crawling logic, similar to Scrapy but with the added benefit of adaptive selectors.
- CSS and XPath selectors Full support for CSS selectors and XPath, plus the adaptive matching system on top.
- Type hints and modern Python Built with full type annotations and 92% test coverage for reliability.
- Async support Supports asynchronous crawling for efficient concurrent scraping.
Scrapling gained massive traction in 2025 as one of the most starred new Python scraping libraries. It is particularly useful for scraping targets that frequently update their HTML structure, where traditional selector-based scrapers would break.
Highlights
css-selectorsxpathfastpopular
Example Use
```javascript
const Crawler = require('crawler');
const c = new Crawler({
maxConnections: 10,
// This will be called for each crawled page
callback: (error, res, done) => {
if (error) {
console.log(error);
} else {
const $ = res.$;
// $ is Cheerio by default
//a lean implementation of core jQuery designed specifically for the server
console.log($('title').text());
}
done();
}
});
// Queue just one URL, with default callback
c.queue('http://www.amazon.com');
// Queue a list of URLs
c.queue(['http://www.google.com/','http://www.yahoo.com']);
// Queue URLs with custom callbacks & parameters
c.queue([{
uri: 'http://parishackers.org/',
jQuery: false,
// The global callback won't be called
callback: (error, res, done) => {
if (error) {
console.log(error);
} else {
console.log('Grabbed', res.body.length, 'bytes');
}
done();
}
}]);
// Queue some HTML code directly without grabbing (mostly for tests)
c.queue([{
html: '
This is a test
' }]); ```
```python
from scrapling import Fetcher, StealthFetcher, PlayWrightFetcher
# Simple fetching with adaptive parsing
fetcher = Fetcher()
page = fetcher.get("https://example.com/products")
# CSS selectors work as expected
products = page.css(".product-card")
for product in products:
name = product.css_first(".name").text()
price = product.css_first(".price").text()
print(f"{name}: {price}")
# Adaptive selector - finds the element even if DOM changes
# Uses element fingerprinting for resilient matching
element = page.find("Product Title", auto_match=True)
# Stealth fetching with anti-bot bypass
stealth = StealthFetcher()
page = stealth.get("https://protected-site.com")
# Playwright-based fetching for JS-rendered pages
pw = PlayWrightFetcher()
page = pw.get("https://spa-example.com", headless=True)
```
Alternatives / Similar
katana
new
crawl4ai
new
scrapling
new
crawlee
new
mechanize
new
scrapegraphai
new
botasaurus
new
goutte
new
kimurai
new
firecrawl
new
jsdom
new
katana
new
crawl4ai
new
crawlee
new
mechanize
new
scrapegraphai
new
botasaurus
new
goutte
new
kimurai
new
simple-html-dom
new
firecrawl
new