playwrightvssplash
Playwright is a Node.js library that provides a high-level API to automate web browsers. It allows you to automate browser tasks such as generating screenshots, creating PDFs, and testing web pages by simulating user interactions. Playwright is similar to Puppeteer, but it supports more browsers and it also provide capabilities for automation of browser like Microsoft Edge and Safari.
Playwright is commonly used for web scraping, end-to-end testing, and browser automation.
Playwright is a spiritual successor to Puppeter and is available in more languages and has access to more browser types.
Splash is a javascript rendering service with an HTTP API. It's a lightweight browser with an HTTP API, implemented in Python 3 using Twisted and QT5.
It is built on top of the QtWebkit library and allows developers to interact with web pages in a headless mode, which means that the web pages are rendered in the background, without displaying them on the screen.
splash is particularly useful for web scraping and web testing tasks, as it allows developers to interact with web pages in a way that is very similar to how a human user would interact with the browser.
It also allows you to execute javascript and interact with web pages even if they use heavy javascript.
Unlike Selenium or Playwright, splash is powered by webkit embedded browser instead of a real browser like Chrome or Firefox. As a down-side splash requests are easy to detect and block when scraping websites with anti-scraping features.
One benefit of splash is that it seemlesly integrates with Scrapy.
Example Use
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch();
const context = await browser.newContext();
const page = await context.newPage();
await page.goto('https://www.example.com/form');
// fill in the form
await page.fill('input[name="name"]', 'John Doe');
await page.fill('input[name="email"]', 'johndoe@example.com');
await page.selectOption('select[name="country"]', 'US');
// submit the form
await page.click('button[type="submit"]');
// wait for the page to load after the form is submitted
await page.waitForNavigation();
// take a screenshot
await page.screenshot({path: 'form-submission.png'});
await browser.close();
})();
# once splash server is started it can be requested to render pages through
# HTTP requests:
import requests
url = "http://localhost:8050/render.html"
payload = {
'url': 'https://www.example.com',
'timeout': 30,
'wait': 2
}
response = requests.get(url, params=payload)
# Get the page HTML
print(response.text)