dudevsmechanize
Dude (dude uncomplicated data extraction) is a very simple framework for writing web scrapers using Python decorators. The design, inspired by Flask, was to easily build a web scraper in just a few lines of code. Dude has an easy-to-learn syntax.
The simplest web scraper will look like this: ```python from dude import select
@select(css="a") def get_link(element): return {"url": element.get_attribute("href")} ```
dude supports multiple parser backends:
- playwright
- lxml
- parsel
- beautifulsoup
- pyppeteer
- selenium
Mechanize is a Ruby library for automating interaction with websites. It automatically stores and sends cookies, follows redirects, and can submit forms — making it behave like a web browser without needing an actual browser engine.
Key features include:
- Automatic cookie management Stores cookies received from servers and sends them back on subsequent requests, maintaining session state across multiple pages.
- Form handling Can find, fill in, and submit HTML forms programmatically. Supports text inputs, selects, checkboxes, radio buttons, and file uploads.
- Link following Navigate through pages by clicking links using their text content, CSS selectors, or href patterns.
- History and back/forward Maintains a browsing history, allowing you to go back and forward through visited pages.
- HTTP authentication Supports basic and digest HTTP authentication.
- Proxy support Can route requests through HTTP proxies.
- Redirect handling Automatically follows HTTP redirects (configurable).
Mechanize is one of the oldest and most established web interaction libraries in Ruby. It is best suited for scraping traditional server-rendered websites with forms and multi-page workflows. For JavaScript-heavy sites, a browser automation tool like Selenium or Playwright is recommended instead.