ip-addressvsrequests-cache
The "ip-address" library is a Node.js library that provides utility functions for working with IP addresses, similar to "ip" library. It provides functions for parsing, validating, and converting IP addresses between different formats, both IPv4 and IPv6.
requests-cache is an extension to the popular requests package and it provides easy request/response caching.
This can be very useful in web scraper development as it'll speed up all requests. requests-cache can also be used for programs that integrate web scrapers as it's an easy caching layer for the most time consuming part of web scraping - http connections.
Some features:
- Ease of use
Keep using the requests library you're already familiar with. Add caching with a drop-in replacement for requests.Session, or install globally to add transparent caching to all requests functions. - Performance
Get sub-millisecond response times for cached responses. When they expire, you still save time with conditional requests. - Persistence
Works with several storage backends including SQLite, Redis, MongoDB, and DynamoDB; or save responses as plain JSON files, YAML, and more - Expiration
Use Cache-Control and other standard HTTP headers, define your own expiration schedule, keep your cache clutter-free with backends that natively support TTL, or any combination of strategies - Customization
Works out of the box with zero config, but with a robust set of features for configuring and extending the library to suit your needs - Compatibility
Can be combined with other popular libraries based on requests
Example Use
const IPAddress = require("ip-address").Address;
// slice and manipulate addresses:
const ipv4 = new IPAddress("192.168.1.1");
console.log(ipv4.addressMinusSuffix); // '192.168.1'
// validate addresses:
console.log(IPAddress.isValid("192.168.1.1")); // true
console.log(IPAddress.isValid("2001:0db8:85a3:0000:0000:8a2e:0370:7334")); // true
import requests_cache
# to use requests_cache just replace requests.Session with requests_cache.CachedSession
session = requests_cache.CachedSession('demo_cache')
for i in range(60):
session.get('https://httpbin.org/delay/1')
# or patch global requests
requests_cache.install_cache('demo_cache')
requests.get('https://httpbin.org/delay/1')
# there are various configuration options:
session = CachedSession(
'demo_cache',
use_cache_dir=True, # Save files in the default user cache dir
cache_control=True, # Use Cache-Control response headers for expiration, if available
expire_after=timedelta(days=1), # Otherwise expire responses after one day
allowable_codes=[200, 400], # Cache 400 responses as a solemn reminder of your failures
allowable_methods=['GET', 'POST'], # Cache whatever HTTP methods you want
ignored_parameters=['api_key'], # Don't match this request param, and redact if from the cache
match_headers=['Accept-Language'], # Cache a different response per language
stale_if_error=True, # In case of request errors, use stale cache data if possible
)