Skip to content

ip-addressvsrequests-cache

MIT 32 1 506
39.2 million (month) May 05 2015 9.0.5(6 months ago)
1,251 5 31 BSD-2-Clause
1.2.0(a month ago) Feb 14 2011 1.5 million (month)

The "ip-address" library is a Node.js library that provides utility functions for working with IP addresses, similar to "ip" library. It provides functions for parsing, validating, and converting IP addresses between different formats, both IPv4 and IPv6.

requests-cache is an extension to the popular requests package and it provides easy request/response caching.

This can be very useful in web scraper development as it'll speed up all requests. requests-cache can also be used for programs that integrate web scrapers as it's an easy caching layer for the most time consuming part of web scraping - http connections.

Some features:

  • Ease of use
    Keep using the requests library you're already familiar with. Add caching with a drop-in replacement for requests.Session, or install globally to add transparent caching to all requests functions.
  • Performance
    Get sub-millisecond response times for cached responses. When they expire, you still save time with conditional requests.
  • Persistence
    Works with several storage backends including SQLite, Redis, MongoDB, and DynamoDB; or save responses as plain JSON files, YAML, and more
  • Expiration
    Use Cache-Control and other standard HTTP headers, define your own expiration schedule, keep your cache clutter-free with backends that natively support TTL, or any combination of strategies
  • Customization
    Works out of the box with zero config, but with a robust set of features for configuring and extending the library to suit your needs
  • Compatibility
    Can be combined with other popular libraries based on requests

Example Use


const IPAddress = require("ip-address").Address;

// slice and manipulate addresses:
const ipv4 = new IPAddress("192.168.1.1");
console.log(ipv4.addressMinusSuffix); // '192.168.1'

// validate addresses:
console.log(IPAddress.isValid("192.168.1.1")); // true
console.log(IPAddress.isValid("2001:0db8:85a3:0000:0000:8a2e:0370:7334")); // true
import requests_cache

# to use requests_cache just replace requests.Session with requests_cache.CachedSession
session = requests_cache.CachedSession('demo_cache')
for i in range(60):
    session.get('https://httpbin.org/delay/1')

# or patch global requests
requests_cache.install_cache('demo_cache')
requests.get('https://httpbin.org/delay/1')

# there are various configuration options: 
session = CachedSession(
  'demo_cache',
  use_cache_dir=True,                # Save files in the default user cache dir
  cache_control=True,                # Use Cache-Control response headers for expiration, if available
  expire_after=timedelta(days=1),    # Otherwise expire responses after one day
  allowable_codes=[200, 400],        # Cache 400 responses as a solemn reminder of your failures
  allowable_methods=['GET', 'POST'], # Cache whatever HTTP methods you want
  ignored_parameters=['api_key'],    # Don't match this request param, and redact if from the cache
  match_headers=['Accept-Language'], # Cache a different response per language
  stale_if_error=True,               # In case of request errors, use stale cache data if possible
) 

Alternatives / Similar