wreckvsrequests
Wreck is an HTTP client library for Node.js. It provides a simple, consistent API for making HTTP requests, including support for both the client and server side of an HTTP transaction.
Wreck is a very minimal but stable as it's part of Hapi web framework project. For web scraping, it doesn't offer required features like proxy configuration or http2 support so it's not recommended.
The requests package is a popular library for making HTTP requests in Python.
It provides a simple, easy-to-use API for sending HTTP/1.1 requests, and it abstracts away many of the low-level details of working with HTTP.
One of the key features of requests is its simple API. You can send a GET request with a single line of code:
import requests
response = requests.get('https://webscraping.fyi/lib/requests/')
pip install requests
Highlights
syncease-of-useno-http2no-asyncpopular
Example Use
const Wreck = require('wreck');
// get request
Wreck.get('http://example.com', (err, res, payload) => {
if (err) {
throw err;
}
console.log(payload.toString());
});
// post request
const options = {
headers: { 'content-type': 'application/json' },
payload: JSON.stringify({ name: 'John Doe' })
};
Wreck.post('http://example.com', options, (err, res, payload) => {
if (err) {
throw err;
}
console.log(payload.toString());
});
import requests
# get request:
response = requests.get("http://webscraping.fyi/")
response.status_code
200
response.text
"text"
response.content
b"bytes"
# requests can automatically convert json responses to Python dictionaries:
response = requests.get("http://httpbin.org/json")
print(response.json())
{'slideshow': {'author': 'Yours Truly', 'date': 'date of publication', 'slides': [{'title': 'Wake up to WonderWidgets!', 'type': 'all'}, {'items': ['Why <em>WonderWidgets</em> are great', 'Who <em>buys</em> WonderWidgets'], 'title': 'Overview', 'type': 'all'}], 'title': 'Sample Slide Show'}}
# for POST request it can ingest Python's dictionaries as JSON:
response = requests.post("http://httpbin.org/post", json={"query": "hello world"})
# or form data:
response = requests.post("http://httpbin.org/post", data={"query": "hello world"})
# Session object can be used to automatically keep track of cookies and set defaults:
from requests import Session
s = Session()
s.headers = {"User-Agent": "webscraping.fyi"}
s.get('http://httpbin.org/cookies/set/foo/bar')
print(s.cookies['foo'])
'bar'
print(s.get('http://httpbin.org/cookies').json())
{'cookies': {'foo': 'bar'}}