Popular Tools

HTTP Debuggers

HTTP debugging tools (aka REST API clients, API testing tools etc.) are GUI programs that allow to test sending of HTTP request.

These tools are useful for testing requests used in web scraping as provides a playground for reverse engineering website behavior and it's hidden APIs.

Here are some of the most common tools used in web scraping:

Tool	Highlights
postman	Very popular, code export
insomnia	similar to postman
httpie web	web client

HTTP interception

Step beyond HTTP debuggers there are tools that can intercept HTTP traffic. This can be used to reverse engineer phone app traffic through MITM traffic capture or as an alternative to browser devtools for inspecting web traffic.

Tool	Highlights
mitmproxy	web+terminal UI, easy to extend with Python, free
http-toolkit	easy setup, great documentation, freemium + 7.5usd/mo
proxyman	powerful interception features, only windows/macos, 69usd/license
charles	feature-rich, 50usd/license
fiddler everywhere	very popular, 12usd/mo