Skip to content

Popular Tools

HTTP Debuggers

HTTP debugging tools (aka REST API clients, API testing tools etc.) are GUI programs that allow to test sending of HTTP request.

These tools are useful for testing requests used in web scraping as provides a playground for reverse engineering website behavior and it's hidden APIs.

Here are some of the most common tools used in web scraping:

Tool Highlights
postman Very popular, code export
insomnia similar to postman
httpie web web client

HTTP interception

Step beyond HTTP debuggers there are tools that can intercept HTTP traffic. This can be used to reverse engineer phone app traffic through MITM traffic capture or as an alternative to browser devtools for inspecting web traffic.

Tool Highlights
mitmproxy web+terminal UI, easy to extend with Python, free
http-toolkit easy setup, great documentation, freemium + 7.5usd/mo
proxyman powerful interception features, only windows/macos, 69usd/license
charles feature-rich, 50usd/license
fiddler everywhere very popular, 12usd/mo