cloudscrapervsyou-get
A simple Python module to bypass Cloudflare's anti-bot page (also known as "I'm Under Attack Mode", or IUAM), implemented with Requests. Cloudflare changes their techniques periodically, so I will update this repo frequently.
This can be useful if you wish to scrape or crawl a website protected with Cloudflare. Cloudflare's anti-bot page currently just checks if the client supports Javascript, though they may add additional techniques in the future.
Due to Cloudflare continually changing and hardening their protection page, cloudscraper requires a JavaScript Engine/interpreter to solve Javascript challenges. This allows the script to easily impersonate a regular web browser without explicitly deobfuscating and parsing Cloudflare's Javascript.
For reference, this is the default message Cloudflare uses for these sorts of pages:
Checking your browser before accessing website.com.
This process is automatic. Your browser will redirect to your requested content shortly.
Please allow up to 5 seconds...
Any script using cloudscraper will sleep for ~5 seconds for the first visit to any site with Cloudflare anti-bots enabled, though no delay will occur after the first request.
Cloudscraper is a great introduction to javascript fingerprint/challenge scraper blocking and is a useful educational tool even if it doesn't always work.
you-get is a command-line utility and a library for downloading multimedia content from various websites, such as YouTube, Instagram, TikTok, and many others. It supports a wide range of video and audio formats, and can be used to download both live streams and on-demand videos. The library is written in Python and can be easily integrated into other Python projects.
Just like Youtube-dl, you-get contains open-source scrapers for hundreds of websites and is a great tool to learn about web scraping and popular web scraping techniques.
Example Use
import cloudscraper
scraper = cloudscraper.create_scraper() # returns a CloudScraper instance
# Or: scraper = cloudscraper.CloudScraper() # CloudScraper inherits from requests.Session
print(scraper.get("http://somesite.com").text) # => "<!DOCTYPE html><html><head>..."
$ you-get 'https://www.youtube.com/watch?v=jNQXAC9IVRw'
site: YouTube
title: Me at the zoo
stream:
- itag: 43
container: webm
quality: medium
size: 0.5 MiB (564215 bytes)
# download-with: you-get --itag=43 [URL]
Downloading Me at the zoo.webm ...
100% ( 0.5/ 0.5MB) ├██████████████████████████████████┤[1/1] 6 MB/s
Saving Me at the zoo.en.srt ... Done.
import you_get
# will save file to `output_video.mp4`
you_get.download("https://www.youtube.com/watch?v=dQw4w9WgXcQ", -o "output_video.mp4")