Python Libraries Used in Web Scraping

seo

advertools
1,116 0.16.1 (22 days ago) Dec 28 2012

http-client

requests
51,959 2.32.3 (3 months ago) Feb 14 2011
aiohttp
14,936 3.10.5 (22 days ago) Jul 26 2019
httpx
12,973 0.27.2 (14 days ago) Jul 26 2019
curl-impersonate
3,625 v0.6.1 (6 months ago) Feb 23 2022
pycurl
1,071 7.45.3 (6 months ago) Feb 25 2003
curl-cffi
1,751 0.7.1 (a month ago) Feb 23 2022
treq
587 23.11.0 (10 months ago) Dec 28 2012
hrequests
621 0.8.2 (5 months ago) Feb 23 2022

framework

scrapy
52,313 2.11.2 (3 months ago) Jul 26 2019
scrapyd
2,927 1.4.3 (11 months ago) Sep 04 2013
autoscraper
6,161 1.1.14 (2 years ago) Jul 26 2019
scrapydweb
3,103 1.5.0 (6 months ago) Sep 30 2018
gracy
245 1.33.0 (6 months ago) Feb 05 2023
gerapy
3,306 0.9.13 (1 year, 1 month ago) Jul 04 2017
ruia
1,744 0.8.5 (2 years ago) Oct 17 2018
photon
10,890 1.1.9 (5 years ago) Aug 24 2018
dude
420 0.1.3 (1 year, 1 month ago) Feb 20 2022

html-parser

beautifulsoup
- 4.12.3 (7 months ago) Jul 26 2019
lxml
2,653 5.3.0 (a month ago) Dec 13 2022
xmltodict
5,457 0.13.0 (2 years ago) Jul 30 2007
html5lib
1,111 1.1 (4 years ago) Jul 30 2007
cssselect
291 1.2.0 (1 year, 10 months ago) Apr 14 2012
feedparser
1,925 6.0.11 (9 months ago) Jun 15 2007
pyquery
2,291 2.0.1 (12 days ago) Dec 05 2008
parsel
1,120 1.9.1 (5 months ago) Jul 26 2019
selectolax
1,092 0.3.21 (6 months ago) Mar 01 2018
requests-html
13,699 0.10.0 (5 years ago) Feb 25 2018
html5-parser
676 0.4.12 (9 months ago) Jun 03 2007
untangle
612 1.2.1 (2 years ago) Jun 09 2011
chompjs
188 1.3.0 (24 days ago) Jul 30 2007
gazpacho
741 1.1 (3 years ago) Dec 28 2012
chopper
22 0.6.0 (1 year, 4 months ago) Jul 24 2014

html-utils

bleach
2,644 6.1.0 (11 months ago) Feb 21 2010

captcha

captcha_solver
238 0.1.5 (5 years ago) Feb 17 2015
unicaps
218 1.3.0 (6 months ago) Jul 27 2020

blocking

ua-parser
553 0.18.0 (1 year, 2 months ago) Dec 29 2012
cloudscraper
4,266 1.2.71 (1 year, 4 months ago) Dec 28 2012

educational

cloudscraper
4,266 1.2.71 (1 year, 4 months ago) Dec 28 2012
youtube-dl
131,198 2021.12.17 (2 years ago) Feb 22 2012
you-get
49,896 0.4.1730 (22 days ago) Sep 01 2012

scrapers

crosslinked
1,240 0.3.0 (1 year, 2 months ago) Feb 14 2011

css

cssutils
79 2.11.1 (3 months ago) Aug 04 2007

stealth

curl-impersonate
3,625 v0.6.1 (6 months ago) Feb 23 2022
undetected-chromedriver
9,539 3.5.5 (6 months ago) Sep 04 2020
curl-cffi
1,751 0.7.1 (a month ago) Feb 23 2022
selenium-driverless
510 1.9.3.1 (3 months ago) Jul 22 2022
hrequests
621 0.8.2 (5 months ago) Feb 23 2022

html-extractor

html2text
1,795 2024.2.26 (6 months ago) Dec 14 2008
extruct
835 0.17.0 (3 months ago) Oct 27 2015
trafilatura
3,405 1.12.1 (21 days ago) Jul 17 2019
newspaper
14,022 0.2.8 (5 years ago) Dec 28 2012
sumy
3,501 0.11.0 (1 year, 10 months ago) Oct 20 2013
readability
2,641 0.8.1 (4 years ago) Jun 30 2011
extractnet
216 2.0.7 (1 year, 10 months ago) Dec 11 2020
photon
10,890 1.1.9 (5 years ago) Aug 24 2018

microformats

extruct
835 0.17.0 (3 months ago) Oct 27 2015

gui

scrapydweb
3,103 1.5.0 (6 months ago) Sep 30 2018
gerapy
3,306 0.9.13 (1 year, 1 month ago) Jul 04 2017

scrapy

scrapy
52,313 2.11.2 (3 months ago) Jul 26 2019
scrapyd
2,927 1.4.3 (11 months ago) Sep 04 2013
scrapydweb
3,103 1.5.0 (6 months ago) Sep 30 2018
gerapy
3,306 0.9.13 (1 year, 1 month ago) Jul 04 2017
splash
4,063 3.5 (4 years ago) Apr 25 2014

http-utils

httptools
1,194 0.6.1 (10 months ago) Dec 28 2012
requests-cache
1,298 1.2.1 (2 months ago) Feb 14 2011

json-parser

jmespath
2,157 1.0.1 (2 years ago) Feb 09 2022
jsonpath-ng
579 1.6.1 (7 months ago) Feb 09 2022
nested-lookup
209 0.2.25 (2 years ago) Feb 09 2022

jmespath

jmespath
2,157 1.0.1 (2 years ago) Feb 09 2022

data

jmespath
2,157 1.0.1 (2 years ago) Feb 09 2022
jsonpath-ng
579 1.6.1 (7 months ago) Feb 09 2022
xhtml2pdf
2,235 0.2.16 (3 months ago) May 19 2011
nested-lookup
209 0.2.25 (2 years ago) Feb 09 2022

jsonpath

jsonpath-ng
579 1.6.1 (7 months ago) Feb 09 2022

playwright

playwright
11,476 1.46.0 (29 days ago) Feb 24 2021

browser-automation

selenium
30,163 4.24.0 (13 days ago) Apr 25 2008
playwright
11,476 1.46.0 (29 days ago) Feb 24 2021
undetected-chromedriver
9,539 3.5.5 (6 months ago) Sep 04 2020
requestium
1,830 0.4.0 (7 months ago) Dec 28 2012
selenium-driverless
510 1.9.3.1 (3 months ago) Jul 22 2022
splash
4,063 3.5 (4 years ago) Apr 25 2014

selenium

selenium
30,163 4.24.0 (13 days ago) Apr 25 2008
undetected-chromedriver
9,539 3.5.5 (6 months ago) Sep 04 2020
requestium
1,830 0.4.0 (7 months ago) Dec 28 2012
selenium-driverless
510 1.9.3.1 (3 months ago) Jul 22 2022

text-parsing

sumy
3,501 0.11.0 (1 year, 10 months ago) Oct 20 2013

scraper

trafilatura
3,405 1.12.1 (21 days ago) Jul 17 2019
youtube-dl
131,198 2021.12.17 (2 years ago) Feb 22 2012
you-get
49,896 0.4.1730 (22 days ago) Sep 01 2012

headers

ua-parser
553 0.18.0 (1 year, 2 months ago) Dec 29 2012

http

ua-parser
553 0.18.0 (1 year, 2 months ago) Dec 29 2012

data-output

xhtml2pdf
2,235 0.2.16 (3 months ago) May 19 2011

cli-tool

youtube-dl
131,198 2021.12.17 (2 years ago) Feb 22 2012
you-get
49,896 0.4.1730 (22 days ago) Sep 01 2012
Was this page helpful?