Python Libraries Used in Web Scraping

seo

advertools
1,093 0.14.3 (18 days ago) Dec 28 2012

http-client

requests
51,729 2.32.3 (a month ago) Feb 14 2011
aiohttp
14,776 3.9.5 (2 months ago) Jul 26 2019
httpx
12,702 0.27.0 (4 months ago) Jul 26 2019
curl-impersonate
3,467 v0.6.1 (4 months ago) Feb 23 2022
pycurl
1,058 7.45.3 (4 months ago) Feb 25 2003
curl-cffi
1,633 0.7.0 (12 days ago) Feb 23 2022
treq
585 23.11.0 (8 months ago) Dec 28 2012
hrequests
541 0.8.2 (3 months ago) Feb 23 2022

framework

scrapy
51,636 2.11.2 (2 months ago) Jul 26 2019
scrapyd
2,884 1.4.3 (9 months ago) Sep 04 2013
autoscraper
6,060 1.1.14 (1 year, 11 months ago) Jul 26 2019
scrapydweb
3,048 1.5.0 (5 months ago) Sep 30 2018
gracy
240 1.33.0 (4 months ago) Feb 05 2023
gerapy
3,257 0.9.13 (11 months ago) Jul 04 2017
ruia
1,737 0.8.5 (1 year, 10 months ago) Oct 17 2018
photon
10,672 1.1.9 (5 years ago) Aug 24 2018
dude
413 0.1.3 (11 months ago) Feb 20 2022

html-parser

beautifulsoup
- 4.12.3 (5 months ago) Jul 26 2019
lxml
2,614 5.2.2 (2 months ago) Dec 13 2022
xmltodict
5,416 0.13.0 (2 years ago) Jul 30 2007
html5lib
1,105 1.1 (4 years ago) Jul 30 2007
cssselect
287 1.2.0 (1 year, 8 months ago) Apr 14 2012
feedparser
1,867 6.0.11 (7 months ago) Jun 15 2007
pyquery
2,279 2.0.0 (1 year, 6 months ago) Dec 05 2008
parsel
1,101 1.9.1 (3 months ago) Jul 26 2019
requests-html
13,615 0.10.0 (5 years ago) Feb 25 2018
selectolax
1,052 0.3.21 (4 months ago) Mar 01 2018
untangle
609 1.2.1 (2 years ago) Jun 09 2011
html5-parser
673 0.4.12 (7 months ago) Jun 03 2007
chompjs
181 1.2.3 (5 months ago) Jul 30 2007
gazpacho
738 1.1 (3 years ago) Dec 28 2012
chopper
22 0.6.0 (1 year, 2 months ago) Jul 24 2014

html-utils

bleach
2,634 6.1.0 (9 months ago) Feb 21 2010

captcha

captcha_solver
236 0.1.5 (5 years ago) Feb 17 2015
unicaps
211 1.3.0 (4 months ago) Jul 27 2020

educational

cloudscraper
4,121 1.2.71 (1 year, 2 months ago) Dec 28 2012
youtube-dl
129,778 2021.12.17 (2 years ago) Feb 22 2012
you-get
49,559 0.4.1710 (21 days ago) Sep 01 2012

blocking

ua-parser
551 0.18.0 (1 year, 8 days ago) Dec 29 2012
cloudscraper
4,121 1.2.71 (1 year, 2 months ago) Dec 28 2012

scrapers

crosslinked
1,195 0.3.0 (1 year, 27 days ago) Feb 14 2011

css

cssutils
70 2.11.0 (2 months ago) Aug 04 2007

stealth

curl-impersonate
3,467 v0.6.1 (4 months ago) Feb 23 2022
undetected-chromedriver
9,064 3.5.5 (4 months ago) Sep 04 2020
curl-cffi
1,633 0.7.0 (12 days ago) Feb 23 2022
selenium-driverless
411 1.9.3.1 (a month ago) Jul 22 2022
hrequests
541 0.8.2 (3 months ago) Feb 23 2022

html-extractor

html2text
1,739 2024.2.26 (4 months ago) Dec 14 2008
sumy
3,457 0.11.0 (1 year, 8 months ago) Oct 20 2013
extruct
830 0.17.0 (a month ago) Oct 27 2015
newspaper
13,882 0.2.8 (5 years ago) Dec 28 2012
trafilatura
3,188 1.11.0 (18 days ago) Jul 17 2019
readability
2,589 0.8.1 (4 years ago) Jun 30 2011
extractnet
194 2.0.7 (1 year, 8 months ago) Dec 11 2020
photon
10,672 1.1.9 (5 years ago) Aug 24 2018

microformats

extruct
830 0.17.0 (a month ago) Oct 27 2015

gui

scrapydweb
3,048 1.5.0 (5 months ago) Sep 30 2018
gerapy
3,257 0.9.13 (11 months ago) Jul 04 2017

scrapy

scrapy
51,636 2.11.2 (2 months ago) Jul 26 2019
scrapyd
2,884 1.4.3 (9 months ago) Sep 04 2013
scrapydweb
3,048 1.5.0 (5 months ago) Sep 30 2018
gerapy
3,257 0.9.13 (11 months ago) Jul 04 2017
splash
4,039 3.5 (4 years ago) Apr 25 2014

http-utils

httptools
1,173 0.6.1 (8 months ago) Dec 28 2012
requests-cache
1,269 1.2.0 (4 months ago) Feb 14 2011

data

jmespath
2,130 1.0.1 (2 years ago) Feb 09 2022
jsonpath-ng
545 1.6.1 (6 months ago) Feb 09 2022
xhtml2pdf
2,209 0.2.16 (a month ago) May 19 2011
nested-lookup
209 0.2.25 (2 years ago) Feb 09 2022

json-parser

jmespath
2,130 1.0.1 (2 years ago) Feb 09 2022
jsonpath-ng
545 1.6.1 (6 months ago) Feb 09 2022
nested-lookup
209 0.2.25 (2 years ago) Feb 09 2022

jmespath

jmespath
2,130 1.0.1 (2 years ago) Feb 09 2022

jsonpath

jsonpath-ng
545 1.6.1 (6 months ago) Feb 09 2022

browser-automation

selenium
29,780 4.22.0 (24 days ago) Apr 25 2008
playwright
11,134 1.45.0 (12 days ago) Feb 24 2021
undetected-chromedriver
9,064 3.5.5 (4 months ago) Sep 04 2020
requestium
1,823 0.4.0 (5 months ago) Dec 28 2012
selenium-driverless
411 1.9.3.1 (a month ago) Jul 22 2022
splash
4,039 3.5 (4 years ago) Apr 25 2014

playwright

playwright
11,134 1.45.0 (12 days ago) Feb 24 2021

selenium

selenium
29,780 4.22.0 (24 days ago) Apr 25 2008
undetected-chromedriver
9,064 3.5.5 (4 months ago) Sep 04 2020
requestium
1,823 0.4.0 (5 months ago) Dec 28 2012
selenium-driverless
411 1.9.3.1 (a month ago) Jul 22 2022

text-parsing

sumy
3,457 0.11.0 (1 year, 8 months ago) Oct 20 2013

scraper

youtube-dl
129,778 2021.12.17 (2 years ago) Feb 22 2012
trafilatura
3,188 1.11.0 (18 days ago) Jul 17 2019
you-get
49,559 0.4.1710 (21 days ago) Sep 01 2012

http

ua-parser
551 0.18.0 (1 year, 8 days ago) Dec 29 2012

headers

ua-parser
551 0.18.0 (1 year, 8 days ago) Dec 29 2012

data-output

xhtml2pdf
2,209 0.2.16 (a month ago) May 19 2011

cli-tool

youtube-dl
129,778 2021.12.17 (2 years ago) Feb 22 2012
you-get
49,559 0.4.1710 (21 days ago) Sep 01 2012
Was this page helpful?