Python Libraries Used in Web Scraping

captcha

captcha_solver
227 0.1.5 (4 years ago) Feb 17 2015
unicaps
179 1.2.2 (2 months ago) Jul 27 2020

html-parser

lxml
2,484 4.9.3 (4 months ago) Dec 13 2022
beautifulsoup
- 4.12.2 (7 months ago) Jul 26 2019
xmltodict
5,269 0.13.0 (1 year, 6 months ago) Jul 30 2007
html5lib
1,072 1.1 (3 years ago) Jul 30 2007
cssselect
275 1.2.0 (1 year, 1 month ago) Apr 14 2012
feedparser
1,734 6.0.10 (1 year, 6 months ago) Jun 15 2007
pyquery
2,243 2.0.0 (11 months ago) Dec 05 2008
parsel
1,020 1.8.1 (7 months ago) Jul 26 2019
requests-html
13,416 0.10.0 (4 years ago) Feb 25 2018
html5-parser
655 0.4.12 (10 days ago) Jun 03 2007
untangle
605 1.2.1 (1 year, 4 months ago) Jun 09 2011
selectolax
895 0.3.17 (a month ago) Mar 01 2018
chompjs
159 1.2.2 (5 months ago) Jul 30 2007
gazpacho
716 1.1 (3 years ago) Dec 28 2012
chopper
22 0.6.0 (7 months ago) Jul 24 2014

html-extractor

html2text
1,518 2020.1.16 (3 years ago) Dec 14 2008
extruct
783 0.16.0 (4 months ago) Oct 27 2015
trafilatura
2,210 1.6.2 (2 months ago) Jul 17 2019
newspaper
13,294 0.2.8 (5 years ago) Dec 28 2012
readability
2,386 0.8.1 (3 years ago) Jun 30 2011
sumy
3,289 0.11.0 (1 year, 1 month ago) Oct 20 2013
photon
10,120 1.1.9 (5 years ago) Aug 24 2018
extractnet
116 2.0.7 (1 year, 23 days ago) Dec 11 2020

text-parsing

sumy
3,289 0.11.0 (1 year, 1 month ago) Oct 20 2013

http-client

requests
50,569 2.31.0 (6 months ago) Feb 14 2011
aiohttp
14,116 3.9.0 (11 days ago) Jul 26 2019
httpx
11,446 0.25.1 (26 days ago) Jul 26 2019
pycurl
1,019 7.45.2 (11 months ago) Feb 25 2003
treq
565 23.11.0 (25 days ago) Dec 28 2012

data-output

xhtml2pdf
2,102 0.2.13 (20 days ago) May 19 2011

data

jmespath
1,972 1.0.1 (1 year, 5 months ago) Feb 09 2022
jsonpath-ng
477 1.6.0 (2 months ago) Feb 09 2022
nested-lookup
207 0.2.25 (1 year, 4 months ago) Feb 09 2022
xhtml2pdf
2,102 0.2.13 (20 days ago) May 19 2011

framework

scrapy
49,364 2.11.0 (2 months ago) Jul 26 2019
scrapyd
2,734 1.4.3 (2 months ago) Sep 04 2013
autoscraper
5,654 1.1.14 (1 year, 4 months ago) Jul 26 2019
ruia
1,701 0.8.5 (1 year, 2 months ago) Oct 17 2018
scrapydweb
2,864 1.4.1 (2 months ago) Sep 30 2018
gerapy
3,118 0.9.13 (4 months ago) Jul 04 2017
photon
10,120 1.1.9 (5 years ago) Aug 24 2018
gracy
224 1.27.1 (3 months ago) Feb 05 2023
dude
386 0.1.3 (3 months ago) Feb 20 2022

educational

youtube-dl
124,737 2021.12.17 (1 year, 11 months ago) Feb 22 2012
cloudscraper
3,596 1.2.71 (7 months ago) Dec 28 2012
you-get
48,498 0.4.1650 (11 months ago) Sep 01 2012

cli-tool

youtube-dl
124,737 2021.12.17 (1 year, 11 months ago) Feb 22 2012
you-get
48,498 0.4.1650 (11 months ago) Sep 01 2012

scraper

youtube-dl
124,737 2021.12.17 (1 year, 11 months ago) Feb 22 2012
trafilatura
2,210 1.6.2 (2 months ago) Jul 17 2019
you-get
48,498 0.4.1650 (11 months ago) Sep 01 2012

blocking

ua-parser
518 0.18.0 (4 months ago) Dec 29 2012
cloudscraper
3,596 1.2.71 (7 months ago) Dec 28 2012

gui

scrapydweb
2,864 1.4.1 (2 months ago) Sep 30 2018
gerapy
3,118 0.9.13 (4 months ago) Jul 04 2017

scrapy

scrapy
49,364 2.11.0 (2 months ago) Jul 26 2019
scrapyd
2,734 1.4.3 (2 months ago) Sep 04 2013
scrapydweb
2,864 1.4.1 (2 months ago) Sep 30 2018
splash
3,944 3.5 (3 years ago) Apr 25 2014
gerapy
3,118 0.9.13 (4 months ago) Jul 04 2017

seo

advertools
948 0.13.5 (3 months ago) Dec 28 2012

http-utils

httptools
1,142 0.6.1 (a month ago) Dec 28 2012
requests-cache
1,183 1.1.1 (10 days ago) Feb 14 2011

css

cssutils
63 2.9.0 (a month ago) Aug 04 2007

microformats

extruct
783 0.16.0 (4 months ago) Oct 27 2015

html-utils

bleach
2,563 6.1.0 (a month ago) Feb 21 2010

browser-automation

selenium
28,273 4.15.2 (25 days ago) Apr 25 2008
playwright
9,689 1.40.0 (7 days ago) Feb 24 2021
requestium
1,755 0.2.1 (6 months ago) Dec 28 2012
splash
3,944 3.5 (3 years ago) Apr 25 2014

jsonpath

jsonpath-ng
477 1.6.0 (2 months ago) Feb 09 2022

json-parser

jmespath
1,972 1.0.1 (1 year, 5 months ago) Feb 09 2022
jsonpath-ng
477 1.6.0 (2 months ago) Feb 09 2022
nested-lookup
207 0.2.25 (1 year, 4 months ago) Feb 09 2022

selenium

selenium
28,273 4.15.2 (25 days ago) Apr 25 2008
requestium
1,755 0.2.1 (6 months ago) Dec 28 2012

http

ua-parser
518 0.18.0 (4 months ago) Dec 29 2012

headers

ua-parser
518 0.18.0 (4 months ago) Dec 29 2012

jmespath

jmespath
1,972 1.0.1 (1 year, 5 months ago) Feb 09 2022

scrapers

crosslinked
1,029 0.3.0 (5 months ago) Feb 14 2011

playwright

playwright
9,689 1.40.0 (7 days ago) Feb 24 2021