Python Libraries Used in Web Scraping

seo

advertools
1,062 0.14.2 (2 months ago) Dec 28 2012

http-client

requests
51,415 2.31.0 (11 months ago) Feb 14 2011
aiohttp
14,607 3.9.5 (25 days ago) Jul 26 2019
httpx
12,422 0.27.0 (2 months ago) Jul 26 2019
pycurl
1,041 7.45.3 (2 months ago) Feb 25 2003
treq
583 23.11.0 (6 months ago) Dec 28 2012

framework

scrapy
51,043 2.11.1 (2 months ago) Jul 26 2019
scrapyd
2,857 1.4.3 (7 months ago) Sep 04 2013
autoscraper
5,964 1.1.14 (1 year, 9 months ago) Jul 26 2019
scrapydweb
3,012 1.5.0 (2 months ago) Sep 30 2018
gerapy
3,219 0.9.13 (9 months ago) Jul 04 2017
gracy
235 1.33.0 (2 months ago) Feb 05 2023
ruia
1,733 0.8.5 (1 year, 8 months ago) Oct 17 2018
photon
10,534 1.1.9 (5 years ago) Aug 24 2018
dude
412 0.1.3 (9 months ago) Feb 20 2022

html-parser

beautifulsoup
- 4.12.3 (3 months ago) Jul 26 2019
lxml
2,580 5.2.1 (a month ago) Dec 13 2022
xmltodict
5,389 0.13.0 (2 years ago) Jul 30 2007
html5lib
1,095 1.1 (3 years ago) Jul 30 2007
cssselect
281 1.2.0 (1 year, 6 months ago) Apr 14 2012
pyquery
2,272 2.0.0 (1 year, 4 months ago) Dec 05 2008
feedparser
1,845 6.0.11 (5 months ago) Jun 15 2007
parsel
1,073 1.9.1 (a month ago) Jul 26 2019
requests-html
13,599 0.10.0 (5 years ago) Feb 25 2018
html5-parser
663 0.4.12 (5 months ago) Jun 03 2007
untangle
607 1.2.1 (1 year, 10 months ago) Jun 09 2011
selectolax
974 0.3.21 (2 months ago) Mar 01 2018
chompjs
177 1.2.3 (3 months ago) Jul 30 2007
gazpacho
732 1.1 (3 years ago) Dec 28 2012
chopper
22 0.6.0 (1 year, 16 days ago) Jul 24 2014

html-utils

bleach
2,619 6.1.0 (7 months ago) Feb 21 2010

captcha

captcha_solver
233 0.1.5 (5 years ago) Feb 17 2015
unicaps
211 1.3.0 (2 months ago) Jul 27 2020

blocking

ua-parser
544 0.18.0 (10 months ago) Dec 29 2012
cloudscraper
4,004 1.2.71 (1 year, 17 days ago) Dec 28 2012

educational

cloudscraper
4,004 1.2.71 (1 year, 17 days ago) Dec 28 2012
youtube-dl
128,778 2021.12.17 (2 years ago) Feb 22 2012
you-get
49,326 0.4.1650 (1 year, 4 months ago) Sep 01 2012

scrapers

crosslinked
1,156 0.3.0 (10 months ago) Feb 14 2011

css

cssutils
69 2.10.2 (a month ago) Aug 04 2007

html-extractor

html2text
1,595 2024.2.26 (2 months ago) Dec 14 2008
newspaper
13,679 0.2.8 (5 years ago) Dec 28 2012
trafilatura
2,911 1.9.0 (9 days ago) Jul 17 2019
readability
2,572 0.8.1 (3 years ago) Jun 30 2011
sumy
3,429 0.11.0 (1 year, 6 months ago) Oct 20 2013
extruct
823 0.16.0 (10 months ago) Oct 27 2015
extractnet
172 2.0.7 (1 year, 6 months ago) Dec 11 2020
photon
10,534 1.1.9 (5 years ago) Aug 24 2018

microformats

extruct
823 0.16.0 (10 months ago) Oct 27 2015

gui

scrapydweb
3,012 1.5.0 (2 months ago) Sep 30 2018
gerapy
3,219 0.9.13 (9 months ago) Jul 04 2017

scrapy

scrapy
51,043 2.11.1 (2 months ago) Jul 26 2019
scrapyd
2,857 1.4.3 (7 months ago) Sep 04 2013
scrapydweb
3,012 1.5.0 (2 months ago) Sep 30 2018
gerapy
3,219 0.9.13 (9 months ago) Jul 04 2017
splash
4,008 3.5 (3 years ago) Apr 25 2014

http-utils

httptools
1,168 0.6.1 (6 months ago) Dec 28 2012
requests-cache
1,261 1.2.0 (2 months ago) Feb 14 2011

json-parser

jmespath
2,089 1.0.1 (1 year, 10 months ago) Feb 09 2022
jsonpath-ng
541 1.6.1 (3 months ago) Feb 09 2022
nested-lookup
209 0.2.25 (1 year, 10 months ago) Feb 09 2022

jmespath

jmespath
2,089 1.0.1 (1 year, 10 months ago) Feb 09 2022

data

jmespath
2,089 1.0.1 (1 year, 10 months ago) Feb 09 2022
jsonpath-ng
541 1.6.1 (3 months ago) Feb 09 2022
xhtml2pdf
2,185 0.2.15 (3 months ago) May 19 2011
nested-lookup
209 0.2.25 (1 year, 10 months ago) Feb 09 2022

jsonpath

jsonpath-ng
541 1.6.1 (3 months ago) Feb 09 2022

playwright

playwright
10,799 1.43.0 (a month ago) Feb 24 2021

browser-automation

selenium
29,399 4.20.0 (17 days ago) Apr 25 2008
playwright
10,799 1.43.0 (a month ago) Feb 24 2021
requestium
1,816 0.4.0 (3 months ago) Dec 28 2012
splash
4,008 3.5 (3 years ago) Apr 25 2014

selenium

selenium
29,399 4.20.0 (17 days ago) Apr 25 2008
requestium
1,816 0.4.0 (3 months ago) Dec 28 2012

text-parsing

sumy
3,429 0.11.0 (1 year, 6 months ago) Oct 20 2013

scraper

trafilatura
2,911 1.9.0 (9 days ago) Jul 17 2019
youtube-dl
128,778 2021.12.17 (2 years ago) Feb 22 2012
you-get
49,326 0.4.1650 (1 year, 4 months ago) Sep 01 2012

headers

ua-parser
544 0.18.0 (10 months ago) Dec 29 2012

http

ua-parser
544 0.18.0 (10 months ago) Dec 29 2012

data-output

xhtml2pdf
2,185 0.2.15 (3 months ago) May 19 2011

cli-tool

youtube-dl
128,778 2021.12.17 (2 years ago) Feb 22 2012
you-get
49,326 0.4.1650 (1 year, 4 months ago) Sep 01 2012