Skip to content

dude

411 2 24 MIT
0.1.3 (1 Aug 2023) Feb 20 2022 27 (month)

Dude (dude uncomplicated data extraction) is a very simple framework for writing web scrapers using Python decorators. The design, inspired by Flask, was to easily build a web scraper in just a few lines of code. Dude has an easy-to-learn syntax.

The simplest web scraper will look like this:

from dude import select


@select(css="a")
def get_link(element):
    return {"url": element.get_attribute("href")}

dude supports multiple parser backends: - playwright
- lxml
- parsel - beautifulsoup - pyppeteer - selenium

Example Use


from dude import select

"""
This example demonstrates how to use Parsel + async HTTPX
To access an attribute, use:
    selector.attrib["href"]
You can also access an attribute using the ::attr(name) pseudo-element, for example "a::attr(href)", then:
    selector.get()
To get the text, use ::text pseudo-element, then:
    selector.get()
"""


@select(css="a.url", priority=2)
async def result_url(selector):
    return {"url": selector.attrib["href"]}


# Option to get url using ::attr(name) pseudo-element
@select(css="a.url::attr(href)", priority=2)
async def result_url2(selector):
    return {"url2": selector.get()}


@select(css=".title::text", priority=1)
async def result_title(selector):
    return {"title": selector.get()}


@select(css=".description::text", priority=0)
async def result_description(selector):
    return {"description": selector.get()}


if __name__ == "__main__":
    import dude

    dude.run(urls=["https://dude.ron.sh"], parser="parsel")

Alternatives / Similar


50,703 2.11.1 (a month ago) Jul 26 2019 compare
2,833 1.4.3 (6 months ago) Sep 04 2013 compare
5,894 1.1.14 (1 year, 8 months ago) Jul 26 2019 compare
2,983 1.5.0 (a month ago) Sep 30 2018 compare
235 1.33.0 (a month ago) Feb 05 2023 compare
3,199 0.9.13 (8 months ago) Jul 04 2017 compare
1,731 0.8.5 (1 year, 6 months ago) Oct 17 2018 compare
10,458 1.1.9 (5 years ago) Aug 24 2018 compare

Other Languages

22,036 v2.1.0 (3 years ago) May 14 2018 compare
7,526 v1.3.4 (4 years ago) Feb 15 2020 compare
2,470 2024-04-04 (a day ago) Jun 06 2019 compare
641 2024-04-04 (a day ago) Feb 09 2017 compare
1,455 1.0.4 (1 year, 7 months ago) Nov 22 2014 compare
2,025 (2 years ago) Nov 20 2016 compare
5,606 v0.18.0 (1 year, 9 days ago) Aug 06 2019 compare
6,609 1.5.0 (3 months ago) Sep 10 2012 compare
2,878 v2.1.1 (4 months ago) Jul 17 2018 compare
1,300 3.0.0 (1 year, 7 months ago) Dec 27 2011 compare
791 0.7.1 (2 months ago) Jul 25 2009 compare
153 2.2.4 (3 years ago) Dec 22 2019 compare
1,315 v3.2.0 (2 days ago) Dec 27 2021 compare
491 2.0.0 (9 months ago) May 04 2020 compare
195 1.0.0-beta8.4 (9 months ago) Apr 18 2019 compare
1,320 v0.7.2 (3 months ago) Mar 16 2013 compare
298 v1.7.2 (17 days ago) Apr 18 2022 compare