Skip to content

scrapyd

2,857 7 31 BSD
1.4.3 (25 Sep 2023) Sep 04 2013 61.9 thousand (month)

Scrapyd is a service for running Scrapy spiders. It allows you to schedule spiders to run at regular intervals and also allows you to run spiders on remote machines. It is built in Python, and it is meant to be used in a server-client architecture, where the scrapyd server runs on a remote machine, and clients can schedule and control spider runs on the server using an HTTP API. With Scrapyd, you can schedule spider runs on a regular basis, schedule spider runs on demand, and view the status of running spiders.

You can also see the logs of completed spiders, and manage spider settings and configurations. Scrapyd also provides an API that allows you to schedule spider runs, cancel spider runs, and view the status of running spiders. You can install the package via pip by running pip install scrapyd and then you can run the package by running scrapyd command in your command prompt. By default, it will start a web server on port 6800, but you can specify a different port using the `--port`` option.

Scrapyd is a good solution if you need to run Scrapy spiders on a remote machine, or if you need to schedule spider runs on a regular basis. It's also useful if you have multiple spiders, and you need a way to manage and monitor them all in one place.

for more web interface see scrapydweb

Example Use


$ scrapyd
$ curl http://localhost:6800/schedule.json -d project=myproject -d spider=spider2

Alternatives / Similar


51,043 2.11.1 (2 months ago) Jul 26 2019 compare
3,012 1.5.0 (2 months ago) Sep 30 2018 compare
3,219 0.9.13 (9 months ago) Jul 04 2017 compare
5,964 1.1.14 (1 year, 9 months ago) Jul 26 2019 compare
235 1.33.0 (2 months ago) Feb 05 2023 compare
1,733 0.8.5 (1 year, 8 months ago) Oct 17 2018 compare
4,008 3.5 (3 years ago) Apr 25 2014 compare
10,534 1.1.9 (5 years ago) Aug 24 2018 compare
412 0.1.3 (9 months ago) Feb 20 2022 compare

Other Languages

22,276 v2.1.0 (3 years ago) May 14 2018 compare
7,533 v1.3.4 (4 years ago) Feb 15 2020 compare
2,488 2024-05-09 (3 days ago) Jun 06 2019 compare
645 2024-05-11 (16 hours ago) Feb 09 2017 compare
1,476 1.0.4 (1 year, 8 months ago) Nov 22 2014 compare
5,630 v0.18.0 (1 year, 1 month ago) Aug 06 2019 compare
2,026 (2 years ago) Nov 20 2016 compare
6,623 1.5.0 (23 hours ago) Sep 10 2012 compare
2,891 v2.1.1 (5 months ago) Jul 17 2018 compare
1,304 3.0.0 (1 year, 8 months ago) Dec 27 2011 compare
792 0.7.1 (3 months ago) Jul 25 2009 compare
152 2.2.4 (3 years ago) Dec 22 2019 compare
1,321 v3.2.0 (a month ago) Dec 27 2021 compare
493 3.0.0 (a month ago) May 04 2020 compare
197 1.0.0-beta8.4 (10 months ago) Apr 18 2019 compare
1,324 v0.7.2 (5 months ago) Mar 16 2013 compare
298 v1.7.2 (a month ago) Apr 18 2022 compare