Skip to content

scrapyd

3,099 8 7 BSD-3-Clause
1.6.0 (22 Jul 2025) Sep 04 2013 38.8 thousand (month)

Scrapyd is a service for running Scrapy spiders. It allows you to schedule spiders to run at regular intervals and also allows you to run spiders on remote machines. It is built in Python, and it is meant to be used in a server-client architecture, where the scrapyd server runs on a remote machine, and clients can schedule and control spider runs on the server using an HTTP API. With Scrapyd, you can schedule spider runs on a regular basis, schedule spider runs on demand, and view the status of running spiders.

You can also see the logs of completed spiders, and manage spider settings and configurations. Scrapyd also provides an API that allows you to schedule spider runs, cancel spider runs, and view the status of running spiders. You can install the package via pip by running pip install scrapyd and then you can run the package by running scrapyd command in your command prompt. By default, it will start a web server on port 6800, but you can specify a different port using the `--port`` option.

Scrapyd is a good solution if you need to run Scrapy spiders on a remote machine, or if you need to schedule spider runs on a regular basis. It's also useful if you have multiple spiders, and you need a way to manage and monitor them all in one place.

for more web interface see scrapydweb

Example Use


$ scrapyd
$ curl http://localhost:6800/schedule.json -d project=myproject -d spider=spider2

Alternatives / Similar


65,141 2.19.0 (2026-09-10 11:14:18 ago) Jul 26 2019 compare
3,411 1.6.0 (2025-02-16 13:18:50 ago) Sep 30 2018 compare
3,517 0.9.13 (2023-07-19 18:53:46 ago) Jul 04 2017 compare
201,818 4.57.0 (2026-11-12 15:19:02 ago) Apr 01 2024 compare
87,360 0.9.5 (2026-10-14 10:25:32 ago) May 01 2024 compare
102,927 0.5.1 (2026-11-04 20:29:38 ago) Aug 01 2024 compare
5,842 4.0.97 (2026-01-06 07:45:54 ago) Oct 01 2023 compare
32,783 2.3.1 (2026-10-06 08:27:56 ago) Jan 15 2024 compare
247 1.34.0 (2024-11-27 14:57:34 ago) Feb 05 2023 compare
8,090 1.1.14 (2022-07-17 17:20:09 ago) Jul 26 2019 compare
13,346 1.1.9 (2018-10-21 03:39:17 ago) Aug 24 2018 compare
1,738 0.8.5 (2022-09-06 08:54:56 ago) Oct 17 2018 compare
4,190 3.5 (2020-06-16 13:27:02 ago) Apr 25 2014 compare
428 0.1.3 (2023-08-01 20:28:33 ago) Feb 20 2022 compare

Other Languages

25,597 v2.2.0 (2025-03-27 10:47:28 ago) May 14 2018 compare
17,845 v1.8.0 (2026-10-05 13:28:18 ago) Nov 07 2022 compare
7,584 v1.4.0 (2026-03-03 03:58:32 ago) Feb 15 2020 compare
2,783 2026-11-13 (2026-11-13 06:19:33 ago) Jun 06 2019 compare
716 2026-11-10 (2026-11-10 11:36:17 ago) Feb 09 2017 compare
26,447 3.18.2 (2026-11-13 21:59:55 ago) Apr 22 2022 compare
1,525 1.0.5 (2026-07-09 06:54:00 ago) Nov 22 2014 compare
4,444 2.14.2 (2026-10-26 20:34:30 ago) Jul 25 2009 compare
2,052 (2021-05-19 15:14:49 ago) Nov 20 2016 compare
6,024 v2.0.0-alpha.62 (2026-11-02 17:33:11 ago) Jul 15 compare
3,072 v2.4.0 (2026-01-08 05:29:21 ago) Jul 17 2018 compare
6,798 2.1.1 (2026-10-24 07:58:18 ago) Sep 10 2012 compare
9,168 v4.0.3 (2023-04-01 09:05:33 ago) Dec 02 2012 compare
- 2.2.0 (2026-01-27 17:36:19 ago) Aug 23 2018 compare
836 0.7.2 (2025-02-03 07:58:27 ago) Jul 25 2009 compare
1,361 3.4.0 (2026-10-27 16:48:25 ago) Dec 27 2011 compare
1,456 v3.2.1 (2025-03-21 06:53:36 ago) Dec 27 2021 compare
165 2.3.0 (2026-07-09 06:48:00 ago) Dec 22 2019 compare
589 3.0.0 (2024-04-09 15:34:59 ago) May 04 2020 compare
215 1.0.0-beta8.4 (2023-06-29 12:37:12 ago) Apr 18 2019 compare
1,339 v0.7.7 (2026-08-04 16:21:12 ago) Mar 16 2013 compare
374 v3.5.8 (2026-05-03 06:50:54 ago) Apr 18 2022 compare
Was this page helpful?