Skip to content

crwlr-crawler

300 2 2 MIT
v1.7.2 (19 Mar 2024) Apr 18 2022 18 (month)

This library provides kind of a framework and a lot of ready to use, so-called steps, that you can use as building blocks, to build your own crawlers and scrapers with.

Some features: - Crawler Politeness innocent (respecting robots.txt, throttling,...) - Load URLs using - a (PSR-18) HTTP client (default is of course Guzzle) - or a headless browser (chrome) to get source after Javascript execution - Get absolute links from HTML documents link - Get sitemaps from robots.txt and get all URLs from those sitemaps - Crawl (load) all pages of a website spider - Use cookies (or don't) cookie - Use any HTTP methods (GET, POST,...) and send any headers or body - Iterate over paginated list pages repeat - Extract data from: - HTML and also XML (using CSS selectors or XPath queries) - JSON (using dot notation) - CSV (map columns) - Extract schema.org structured data in JSON-LD format from HTML documents - Keep memory usage low by using PHP Generators muscle - Cache HTTP responses during development, so you don't have to load pages again and again after every code change - Get logs about what your crawler is doing (accepts any PSR-3 LoggerInterface)

Example Use


<?php
require_once 'vendor/autoload.php';

use Crwlr\Crawler;

$crawler = new Crawler();
$crawler->get('https://example.com', ['User-Agent' => 'webscraping.fyi']);


// more links can be followed:
$crawler->followLinks();

// and current page can be parsed:
$response = $crawler->response();
$title = $crawler->filter('title')->text();
echo $response->getContent();
```

Alternatives / Similar


2,892 v2.1.1 (6 months ago) Jul 17 2018 compare
1,327 v3.2.0 (2 months ago) Dec 27 2021 compare
498 3.0.0 (2 months ago) May 04 2020 compare
1,325 v0.7.2 (6 months ago) Mar 16 2013 compare

Other Languages

22,347 v2.1.0 (4 years ago) May 14 2018 compare
7,533 v1.3.4 (4 years ago) Feb 15 2020 compare
2,488 2024-05-09 (a month ago) Jun 06 2019 compare
645 2024-05-11 (a month ago) Feb 09 2017 compare
51,221 2.11.2 (a month ago) Jul 26 2019 compare
1,479 1.0.4 (1 year, 9 months ago) Nov 22 2014 compare
5,641 v0.18.0 (1 year, 2 months ago) Aug 06 2019 compare
2,025 (3 years ago) Nov 20 2016 compare
6,630 1.5.0 (30 days ago) Sep 10 2012 compare
2,862 1.4.3 (8 months ago) Sep 04 2013 compare
5,994 1.1.14 (1 year, 10 months ago) Jul 26 2019 compare
793 0.7.1 (4 months ago) Jul 25 2009 compare
1,305 3.0.0 (1 year, 9 months ago) Dec 27 2011 compare
3,015 1.5.0 (4 months ago) Sep 30 2018 compare
152 2.2.4 (3 years ago) Dec 22 2019 compare
239 1.33.0 (3 months ago) Feb 05 2023 compare
3,229 0.9.13 (10 months ago) Jul 04 2017 compare
1,734 0.8.5 (1 year, 9 months ago) Oct 17 2018 compare
10,575 1.1.9 (5 years ago) Aug 24 2018 compare
198 1.0.0-beta8.4 (11 months ago) Apr 18 2019 compare
412 0.1.3 (10 months ago) Feb 20 2022 compare
Was this page helpful?