Web scraping
suggest changeWeb scraping means downloading web pages with a program and pulling structured data out of their HTML. A scraper does three things in a loop:
- Fetch a page over HTTP (cURL or
file_get_contents()). - Parse the HTML into a DOM tree (
Dom\HTMLDocumentin PHP 8.4+,DOMDocumentbefore that). - Extract data with CSS selectors or XPath, then find the next URLs to visit.
Everything in this article uses only bundled extensions (curl, dom). The examples need PHP 8.4 or later for the HTML5 parser, and PHP 8.5 for the built-in Uri\WhatWg\Url class.
A local site to practice on
Real websites change their markup all the time, so the examples scrape a tiny fake bookshop served by the PHP built-in server. Save this as server.php:
<?php
// A tiny fake bookshop to scrape. Run: php -S 127.0.0.1:8080 server.php
$books = [
1 => ['Dune', 'Frank Herbert', 9.99, true],
2 => ['Neuromancer', 'William Gibson', 7.50, true],
3 => ['Foundation', 'Isaac Asimov', 8.25, false],
4 => ['Hyperion', 'Dan Simmons', 10.00, true],
5 => ['Snow Crash', 'Neal Stephenson', 6.75, true],
];
$path = parse_url($_SERVER['REQUEST_URI'], PHP_URL_PATH);
if ($path === '/robots.txt') {
header('Content-Type: text/plain');
echo "User-agent: *\nDisallow: /admin/\nCrawl-delay: 1\n";
return;
}
if ($path === '/books') {
$page = max(1, (int) ($_GET['page'] ?? 1));
$ids = array_slice(array_keys($books), ($page - 1) * 2, 2);
echo "<!DOCTYPE html><html><head><title>Books - page $page</title></head><body>\n";
echo "<ul class=\"books\">\n";
foreach ($ids as $id) {
[$title, $author, $price, $inStock] = $books[$id];
$stock = $inStock ? 'In stock' : 'Sold out';
$price = number_format($price, 2);
// unclosed <li> and an unquoted attribute, like on many real sites
echo <<<HTML
<li class="book" data-id="$id"><a href="book/$id">$title</a>
<span class=author>$author</span>
<span class="price">\$$price</span> <em class="stock">$stock</em>
HTML;
}
echo "</ul>\n";
if ($page * 2 < count($books)) {
printf("<a class=\"next\" href=\"?page=%d\">Next »</a>\n", $page + 1);
}
echo "</body></html>\n";
return;
}
if (preg_match('#^/book/(\d+)$#', $path, $m) && isset($books[(int) $m[1]])) {
usleep(300_000); // pretend this page is slow
[$title, $author, $price, $inStock] = $books[(int) $m[1]];
$json = json_encode(['@context' => 'https://schema.org', '@type' => 'Book',
'name' => $title, 'author' => ['@type' => 'Person', 'name' => $author],
'offers' => ['price' => $price, 'priceCurrency' => 'USD']]);
echo "<!DOCTYPE html><html><head><title>$title</title>\n";
echo "<script type=\"application/ld+json\">$json</script></head>\n";
echo "<body><h1>$title</h1><p class=\"desc\">A classic by $author.</p></body></html>\n";
return;
}
if (in_array($path, ['/login', '/account'], true)) {
session_start();
}
if ($path === '/login') {
if ($_SERVER['REQUEST_METHOD'] === 'POST') {
if (hash_equals($_SESSION['csrf'] ?? '', $_POST['csrf'] ?? '')
&& $_POST['user'] === 'demo' && $_POST['pass'] === 'secret') {
$_SESSION['user'] = 'demo';
header('Location: /account');
} else {
http_response_code(403);
echo 'Login failed';
}
return;
}
$_SESSION['csrf'] = bin2hex(random_bytes(16));
echo "<form method=post action=/login>\n";
echo "<input type=hidden name=csrf value=\"{$_SESSION['csrf']}\">\n";
echo "<input name=user> <input type=password name=pass> <button>Log in</button></form>\n";
return;
}
if ($path === '/account') {
if (!isset($_SESSION['user'])) {
header('Location: /login');
return;
}
echo "<p id=\"welcome\">Welcome back, {$_SESSION['user']}! You have 3 orders.</p>\n";
return;
}
if ($path === '/flaky') {
// fails with 429 twice, then succeeds
$file = sys_get_temp_dir() . '/flaky-counter';
$n = (int) @file_get_contents($file) + 1;
file_put_contents($file, $n % 3);
if ($n % 3 !== 0) {
http_response_code(429);
header('Retry-After: 1');
echo 'Too many requests';
return;
}
echo 'OK';
return;
}
http_response_code(404);
echo "Not found\n";
Start it in a separate terminal and leave it running:
php -S 127.0.0.1:8080 server.php
The listing page /books?page=1 returns this HTML. Note the missing </li> tags and the unquoted class=author attribute: real-world HTML is rarely valid, and the parser has to cope with that.
<!DOCTYPE html><html><head><title>Books - page 1</title></head><body>
<ul class="books">
<li class="book" data-id="1"><a href="book/1">Dune</a>
<span class=author>Frank Herbert</span>
<span class="price">$9.99</span> <em class="stock">In stock</em>
<li class="book" data-id="2"><a href="book/2">Neuromancer</a>
<span class=author>William Gibson</span>
<span class="price">$7.50</span> <em class="stock">In stock</em>
</ul>
<a class="next" href="?page=2">Next »</a>
</body></html>
Fetching pages with file_get_contents()
For simple GET requests, file_get_contents() with an HTTP stream context is enough. Always set a descriptive User-Agent and a timeout:
<?php
$context = stream_context_create([
'http' => [
'user_agent' => 'BookBot/1.0 (+mailto:me@example.com)',
'timeout' => 10,
'ignore_errors' => true, // return the body even for 4xx/5xx
],
]);
$html = file_get_contents('http://127.0.0.1:8080/books', false, $context);
$headers = http_get_last_response_headers();
echo $headers[0], "\n";
echo strlen($html), " bytes\n";
echo substr($html, 0, 66), "\n";
$missing = file_get_contents('http://127.0.0.1:8080/nope', false, $context);
echo http_get_last_response_headers()[0], ': ', $missing;
Output:
HTTP/1.1 200 OK
494 bytes
<!DOCTYPE html><html><head><title>Books - page 1</title></head><bo
HTTP/1.1 404 Not Found: Not found
By default a 4xx/5xx response makes file_get_contents() return false with a warning; ignore_errors returns the body anyway so you can inspect the status line. http_get_last_response_headers() (PHP 8.4+) returns the response headers of the last HTTP request; in older versions use the magic $http_response_header variable.
Fetching pages with cURL
For a real scraper use cURL. It gives you redirects with a limit, separate connect and total timeouts, compression, cookies and parallel requests. The rest of the article uses this helper, saved as fetch.php:
<?php
final class FetchError extends RuntimeException {}
/** @return array{status: int, url: string, headers: array<string, string>, body: string} */
function fetch(string $url): array
{
$headers = [];
$ch = curl_init($url);
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true, // return the body instead of printing it
CURLOPT_FOLLOWLOCATION => true, // follow redirects...
CURLOPT_MAXREDIRS => 5, // ...but not forever
CURLOPT_CONNECTTIMEOUT => 5,
CURLOPT_TIMEOUT => 20,
CURLOPT_ENCODING => '', // accept gzip/deflate/br, decode automatically
CURLOPT_USERAGENT => 'BookBot/1.0 (+mailto:me@example.com)',
CURLOPT_HTTPHEADER => ['Accept-Language: en'],
// collect response headers; after a redirect only the last response's survive
CURLOPT_HEADERFUNCTION => function ($ch, string $line) use (&$headers): int {
if (str_starts_with($line, 'HTTP/')) {
$headers = [];
} elseif (str_contains($line, ':')) {
[$name, $value] = explode(':', $line, 2);
$headers[strtolower(trim($name))] = trim($value);
}
return strlen($line);
},
]);
$body = curl_exec($ch);
if ($body === false) {
throw new FetchError(curl_error($ch) . " ($url)");
}
return [
'status' => curl_getinfo($ch, CURLINFO_RESPONSE_CODE),
'url' => curl_getinfo($ch, CURLINFO_EFFECTIVE_URL), // final URL after redirects
'headers' => $headers,
'body' => $body,
];
}
curl_exec() returns false only for transport errors (DNS failure, refused connection, timeout). An HTTP 404 or 500 is a successful transfer, so always check the status code yourself:
<?php
require __DIR__ . '/fetch.php';
$res = fetch('http://127.0.0.1:8080/account'); // redirects to /login
echo "{$res['status']} {$res['url']}\n";
echo 'content-type: ', $res['headers']['content-type'], "\n";
$res = fetch('http://127.0.0.1:8080/missing');
echo "{$res['status']} {$res['url']}\n";
try {
fetch('http://127.0.0.1:9/');
} catch (FetchError $e) {
echo $e->getMessage(), "\n";
}
Output:
200 http://127.0.0.1:8080/login
content-type: text/html; charset=UTF-8
404 http://127.0.0.1:8080/missing
Failed to connect to 127.0.0.1:9 after 0 ms: Could not connect to server (http://127.0.0.1:9/)
Since PHP 8.0 cURL handles are CurlHandle objects that are freed automatically when they go out of scope, so curl_close() isn't needed (it's deprecated as of PHP 8.5).
Parsing HTML with Dom\HTMLDocument
PHP 8.4 added Dom\HTMLDocument, a spec-compliant HTML5 parser (the same algorithm browsers use) with querySelector() and querySelectorAll(). It's the best choice for scraping:
<?php
require __DIR__ . '/fetch.php';
$res = fetch('http://127.0.0.1:8080/books');
$doc = Dom\HTMLDocument::createFromString($res['body'], LIBXML_NOERROR);
echo $doc->title, "\n";
foreach ($doc->querySelectorAll('li.book') as $li) {
$link = $li->querySelector('a');
printf(
"#%s %-12s by %-15s %s %s -> %s\n",
$li->getAttribute('data-id'),
trim($link->textContent),
$li->querySelector('.author')->textContent,
$li->querySelector('.price')->textContent,
$li->querySelector('.stock')->textContent,
$link->getAttribute('href'),
);
}
$next = $doc->querySelector('a.next');
echo 'next: ', $next?->getAttribute('href') ?? 'none', "\n";
echo 'text: ', $next->textContent, "\n";
Output:
Books - page 1
#1 Dune by Frank Herbert $9.99 In stock -> book/1
#2 Neuromancer by William Gibson $7.50 In stock -> book/2
next: ?page=2
text: Next »
Things to note:
createFromString()builds a tree the same way a browser would, so the unclosed<li>elements become separate siblings.LIBXML_NOERRORsilences parse warnings. Without it, minor issues (like a missing doctype) produce warnings for almost every real page.querySelector()returnsnullwhen nothing matches. Use the nullsafe operator?->when an element is optional, otherwise you'll get an "Attempt to read property on null" error.textContentreturns decoded text:»became». Usetrim()on it, because it includes whitespace from the source.getAttribute()returns the raw attribute value, so links are often relative (book/1,?page=2). See below for how to resolve them.
To read a document from a file use Dom\HTMLDocument::createFromFile(). To get the HTML of an element back (for debugging), use $doc->saveHtml($element) or the $element->innerHTML property.
Why not DOMDocument?
The older DOMDocument::loadHTML() uses libxml2's HTML 4 parser. It doesn't know HTML5 elements and, without a <meta charset> in the document, it assumes ISO-8859-1 and mangles UTF-8 text:
<?php
$html = '<nav><p class="name">Café Müller</nav>';
// Legacy libxml HTML 4 parser
libxml_use_internal_errors(true);
$old = new DOMDocument();
$old->loadHTML($html);
echo 'DOMDocument: ', $old->getElementsByTagName('p')->item(0)->textContent, "\n";
foreach (libxml_get_errors() as $error) {
echo ' libxml: ', trim($error->message), "\n";
}
libxml_clear_errors();
// HTML5 parser (PHP 8.4+)
$new = Dom\HTMLDocument::createFromString($html, LIBXML_NOERROR);
echo 'Dom\HTMLDocument: ', $new->querySelector('p.name')->textContent, "\n";
Output:
DOMDocument: Café Müller
libxml: Tag nav invalid
Dom\HTMLDocument: Café Müller
If you're stuck on PHP 8.3 or older, see the section on the legacy API at the end of this article.
Extracting data with XPath
CSS selectors cover most needs, but XPath can also select by text content, navigate to parents and return numbers or strings directly. Use Dom\XPath with the new document classes. One gotcha: the HTML5 parser puts elements into the XHTML namespace, so you must register a prefix and use it in every step of the path:
<?php
require __DIR__ . '/fetch.php';
$doc = Dom\HTMLDocument::createFromString(fetch('http://127.0.0.1:8080/books?page=2')['body'], LIBXML_NOERROR);
$xpath = new Dom\XPath($doc);
// HTML elements live in the XHTML namespace, so plain //li matches nothing
echo 'without prefix: ', $xpath->evaluate('count(//li)'), "\n";
$xpath->registerNamespace('h', 'http://www.w3.org/1999/xhtml');
echo 'with prefix: ', $xpath->evaluate('count(//h:li)'), "\n";
// titles of the books that are in stock
foreach ($xpath->query('//h:li[h:em[@class="stock"] = "In stock"]/h:a') as $a) {
echo 'in stock: ', $a->textContent, "\n";
}
// XPath can return scalar values directly
echo 'first price: ', $xpath->evaluate('string(//h:span[@class="price"])'), "\n";
// a query relative to a context node
foreach ($xpath->query('//h:li[@class="book"]') as $li) {
echo $li->getAttribute('data-id'), ': ', $xpath->evaluate('string(h:span[@class="author"])', $li), "\n";
}
Output:
without prefix: 0
with prefix: 2
in stock: Hyperion
first price: $8.25
3: Isaac Asimov
4: Dan Simmons
query() returns a node list; evaluate() also returns strings, numbers and booleans from XPath functions like string() and count(). The second argument is the context node for relative paths. For more XPath syntax see Using XPath.
Crawling multiple pages
To scrape all pages, follow the "next" link until there is none. Three rules keep a crawler correct and polite:
- Resolve relative links against the URL of the page they came from. PHP 8.5's
Uri\WhatWg\Url::resolve()does this the same way a browser does (?page=2relative to/books?page=1is/books?page=2). - Remember visited URLs so that a link cycle doesn't make the crawler loop forever.
- Pause between requests.
<?php
require __DIR__ . '/fetch.php';
use Uri\WhatWg\Url;
$url = 'http://127.0.0.1:8080/books';
$seen = [];
$books = [];
while ($url !== null && !isset($seen[$url])) {
$seen[$url] = true;
echo "GET $url\n";
$doc = Dom\HTMLDocument::createFromString(fetch($url)['body'], LIBXML_NOERROR);
$base = Url::parse($url);
foreach ($doc->querySelectorAll('li.book') as $li) {
$a = $li->querySelector('a');
$books[] = [
'title' => trim($a->textContent),
'price' => (float) ltrim($li->querySelector('.price')->textContent, '$'),
'url' => $base->resolve($a->getAttribute('href'))->toAsciiString(),
];
}
$next = $doc->querySelector('a.next')?->getAttribute('href');
$url = $next === null ? null : $base->resolve($next)->toAsciiString();
usleep(500_000); // be polite: pause between requests
}
foreach ($books as $b) {
printf("%-12s %6.2f %s\n", $b['title'], $b['price'], $b['url']);
}
$out = fopen('php://stdout', 'w');
fputcsv($out, array_keys($books[0]), escape: '');
foreach ($books as $b) {
fputcsv($out, $b, escape: '');
}
Output:
GET http://127.0.0.1:8080/books
GET http://127.0.0.1:8080/books?page=2
GET http://127.0.0.1:8080/books?page=3
Dune 9.99 http://127.0.0.1:8080/book/1
Neuromancer 7.50 http://127.0.0.1:8080/book/2
Foundation 8.25 http://127.0.0.1:8080/book/3
Hyperion 10.00 http://127.0.0.1:8080/book/4
Snow Crash 6.75 http://127.0.0.1:8080/book/5
title,price,url
Dune,9.99,http://127.0.0.1:8080/book/1
Neuromancer,7.5,http://127.0.0.1:8080/book/2
Foundation,8.25,http://127.0.0.1:8080/book/3
Hyperion,10,http://127.0.0.1:8080/book/4
"Snow Crash",6.75,http://127.0.0.1:8080/book/5
fputcsv() writes the results as CSV; open a file with fopen('books.csv', 'w') instead of php://stdout to save them. Passing escape: '' turns off PHP's non-standard backslash escaping, which avoids a deprecation notice in PHP 8.4+ and produces RFC 4180 output.
Prices are parsed from text like $9.99. Real sites use thousands separators and different currencies, so clean such values carefully (for example with NumberFormatter::parseCurrency() from the intl extension).
Fetching pages in parallel with curl_multi
Fetching pages one by one is slow when each request takes a while. curl_multi runs several transfers at the same time. This helper keeps at most $concurrency requests in flight, so you don't overload the server. Save it as fetch_all.php:
<?php
/**
* Fetch many URLs in parallel, at most $concurrency at a time.
* @param list<string> $urls
* @return array<string, string> url => body (failed requests are skipped)
*/
function fetchAll(array $urls, int $concurrency = 4): array
{
$mh = curl_multi_init();
$queue = $urls;
$results = [];
$active = 0;
$add = function () use (&$queue, &$active, $mh, $concurrency): void {
while ($active < $concurrency && $queue) {
$ch = curl_init(array_shift($queue));
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_TIMEOUT => 20,
CURLOPT_USERAGENT => 'BookBot/1.0',
]);
curl_multi_add_handle($mh, $ch);
$active++;
}
};
$add();
do {
curl_multi_exec($mh, $running);
curl_multi_select($mh, 1.0); // wait for activity instead of busy looping
while ($info = curl_multi_info_read($mh)) {
$ch = $info['handle'];
$url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
if ($info['result'] === CURLE_OK && curl_getinfo($ch, CURLINFO_RESPONSE_CODE) === 200) {
$results[$url] = curl_multi_getcontent($ch);
} else {
fwrite(STDERR, "failed: $url\n");
}
curl_multi_remove_handle($mh, $ch);
$active--;
$add();
}
} while ($running > 0 || $active > 0);
return $results;
}
Many sites embed structured data as JSON-LD in a <script type="application/ld+json"> tag. When it's there, reading it is far more robust than picking text out of the markup:
<?php
require __DIR__ . '/fetch_all.php';
$urls = array_map(fn(int $i) => "http://127.0.0.1:8080/book/$i", range(1, 5));
$pages = fetchAll($urls, concurrency: 3);
echo "fetched ", count($pages), " pages
";
foreach ($pages as $url => $html) {
$doc = Dom\HTMLDocument::createFromString($html, LIBXML_NOERROR);
$script = $doc->querySelector('script[type="application/ld+json"]');
$data = json_decode($script->textContent, true, flags: JSON_THROW_ON_ERROR);
printf("%-12s %-16s %s %s\n", $data['name'], $data['author']['name'],
$data['offers']['price'], $data['offers']['priceCurrency']);
}
Output:
fetched 5 pages
Dune Frank Herbert 9.99 USD
Neuromancer William Gibson 7.5 USD
Foundation Isaac Asimov 8.25 USD
Hyperion Dan Simmons 10 USD
Snow Crash Neal Stephenson 6.75 USD
Results arrive in completion order, not request order, which is why they are keyed by URL. On Windows the PHP built-in server handles one request at a time, so the demo won't be faster than sequential fetching there. On Linux and macOS you can start the server with PHP_CLI_SERVER_WORKERS=4 to see the speed-up. Against a real web server, 5 requests of 300 ms each with a concurrency of 3 take about 600 ms instead of 1.5 s.
Logging in: forms, CSRF tokens and cookies
Pages behind a login need a session cookie. Reuse a single cURL handle and set CURLOPT_COOKIEFILE to an empty string to turn on the in-memory cookie engine. (Set CURLOPT_COOKIEJAR to a file name if you also want to save cookies between runs.)
Most login forms contain a hidden CSRF token, so first load the form, copy all its <input> values, fill in the credentials and post everything back:
<?php
$base = 'http://127.0.0.1:8080';
// One handle for the whole session: curl keeps cookies in memory
// once CURLOPT_COOKIEFILE is set (an empty string means "no file").
$ch = curl_init();
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_COOKIEFILE => '',
CURLOPT_USERAGENT => 'BookBot/1.0',
]);
// 1. load the form and pick up the hidden CSRF token
curl_setopt($ch, CURLOPT_URL, "$base/login");
$form = Dom\HTMLDocument::createFromString(curl_exec($ch), LIBXML_NOERROR);
$fields = [];
foreach ($form->querySelectorAll('form input[name]') as $input) {
$fields[$input->getAttribute('name')] = $input->getAttribute('value') ?? '';
}
$fields['user'] = 'demo';
$fields['pass'] = 'secret';
echo 'posting fields: ', implode(', ', array_keys($fields)), "\n";
// 2. submit it; the server redirects to /account
curl_setopt_array($ch, [
CURLOPT_URL => "$base/login",
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => http_build_query($fields),
]);
$html = curl_exec($ch);
echo curl_getinfo($ch, CURLINFO_RESPONSE_CODE), ' ', curl_getinfo($ch, CURLINFO_EFFECTIVE_URL), "\n";
$page = Dom\HTMLDocument::createFromString($html, LIBXML_NOERROR);
echo $page->getElementById('welcome')->textContent, "\n";
// the cookies curl is holding (Netscape cookie format)
foreach (curl_getinfo($ch, CURLINFO_COOKIELIST) as $line) {
$parts = explode("\t", $line);
echo "cookie {$parts[5]} for {$parts[0]}\n";
}
Output:
posting fields: csrf, user, pass
200 http://127.0.0.1:8080/account
Welcome back, demo! You have 3 orders.
cookie PHPSESSID for 127.0.0.1
After a 302 redirect, cURL switches from POST to GET, just like a browser, so the redirect to /account works. See also Using Cookies in the cURL chapter.
Handling errors and rate limits
Servers fail and throttle. Retry on network errors, 429 Too Many Requests and 5xx responses with an increasing delay, and respect the Retry-After header when the server sends one. Don't retry other 4xx errors: a 404 won't fix itself.
<?php
require __DIR__ . '/fetch.php';
function fetchWithRetry(string $url, int $maxAttempts = 4): array
{
for ($attempt = 1; ; $attempt++) {
try {
$res = fetch($url);
$retryable = $res['status'] === 429 || $res['status'] >= 500;
if (!$retryable || $attempt === $maxAttempts) {
return $res;
}
// honour Retry-After (in seconds) if present, otherwise back off: 1, 2, 4...
$wait = (int) ($res['headers']['retry-after'] ?? 2 ** ($attempt - 1));
echo "attempt $attempt: HTTP {$res['status']}, retrying in {$wait}s\n";
} catch (FetchError $e) {
if ($attempt === $maxAttempts) {
throw $e;
}
$wait = 2 ** ($attempt - 1);
echo "attempt $attempt: {$e->getMessage()}, retrying in {$wait}s\n";
}
sleep($wait);
}
}
$res = fetchWithRetry('http://127.0.0.1:8080/flaky');
echo "HTTP {$res['status']}: {$res['body']}\n";
Output:
attempt 1: HTTP 429, retrying in 1s
attempt 2: HTTP 429, retrying in 1s
HTTP 200: OK
Respecting robots.txt
robots.txt tells crawlers which paths they shouldn't visit and how fast they may go. A minimal check that handles the User-agent: * group:
<?php
require __DIR__ . '/fetch.php';
/** Very small robots.txt check: only the "User-agent: *" group, prefix rules. */
function robotsRules(string $siteRoot): array
{
$res = fetch(rtrim($siteRoot, '/') . '/robots.txt');
$rules = ['disallow' => [], 'delay' => 0.0];
if ($res['status'] !== 200) {
return $rules; // no robots.txt: everything allowed
}
$applies = false;
foreach (preg_split('/\R/', $res['body']) as $line) {
$line = trim(preg_replace('/#.*/', '', $line));
if (!str_contains($line, ':')) {
continue;
}
[$key, $value] = array_map('trim', explode(':', $line, 2));
$key = strtolower($key);
if ($key === 'user-agent') {
$applies = $value === '*';
} elseif ($applies && $key === 'disallow' && $value !== '') {
$rules['disallow'][] = $value;
} elseif ($applies && $key === 'crawl-delay') {
$rules['delay'] = (float) $value;
}
}
return $rules;
}
function isAllowed(array $rules, string $path): bool
{
foreach ($rules['disallow'] as $prefix) {
if (str_starts_with($path, $prefix)) {
return false;
}
}
return true;
}
$rules = robotsRules('http://127.0.0.1:8080');
print_r($rules);
foreach (['/books', '/admin/users'] as $path) {
echo $path, ': ', isAllowed($rules, $path) ? 'allowed' : 'disallowed', "\n";
}
Output:
Array
(
[disallow] => Array
(
[0] => /admin/
)
[delay] => 1
)
/books: allowed
/admin/users: disallowed
This is deliberately simplified. Full parsing (agent-specific groups, Allow rules, * and $ wildcards) is defined in RFC 9309; for production crawlers use a library such as spatie/robots-txt from Composer.
Legacy API: DOMDocument and DOMXPath
On PHP 8.3 and older, use DOMDocument with two workarounds: suppress libxml errors and declare the encoding by prepending an XML declaration. Elements are not namespaced here, so XPath works without a prefix:
<?php
$html = file_get_contents('http://127.0.0.1:8080/books');
libxml_use_internal_errors(true);
$doc = new DOMDocument();
// the XML declaration tells libxml the input is UTF-8
$doc->loadHTML('<?xml encoding="UTF-8">' . $html);
libxml_clear_errors();
$xpath = new DOMXPath($doc);
// no namespace prefix needed with the legacy parser;
// contains(concat(...)) matches one class among several
$query = '//li[contains(concat(" ", normalize-space(@class), " "), " book ")]';
foreach ($xpath->query($query) as $li) {
echo $xpath->evaluate('string(a)', $li), ' - ',
$xpath->evaluate('string(span[@class="price"])', $li), "\n";
}
echo $xpath->evaluate('string(//a[@class="next"])'), "\n";
Output:
Dune - $9.99
Neuromancer - $7.50
Next »
The contains(concat(...)) expression is the XPath 1.0 idiom for "has the class book" when an element may have several classes. With Dom\HTMLDocument you'd simply write the CSS selector li.book.
Pages rendered by JavaScript
PHP doesn't execute JavaScript. If the data you see in the browser isn't in the HTML that cURL downloads, it's loaded later by a script. Before reaching for a headless browser:
- Open the browser's developer tools, Network tab, and look for the XHR/fetch request that returns the data. It's usually JSON that you can fetch directly and decode with
json_decode(), which is easier and faster than parsing HTML. - Look for data embedded in the page: JSON-LD,
<script id="__NEXT_DATA__">orwindow.__INITIAL_STATE__ = {...}blocks.
If you really need a browser, symfony/panther drives headless Chrome or Firefox from PHP.
Libraries
The built-in tools go a long way, but these Composer packages are popular for larger scrapers:
guzzlehttp/guzzleorsymfony/http-client: HTTP clients with middleware, retries and concurrent requests.symfony/dom-crawlerandsymfony/css-selector: a jQuery-like API on top of DOM, plus form handling.symfony/browser-kit(HttpBrowser): a stateful "browser" that clicks links and submits forms.
Best practices
- Check the terms and the law. A site's terms of service may forbid scraping, and personal data is covered by laws such as GDPR. Prefer an official API when one exists.
- Identify yourself with a User-Agent containing contact information, and throttle: one request per second (or the
Crawl-delay) is a reasonable default. - Cache pages on disk while developing, so that you don't hit the server every time you tweak a selector.
- Prefer stable hooks: ids,
data-*attributes and JSON-LD change less often than CSS class names and element positions. - Validate what you extract. When a selector stops matching, fail loudly instead of silently writing empty values.
- Don't parse HTML with regular expressions. Use a DOM parser; regexes are fine for cleaning the text you've already extracted.
Summary
- Fetch with cURL: set a User-Agent, timeouts, a redirect limit and
CURLOPT_ENCODING, and check the status code, becausecurl_exec()only fails on transport errors. - Parse with
Dom\HTMLDocument::createFromString($html, LIBXML_NOERROR)(PHP 8.4+) and extract withquerySelector()/querySelectorAll(),textContentandgetAttribute(). - With
Dom\XPath, register the XHTML namespace and prefix element names. - Resolve links with
Uri\WhatWg\Url::parse($pageUrl)->resolve($href)(PHP 8.5+), track visited URLs and pause between requests. - Use
curl_multiwith a concurrency limit for many pages, a shared handle withCURLOPT_COOKIEFILE => ''for logins, and retries with backoff for429and5xx. - Look for JSON-LD and JSON APIs first; they're more stable than HTML.