Web scraping

suggest change

Web scraping means downloading web pages with a program and pulling structured data out of their HTML. A scraper does three things in a loop:

  1. Fetch a page over HTTP (cURL or file_get_contents()).
  2. Parse the HTML into a DOM tree (Dom\HTMLDocument in PHP 8.4+, DOMDocument before that).
  3. Extract data with CSS selectors or XPath, then find the next URLs to visit.

Everything in this article uses only bundled extensions (curl, dom). The examples need PHP 8.4 or later for the HTML5 parser, and PHP 8.5 for the built-in Uri\WhatWg\Url class.

A local site to practice on

Real websites change their markup all the time, so the examples scrape a tiny fake bookshop served by the PHP built-in server. Save this as server.php:

<?php
// A tiny fake bookshop to scrape. Run: php -S 127.0.0.1:8080 server.php
$books = [
    1 => ['Dune', 'Frank Herbert', 9.99, true],
    2 => ['Neuromancer', 'William Gibson', 7.50, true],
    3 => ['Foundation', 'Isaac Asimov', 8.25, false],
    4 => ['Hyperion', 'Dan Simmons', 10.00, true],
    5 => ['Snow Crash', 'Neal Stephenson', 6.75, true],
];
$path = parse_url($_SERVER['REQUEST_URI'], PHP_URL_PATH);

if ($path === '/robots.txt') {
    header('Content-Type: text/plain');
    echo "User-agent: *\nDisallow: /admin/\nCrawl-delay: 1\n";
    return;
}

if ($path === '/books') {
    $page = max(1, (int) ($_GET['page'] ?? 1));
    $ids = array_slice(array_keys($books), ($page - 1) * 2, 2);
    echo "<!DOCTYPE html><html><head><title>Books - page $page</title></head><body>\n";
    echo "<ul class=\"books\">\n";
    foreach ($ids as $id) {
        [$title, $author, $price, $inStock] = $books[$id];
        $stock = $inStock ? 'In stock' : 'Sold out';
        $price = number_format($price, 2);
        // unclosed <li> and an unquoted attribute, like on many real sites
        echo <<<HTML
        <li class="book" data-id="$id"><a href="book/$id">$title</a>
          <span class=author>$author</span>
          <span class="price">\$$price</span> <em class="stock">$stock</em>

        HTML;
    }
    echo "</ul>\n";
    if ($page * 2 < count($books)) {
        printf("<a class=\"next\" href=\"?page=%d\">Next &raquo;</a>\n", $page + 1);
    }
    echo "</body></html>\n";
    return;
}

if (preg_match('#^/book/(\d+)$#', $path, $m) && isset($books[(int) $m[1]])) {
    usleep(300_000); // pretend this page is slow
    [$title, $author, $price, $inStock] = $books[(int) $m[1]];
    $json = json_encode(['@context' => 'https://schema.org', '@type' => 'Book',
        'name' => $title, 'author' => ['@type' => 'Person', 'name' => $author],
        'offers' => ['price' => $price, 'priceCurrency' => 'USD']]);
    echo "<!DOCTYPE html><html><head><title>$title</title>\n";
    echo "<script type=\"application/ld+json\">$json</script></head>\n";
    echo "<body><h1>$title</h1><p class=\"desc\">A classic by $author.</p></body></html>\n";
    return;
}

if (in_array($path, ['/login', '/account'], true)) {
    session_start();
}

if ($path === '/login') {
    if ($_SERVER['REQUEST_METHOD'] === 'POST') {
        if (hash_equals($_SESSION['csrf'] ?? '', $_POST['csrf'] ?? '')
            && $_POST['user'] === 'demo' && $_POST['pass'] === 'secret') {
            $_SESSION['user'] = 'demo';
            header('Location: /account');
        } else {
            http_response_code(403);
            echo 'Login failed';
        }
        return;
    }
    $_SESSION['csrf'] = bin2hex(random_bytes(16));
    echo "<form method=post action=/login>\n";
    echo "<input type=hidden name=csrf value=\"{$_SESSION['csrf']}\">\n";
    echo "<input name=user> <input type=password name=pass> <button>Log in</button></form>\n";
    return;
}

if ($path === '/account') {
    if (!isset($_SESSION['user'])) {
        header('Location: /login');
        return;
    }
    echo "<p id=\"welcome\">Welcome back, {$_SESSION['user']}! You have 3 orders.</p>\n";
    return;
}

if ($path === '/flaky') {
    // fails with 429 twice, then succeeds
    $file = sys_get_temp_dir() . '/flaky-counter';
    $n = (int) @file_get_contents($file) + 1;
    file_put_contents($file, $n % 3);
    if ($n % 3 !== 0) {
        http_response_code(429);
        header('Retry-After: 1');
        echo 'Too many requests';
        return;
    }
    echo 'OK';
    return;
}

http_response_code(404);
echo "Not found\n";

Start it in a separate terminal and leave it running:

php -S 127.0.0.1:8080 server.php

The listing page /books?page=1 returns this HTML. Note the missing </li> tags and the unquoted class=author attribute: real-world HTML is rarely valid, and the parser has to cope with that.

<!DOCTYPE html><html><head><title>Books - page 1</title></head><body>
<ul class="books">
<li class="book" data-id="1"><a href="book/1">Dune</a>
  <span class=author>Frank Herbert</span>
  <span class="price">$9.99</span> <em class="stock">In stock</em>
<li class="book" data-id="2"><a href="book/2">Neuromancer</a>
  <span class=author>William Gibson</span>
  <span class="price">$7.50</span> <em class="stock">In stock</em>
</ul>
<a class="next" href="?page=2">Next &raquo;</a>
</body></html>

Fetching pages with file_get_contents()

For simple GET requests, file_get_contents() with an HTTP stream context is enough. Always set a descriptive User-Agent and a timeout:

<?php
$context = stream_context_create([
    'http' => [
        'user_agent' => 'BookBot/1.0 (+mailto:me@example.com)',
        'timeout' => 10,
        'ignore_errors' => true, // return the body even for 4xx/5xx
    ],
]);

$html = file_get_contents('http://127.0.0.1:8080/books', false, $context);
$headers = http_get_last_response_headers();

echo $headers[0], "\n";
echo strlen($html), " bytes\n";
echo substr($html, 0, 66), "\n";

$missing = file_get_contents('http://127.0.0.1:8080/nope', false, $context);
echo http_get_last_response_headers()[0], ': ', $missing;

Output:

HTTP/1.1 200 OK
494 bytes
<!DOCTYPE html><html><head><title>Books - page 1</title></head><bo
HTTP/1.1 404 Not Found: Not found

By default a 4xx/5xx response makes file_get_contents() return false with a warning; ignore_errors returns the body anyway so you can inspect the status line. http_get_last_response_headers() (PHP 8.4+) returns the response headers of the last HTTP request; in older versions use the magic $http_response_header variable.

Fetching pages with cURL

For a real scraper use cURL. It gives you redirects with a limit, separate connect and total timeouts, compression, cookies and parallel requests. The rest of the article uses this helper, saved as fetch.php:

<?php
final class FetchError extends RuntimeException {}

/** @return array{status: int, url: string, headers: array<string, string>, body: string} */
function fetch(string $url): array
{
    $headers = [];
    $ch = curl_init($url);
    curl_setopt_array($ch, [
        CURLOPT_RETURNTRANSFER => true,   // return the body instead of printing it
        CURLOPT_FOLLOWLOCATION => true,   // follow redirects...
        CURLOPT_MAXREDIRS => 5,           // ...but not forever
        CURLOPT_CONNECTTIMEOUT => 5,
        CURLOPT_TIMEOUT => 20,
        CURLOPT_ENCODING => '',           // accept gzip/deflate/br, decode automatically
        CURLOPT_USERAGENT => 'BookBot/1.0 (+mailto:me@example.com)',
        CURLOPT_HTTPHEADER => ['Accept-Language: en'],
        // collect response headers; after a redirect only the last response's survive
        CURLOPT_HEADERFUNCTION => function ($ch, string $line) use (&$headers): int {
            if (str_starts_with($line, 'HTTP/')) {
                $headers = [];
            } elseif (str_contains($line, ':')) {
                [$name, $value] = explode(':', $line, 2);
                $headers[strtolower(trim($name))] = trim($value);
            }
            return strlen($line);
        },
    ]);
    $body = curl_exec($ch);
    if ($body === false) {
        throw new FetchError(curl_error($ch) . " ($url)");
    }
    return [
        'status' => curl_getinfo($ch, CURLINFO_RESPONSE_CODE),
        'url' => curl_getinfo($ch, CURLINFO_EFFECTIVE_URL), // final URL after redirects
        'headers' => $headers,
        'body' => $body,
    ];
}

curl_exec() returns false only for transport errors (DNS failure, refused connection, timeout). An HTTP 404 or 500 is a successful transfer, so always check the status code yourself:

<?php
require __DIR__ . '/fetch.php';

$res = fetch('http://127.0.0.1:8080/account'); // redirects to /login
echo "{$res['status']} {$res['url']}\n";
echo 'content-type: ', $res['headers']['content-type'], "\n";

$res = fetch('http://127.0.0.1:8080/missing');
echo "{$res['status']} {$res['url']}\n";

try {
    fetch('http://127.0.0.1:9/');
} catch (FetchError $e) {
    echo $e->getMessage(), "\n";
}

Output:

200 http://127.0.0.1:8080/login
content-type: text/html; charset=UTF-8
404 http://127.0.0.1:8080/missing
Failed to connect to 127.0.0.1:9 after 0 ms: Could not connect to server (http://127.0.0.1:9/)

Since PHP 8.0 cURL handles are CurlHandle objects that are freed automatically when they go out of scope, so curl_close() isn't needed (it's deprecated as of PHP 8.5).

Parsing HTML with Dom\HTMLDocument

PHP 8.4 added Dom\HTMLDocument, a spec-compliant HTML5 parser (the same algorithm browsers use) with querySelector() and querySelectorAll(). It's the best choice for scraping:

<?php
require __DIR__ . '/fetch.php';

$res = fetch('http://127.0.0.1:8080/books');
$doc = Dom\HTMLDocument::createFromString($res['body'], LIBXML_NOERROR);

echo $doc->title, "\n";

foreach ($doc->querySelectorAll('li.book') as $li) {
    $link = $li->querySelector('a');
    printf(
        "#%s %-12s by %-15s %s %s -> %s\n",
        $li->getAttribute('data-id'),
        trim($link->textContent),
        $li->querySelector('.author')->textContent,
        $li->querySelector('.price')->textContent,
        $li->querySelector('.stock')->textContent,
        $link->getAttribute('href'),
    );
}

$next = $doc->querySelector('a.next');
echo 'next: ', $next?->getAttribute('href') ?? 'none', "\n";
echo 'text: ', $next->textContent, "\n";

Output:

Books - page 1
#1 Dune         by Frank Herbert   $9.99 In stock -> book/1
#2 Neuromancer  by William Gibson  $7.50 In stock -> book/2
next: ?page=2
text: Next »

Things to note:

To read a document from a file use Dom\HTMLDocument::createFromFile(). To get the HTML of an element back (for debugging), use $doc->saveHtml($element) or the $element->innerHTML property.

Why not DOMDocument?

The older DOMDocument::loadHTML() uses libxml2's HTML 4 parser. It doesn't know HTML5 elements and, without a <meta charset> in the document, it assumes ISO-8859-1 and mangles UTF-8 text:

<?php
$html = '<nav><p class="name">Café Müller</nav>';

// Legacy libxml HTML 4 parser
libxml_use_internal_errors(true);
$old = new DOMDocument();
$old->loadHTML($html);
echo 'DOMDocument:      ', $old->getElementsByTagName('p')->item(0)->textContent, "\n";
foreach (libxml_get_errors() as $error) {
    echo '  libxml: ', trim($error->message), "\n";
}
libxml_clear_errors();

// HTML5 parser (PHP 8.4+)
$new = Dom\HTMLDocument::createFromString($html, LIBXML_NOERROR);
echo 'Dom\HTMLDocument: ', $new->querySelector('p.name')->textContent, "\n";

Output:

DOMDocument:      Café Müller
  libxml: Tag nav invalid
Dom\HTMLDocument: Café Müller

If you're stuck on PHP 8.3 or older, see the section on the legacy API at the end of this article.

Extracting data with XPath

CSS selectors cover most needs, but XPath can also select by text content, navigate to parents and return numbers or strings directly. Use Dom\XPath with the new document classes. One gotcha: the HTML5 parser puts elements into the XHTML namespace, so you must register a prefix and use it in every step of the path:

<?php
require __DIR__ . '/fetch.php';

$doc = Dom\HTMLDocument::createFromString(fetch('http://127.0.0.1:8080/books?page=2')['body'], LIBXML_NOERROR);
$xpath = new Dom\XPath($doc);

// HTML elements live in the XHTML namespace, so plain //li matches nothing
echo 'without prefix: ', $xpath->evaluate('count(//li)'), "\n";

$xpath->registerNamespace('h', 'http://www.w3.org/1999/xhtml');
echo 'with prefix:    ', $xpath->evaluate('count(//h:li)'), "\n";

// titles of the books that are in stock
foreach ($xpath->query('//h:li[h:em[@class="stock"] = "In stock"]/h:a') as $a) {
    echo 'in stock: ', $a->textContent, "\n";
}

// XPath can return scalar values directly
echo 'first price: ', $xpath->evaluate('string(//h:span[@class="price"])'), "\n";

// a query relative to a context node
foreach ($xpath->query('//h:li[@class="book"]') as $li) {
    echo $li->getAttribute('data-id'), ': ', $xpath->evaluate('string(h:span[@class="author"])', $li), "\n";
}

Output:

without prefix: 0
with prefix:    2
in stock: Hyperion
first price: $8.25
3: Isaac Asimov
4: Dan Simmons

query() returns a node list; evaluate() also returns strings, numbers and booleans from XPath functions like string() and count(). The second argument is the context node for relative paths. For more XPath syntax see Using XPath.

Crawling multiple pages

To scrape all pages, follow the "next" link until there is none. Three rules keep a crawler correct and polite:

<?php
require __DIR__ . '/fetch.php';

use Uri\WhatWg\Url;

$url = 'http://127.0.0.1:8080/books';
$seen = [];
$books = [];

while ($url !== null && !isset($seen[$url])) {
    $seen[$url] = true;
    echo "GET $url\n";
    $doc = Dom\HTMLDocument::createFromString(fetch($url)['body'], LIBXML_NOERROR);
    $base = Url::parse($url);

    foreach ($doc->querySelectorAll('li.book') as $li) {
        $a = $li->querySelector('a');
        $books[] = [
            'title' => trim($a->textContent),
            'price' => (float) ltrim($li->querySelector('.price')->textContent, '$'),
            'url' => $base->resolve($a->getAttribute('href'))->toAsciiString(),
        ];
    }

    $next = $doc->querySelector('a.next')?->getAttribute('href');
    $url = $next === null ? null : $base->resolve($next)->toAsciiString();
    usleep(500_000); // be polite: pause between requests
}

foreach ($books as $b) {
    printf("%-12s %6.2f  %s\n", $b['title'], $b['price'], $b['url']);
}

$out = fopen('php://stdout', 'w');
fputcsv($out, array_keys($books[0]), escape: '');
foreach ($books as $b) {
    fputcsv($out, $b, escape: '');
}

Output:

GET http://127.0.0.1:8080/books
GET http://127.0.0.1:8080/books?page=2
GET http://127.0.0.1:8080/books?page=3
Dune           9.99  http://127.0.0.1:8080/book/1
Neuromancer    7.50  http://127.0.0.1:8080/book/2
Foundation     8.25  http://127.0.0.1:8080/book/3
Hyperion      10.00  http://127.0.0.1:8080/book/4
Snow Crash     6.75  http://127.0.0.1:8080/book/5
title,price,url
Dune,9.99,http://127.0.0.1:8080/book/1
Neuromancer,7.5,http://127.0.0.1:8080/book/2
Foundation,8.25,http://127.0.0.1:8080/book/3
Hyperion,10,http://127.0.0.1:8080/book/4
"Snow Crash",6.75,http://127.0.0.1:8080/book/5

fputcsv() writes the results as CSV; open a file with fopen('books.csv', 'w') instead of php://stdout to save them. Passing escape: '' turns off PHP's non-standard backslash escaping, which avoids a deprecation notice in PHP 8.4+ and produces RFC 4180 output.

Prices are parsed from text like $9.99. Real sites use thousands separators and different currencies, so clean such values carefully (for example with NumberFormatter::parseCurrency() from the intl extension).

Fetching pages in parallel with curl_multi

Fetching pages one by one is slow when each request takes a while. curl_multi runs several transfers at the same time. This helper keeps at most $concurrency requests in flight, so you don't overload the server. Save it as fetch_all.php:

<?php
/**
 * Fetch many URLs in parallel, at most $concurrency at a time.
 * @param list<string> $urls
 * @return array<string, string> url => body (failed requests are skipped)
 */
function fetchAll(array $urls, int $concurrency = 4): array
{
    $mh = curl_multi_init();
    $queue = $urls;
    $results = [];
    $active = 0;

    $add = function () use (&$queue, &$active, $mh, $concurrency): void {
        while ($active < $concurrency && $queue) {
            $ch = curl_init(array_shift($queue));
            curl_setopt_array($ch, [
                CURLOPT_RETURNTRANSFER => true,
                CURLOPT_TIMEOUT => 20,
                CURLOPT_USERAGENT => 'BookBot/1.0',
            ]);
            curl_multi_add_handle($mh, $ch);
            $active++;
        }
    };
    $add();

    do {
        curl_multi_exec($mh, $running);
        curl_multi_select($mh, 1.0); // wait for activity instead of busy looping
        while ($info = curl_multi_info_read($mh)) {
            $ch = $info['handle'];
            $url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
            if ($info['result'] === CURLE_OK && curl_getinfo($ch, CURLINFO_RESPONSE_CODE) === 200) {
                $results[$url] = curl_multi_getcontent($ch);
            } else {
                fwrite(STDERR, "failed: $url\n");
            }
            curl_multi_remove_handle($mh, $ch);
            $active--;
            $add();
        }
    } while ($running > 0 || $active > 0);

    return $results;
}

Many sites embed structured data as JSON-LD in a <script type="application/ld+json"> tag. When it's there, reading it is far more robust than picking text out of the markup:

<?php
require __DIR__ . '/fetch_all.php';

$urls = array_map(fn(int $i) => "http://127.0.0.1:8080/book/$i", range(1, 5));

$pages = fetchAll($urls, concurrency: 3);
echo "fetched ", count($pages), " pages

";

foreach ($pages as $url => $html) {
    $doc = Dom\HTMLDocument::createFromString($html, LIBXML_NOERROR);
    $script = $doc->querySelector('script[type="application/ld+json"]');
    $data = json_decode($script->textContent, true, flags: JSON_THROW_ON_ERROR);
    printf("%-12s %-16s %s %s\n", $data['name'], $data['author']['name'],
        $data['offers']['price'], $data['offers']['priceCurrency']);
}

Output:

fetched 5 pages

Dune         Frank Herbert    9.99 USD
Neuromancer  William Gibson   7.5 USD
Foundation   Isaac Asimov     8.25 USD
Hyperion     Dan Simmons      10 USD
Snow Crash   Neal Stephenson  6.75 USD

Results arrive in completion order, not request order, which is why they are keyed by URL. On Windows the PHP built-in server handles one request at a time, so the demo won't be faster than sequential fetching there. On Linux and macOS you can start the server with PHP_CLI_SERVER_WORKERS=4 to see the speed-up. Against a real web server, 5 requests of 300 ms each with a concurrency of 3 take about 600 ms instead of 1.5 s.

Logging in: forms, CSRF tokens and cookies

Pages behind a login need a session cookie. Reuse a single cURL handle and set CURLOPT_COOKIEFILE to an empty string to turn on the in-memory cookie engine. (Set CURLOPT_COOKIEJAR to a file name if you also want to save cookies between runs.)

Most login forms contain a hidden CSRF token, so first load the form, copy all its <input> values, fill in the credentials and post everything back:

<?php
$base = 'http://127.0.0.1:8080';

// One handle for the whole session: curl keeps cookies in memory
// once CURLOPT_COOKIEFILE is set (an empty string means "no file").
$ch = curl_init();
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_FOLLOWLOCATION => true,
    CURLOPT_COOKIEFILE => '',
    CURLOPT_USERAGENT => 'BookBot/1.0',
]);

// 1. load the form and pick up the hidden CSRF token
curl_setopt($ch, CURLOPT_URL, "$base/login");
$form = Dom\HTMLDocument::createFromString(curl_exec($ch), LIBXML_NOERROR);
$fields = [];
foreach ($form->querySelectorAll('form input[name]') as $input) {
    $fields[$input->getAttribute('name')] = $input->getAttribute('value') ?? '';
}
$fields['user'] = 'demo';
$fields['pass'] = 'secret';
echo 'posting fields: ', implode(', ', array_keys($fields)), "\n";

// 2. submit it; the server redirects to /account
curl_setopt_array($ch, [
    CURLOPT_URL => "$base/login",
    CURLOPT_POST => true,
    CURLOPT_POSTFIELDS => http_build_query($fields),
]);
$html = curl_exec($ch);
echo curl_getinfo($ch, CURLINFO_RESPONSE_CODE), ' ', curl_getinfo($ch, CURLINFO_EFFECTIVE_URL), "\n";

$page = Dom\HTMLDocument::createFromString($html, LIBXML_NOERROR);
echo $page->getElementById('welcome')->textContent, "\n";

// the cookies curl is holding (Netscape cookie format)
foreach (curl_getinfo($ch, CURLINFO_COOKIELIST) as $line) {
    $parts = explode("\t", $line);
    echo "cookie {$parts[5]} for {$parts[0]}\n";
}

Output:

posting fields: csrf, user, pass
200 http://127.0.0.1:8080/account
Welcome back, demo! You have 3 orders.
cookie PHPSESSID for 127.0.0.1

After a 302 redirect, cURL switches from POST to GET, just like a browser, so the redirect to /account works. See also Using Cookies in the cURL chapter.

Handling errors and rate limits

Servers fail and throttle. Retry on network errors, 429 Too Many Requests and 5xx responses with an increasing delay, and respect the Retry-After header when the server sends one. Don't retry other 4xx errors: a 404 won't fix itself.

<?php
require __DIR__ . '/fetch.php';

function fetchWithRetry(string $url, int $maxAttempts = 4): array
{
    for ($attempt = 1; ; $attempt++) {
        try {
            $res = fetch($url);
            $retryable = $res['status'] === 429 || $res['status'] >= 500;
            if (!$retryable || $attempt === $maxAttempts) {
                return $res;
            }
            // honour Retry-After (in seconds) if present, otherwise back off: 1, 2, 4...
            $wait = (int) ($res['headers']['retry-after'] ?? 2 ** ($attempt - 1));
            echo "attempt $attempt: HTTP {$res['status']}, retrying in {$wait}s\n";
        } catch (FetchError $e) {
            if ($attempt === $maxAttempts) {
                throw $e;
            }
            $wait = 2 ** ($attempt - 1);
            echo "attempt $attempt: {$e->getMessage()}, retrying in {$wait}s\n";
        }
        sleep($wait);
    }
}

$res = fetchWithRetry('http://127.0.0.1:8080/flaky');
echo "HTTP {$res['status']}: {$res['body']}\n";

Output:

attempt 1: HTTP 429, retrying in 1s
attempt 2: HTTP 429, retrying in 1s
HTTP 200: OK

Respecting robots.txt

robots.txt tells crawlers which paths they shouldn't visit and how fast they may go. A minimal check that handles the User-agent: * group:

<?php
require __DIR__ . '/fetch.php';

/** Very small robots.txt check: only the "User-agent: *" group, prefix rules. */
function robotsRules(string $siteRoot): array
{
    $res = fetch(rtrim($siteRoot, '/') . '/robots.txt');
    $rules = ['disallow' => [], 'delay' => 0.0];
    if ($res['status'] !== 200) {
        return $rules; // no robots.txt: everything allowed
    }
    $applies = false;
    foreach (preg_split('/\R/', $res['body']) as $line) {
        $line = trim(preg_replace('/#.*/', '', $line));
        if (!str_contains($line, ':')) {
            continue;
        }
        [$key, $value] = array_map('trim', explode(':', $line, 2));
        $key = strtolower($key);
        if ($key === 'user-agent') {
            $applies = $value === '*';
        } elseif ($applies && $key === 'disallow' && $value !== '') {
            $rules['disallow'][] = $value;
        } elseif ($applies && $key === 'crawl-delay') {
            $rules['delay'] = (float) $value;
        }
    }
    return $rules;
}

function isAllowed(array $rules, string $path): bool
{
    foreach ($rules['disallow'] as $prefix) {
        if (str_starts_with($path, $prefix)) {
            return false;
        }
    }
    return true;
}

$rules = robotsRules('http://127.0.0.1:8080');
print_r($rules);
foreach (['/books', '/admin/users'] as $path) {
    echo $path, ': ', isAllowed($rules, $path) ? 'allowed' : 'disallowed', "\n";
}

Output:

Array
(
    [disallow] => Array
        (
            [0] => /admin/
        )

    [delay] => 1
)
/books: allowed
/admin/users: disallowed

This is deliberately simplified. Full parsing (agent-specific groups, Allow rules, * and $ wildcards) is defined in RFC 9309; for production crawlers use a library such as spatie/robots-txt from Composer.

Legacy API: DOMDocument and DOMXPath

On PHP 8.3 and older, use DOMDocument with two workarounds: suppress libxml errors and declare the encoding by prepending an XML declaration. Elements are not namespaced here, so XPath works without a prefix:

<?php
$html = file_get_contents('http://127.0.0.1:8080/books');

libxml_use_internal_errors(true);
$doc = new DOMDocument();
// the XML declaration tells libxml the input is UTF-8
$doc->loadHTML('<?xml encoding="UTF-8">' . $html);
libxml_clear_errors();

$xpath = new DOMXPath($doc);
// no namespace prefix needed with the legacy parser;
// contains(concat(...)) matches one class among several
$query = '//li[contains(concat(" ", normalize-space(@class), " "), " book ")]';
foreach ($xpath->query($query) as $li) {
    echo $xpath->evaluate('string(a)', $li), ' - ',
         $xpath->evaluate('string(span[@class="price"])', $li), "\n";
}
echo $xpath->evaluate('string(//a[@class="next"])'), "\n";

Output:

Dune - $9.99
Neuromancer - $7.50
Next »

The contains(concat(...)) expression is the XPath 1.0 idiom for "has the class book" when an element may have several classes. With Dom\HTMLDocument you'd simply write the CSS selector li.book.

Pages rendered by JavaScript

PHP doesn't execute JavaScript. If the data you see in the browser isn't in the HTML that cURL downloads, it's loaded later by a script. Before reaching for a headless browser:

If you really need a browser, symfony/panther drives headless Chrome or Firefox from PHP.

Libraries

The built-in tools go a long way, but these Composer packages are popular for larger scrapers:

Best practices

Summary

Feedback about page:

Feedback:
Optional: your email if you want me to get back to you:



Table Of Contents