Web scraping

suggest change

Web scraping involves retrieving web pages over HTTP and extracting structured data from their HTML markup. In the Java ecosystem, Jsoup is the standard library for fetching, parsing, and querying HTML documents using CSS selectors. For pages with custom HTTP workflows, Java 11's HttpClient can also be combined with HTML parsers.

Web Scraping with Jsoup

Jsoup combines an HTTP client and an HTML parser with a familiar CSS selector API modeled after jQuery.

Add Jsoup to your project:

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.23.2</version>
</dependency>

Fetching and Parsing a Web Page

Use Jsoup.connect() to send an HTTP GET request and parse the response into a Document:

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;

public class Scraper {
    public static void main(String[] args) {
        String url = "https://example.com";

        try {
            Document doc = Jsoup.connect(url)
                    .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64)")
                    .timeout(10_000)
                    .get();

            // Extract the page title
            String title = doc.title();
            System.out.println("Page Title: " + title);

            // Select all links within paragraphs
            Elements links = doc.select("p a[href]");
            for (Element link : links) {
                String linkText = link.text();
                // abs:href resolves relative URLs to absolute URLs
                String linkUrl = link.attr("abs:href");
                System.out.println(linkText + " -> " + linkUrl);
            }
        } catch (IOException e) {
            System.err.println("Scraping error: " + e.getMessage());
        }
    }
}

Extracting Elements with CSS Selectors

Jsoup supports standard CSS selectors:

Selector Meaning Example
tag Elements of type tag doc.select("h1")
.class Elements with CSS class doc.select(".article-body")
#id Element with specific ID doc.select("#content")
[attr] Elements with attribute doc.select("img[src]")
[attr=val] Elements with exact attribute value doc.select("input[type=hidden]")
parent > child Direct child element doc.select("table > tbody > tr")
ancestor descendant Any descendant element doc.select("div.card a")
// Extracting table data into structured objects
Elements rows = doc.select("table.data-table tbody tr");
for (Element row : rows) {
    Elements cols = row.select("td");
    if (cols.size() >= 3) {
        String name = cols.get(0).text().trim();
        String price = cols.get(1).text().trim();
        String stock = cols.get(2).text().trim();
        System.out.println(name + " | " + price + " | " + stock);
    }
}

Parsing Static HTML Strings

If you already fetched HTML (e.g., using java.net.http.HttpClient or from a local cache file), pass the raw HTML string directly to Jsoup.parse():

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

String html = "<div id='welcome'>Hello, <span class='name'>World</span>!</div>";
Document doc = Jsoup.parse(html);

String name = doc.select("#welcome .name").text();
System.out.println(name); // World

Handling Authentication and Cookies

When scraping pages behind a login screen:

import org.jsoup.Connection;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import java.io.IOException;
import java.util.Map;

public class AuthenticatedScraper {
    public static void main(String[] args) throws IOException {
        // Step 1: Submit login form via POST and capture session cookies
        Connection.Response loginResponse = Jsoup.connect("https://example.com/login")
                .data("username", "myuser")
                .data("password", "mypass")
                .method(Connection.Method.POST)
                .execute();

        Map<String, String> sessionCookies = loginResponse.cookies();

        // Step 2: Use captured cookies for subsequent requests
        Document dashboard = Jsoup.connect("https://example.com/dashboard")
                .cookies(sessionCookies)
                .get();

        System.out.println("Dashboard Heading: " + dashboard.select("h1").text());
    }
}

Best Practices

Feedback about page:

Feedback:
Optional: your email if you want me to get back to you:



Table Of Contents