Web scraping
suggest changeWeb scraping involves retrieving web pages over HTTP and extracting structured data from their HTML markup. In the Java ecosystem, Jsoup is the standard library for fetching, parsing, and querying HTML documents using CSS selectors. For pages with custom HTTP workflows, Java 11's HttpClient can also be combined with HTML parsers.
Web Scraping with Jsoup
Jsoup combines an HTTP client and an HTML parser with a familiar CSS selector API modeled after jQuery.
Add Jsoup to your project:
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.23.2</version>
</dependency>
Fetching and Parsing a Web Page
Use Jsoup.connect() to send an HTTP GET request and parse the response into a Document:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
public class Scraper {
public static void main(String[] args) {
String url = "https://example.com";
try {
Document doc = Jsoup.connect(url)
.userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64)")
.timeout(10_000)
.get();
// Extract the page title
String title = doc.title();
System.out.println("Page Title: " + title);
// Select all links within paragraphs
Elements links = doc.select("p a[href]");
for (Element link : links) {
String linkText = link.text();
// abs:href resolves relative URLs to absolute URLs
String linkUrl = link.attr("abs:href");
System.out.println(linkText + " -> " + linkUrl);
}
} catch (IOException e) {
System.err.println("Scraping error: " + e.getMessage());
}
}
}
Extracting Elements with CSS Selectors
Jsoup supports standard CSS selectors:
| Selector | Meaning | Example |
|---|---|---|
tag |
Elements of type tag |
doc.select("h1") |
.class |
Elements with CSS class | doc.select(".article-body") |
#id |
Element with specific ID | doc.select("#content") |
[attr] |
Elements with attribute | doc.select("img[src]") |
[attr=val] |
Elements with exact attribute value | doc.select("input[type=hidden]") |
parent > child |
Direct child element | doc.select("table > tbody > tr") |
ancestor descendant |
Any descendant element | doc.select("div.card a") |
// Extracting table data into structured objects
Elements rows = doc.select("table.data-table tbody tr");
for (Element row : rows) {
Elements cols = row.select("td");
if (cols.size() >= 3) {
String name = cols.get(0).text().trim();
String price = cols.get(1).text().trim();
String stock = cols.get(2).text().trim();
System.out.println(name + " | " + price + " | " + stock);
}
}
Parsing Static HTML Strings
If you already fetched HTML (e.g., using java.net.http.HttpClient or from a local cache file), pass the raw HTML string directly to Jsoup.parse():
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
String html = "<div id='welcome'>Hello, <span class='name'>World</span>!</div>";
Document doc = Jsoup.parse(html);
String name = doc.select("#welcome .name").text();
System.out.println(name); // World
Handling Authentication and Cookies
When scraping pages behind a login screen:
import org.jsoup.Connection;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import java.io.IOException;
import java.util.Map;
public class AuthenticatedScraper {
public static void main(String[] args) throws IOException {
// Step 1: Submit login form via POST and capture session cookies
Connection.Response loginResponse = Jsoup.connect("https://example.com/login")
.data("username", "myuser")
.data("password", "mypass")
.method(Connection.Method.POST)
.execute();
Map<String, String> sessionCookies = loginResponse.cookies();
// Step 2: Use captured cookies for subsequent requests
Document dashboard = Jsoup.connect("https://example.com/dashboard")
.cookies(sessionCookies)
.get();
System.out.println("Dashboard Heading: " + dashboard.select("h1").text());
}
}
Best Practices
- Set a descriptive User-Agent: Some servers block default Java user agent strings.
- Configure Timeouts: Always specify
.timeout(milliseconds)to avoid threads hanging indefinitely. - Respect robots.txt and Rate Limits: Introduce deliberate delays (
Thread.sleep(...)) between requests to avoid overloading target servers. - JavaScript-Rendered Pages: For Single Page Applications (SPAs) where HTML is populated via client-side JavaScript, use tools like Playwright for Java or Selenium WebDriver instead of pure HTML parsers.