Parsing HTML
suggest changeHTML is not XML. Real pages have unclosed <p> and <li> tags, wrongly nested elements, unquoted attributes and stray end tags. XML parsers such as JAXP (see XML Parsing using the JAXP APIs) reject that input, and regular expressions break on it. The JDK has no HTML parser, so the standard choice is jsoup. It implements the WHATWG HTML5 parsing algorithm, the same one browsers use, so it builds the same tree a browser would. It then lets you query that tree with CSS selectors or XPath, change it, and write it back out.
This article is about parsing HTML you already have: a string, a file, or an HTTP response body. Downloading pages, cookies and logins are covered in Web scraping.
Add jsoup to your build:
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.23.2</version>
</dependency>
Gradle: implementation("org.jsoup:jsoup:1.23.2"). jsoup has no dependencies of its own. Every example below is a single file you can run with java -cp jsoup-1.23.2.jar Example.java. The output shown comes from JDK 25.
Parsing a string
Jsoup.parse(String) returns a Document. It never throws on bad markup. Instead it fixes the markup the way a browser would:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
public class ParseBasics {
public static void main(String[] args) {
// Broken HTML: no <html>/<head>/<body>, unclosed <p> and <li>, stray </div>
String html = """
<title>Shop</title>
<h1>Products</h1>
<p>First paragraph
<p>Second <b>bold <i>both</b> italic?</i>
<ul><li>One<li>Two</ul>
</div>
""";
Document doc = Jsoup.parse(html);
System.out.println("Title: " + doc.title());
System.out.println(doc.html());
}
}
Output:
Title: Shop
<html>
<head>
<title>Shop</title>
</head>
<body>
<h1>Products</h1>
<p>First paragraph</p>
<p>Second <b>bold <i>both</i></b><i> italic?</i></p>
<ul>
<li>One</li>
<li>Two</li>
</ul>
</body>
</html>
The parser added the <html>, <head> and <body> elements, moved <title> into <head>, closed the paragraphs and list items, dropped the stray </div>, and split the overlapping <b>/<i> into a properly nested tree.
Other ways to get a Document:
| Method | Use it for |
|---|---|
Jsoup.parse(html) |
A full page as a string |
Jsoup.parse(html, baseUri) |
Same, with a base URI for resolving relative links |
Jsoup.parseBodyFragment(html) |
A snippet such as a user comment. It goes into <body> |
Jsoup.parse(path, charset) |
A file (Path or File). Pass null as charset to detect it |
Jsoup.parse(inputStream, charset, baseUri) |
An HTTP response or other byte stream |
Jsoup.parse(html, baseUri, Parser.xmlParser()) |
XML, or HTML that must not be fixed |
Getting text, HTML and attributes
After parsing, you work with Element objects. These are the methods you will use most:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
public class Navigate {
public static void main(String[] args) {
String html = """
<div id="post" class="article featured" data-id="42">
<h2>Parsing HTML</h2>
<p>Use <a href="/docs/jsoup">jsoup</a> & CSS selectors.<br>It is easy.</p>
<img src="img/logo.png" alt="Logo">
</div>
""";
// The base URI is used to resolve relative links
Document doc = Jsoup.parse(html, "https://example.com/blog/");
Element post = doc.getElementById("post");
System.out.println("id attr: " + post.id());
System.out.println("classes: " + post.classNames());
System.out.println("hasClass: " + post.hasClass("featured"));
System.out.println("data-id: " + post.attr("data-id"));
System.out.println("dataset: " + post.dataset());
System.out.println("missing: '" + post.attr("nope") + "'");
Element p = post.selectFirst("p");
System.out.println("text(): " + p.text());
System.out.println("ownText(): " + p.ownText());
System.out.println("wholeText():" + p.wholeText());
System.out.println("html(): " + p.html());
System.out.println("outerHtml():" + post.selectFirst("h2").outerHtml());
Element a = p.selectFirst("a");
System.out.println("href: " + a.attr("href"));
System.out.println("abs:href: " + a.attr("abs:href"));
System.out.println("absUrl src: " + post.selectFirst("img").absUrl("src"));
System.out.println("children: " + post.children().size());
System.out.println("parent: " + a.parent().tagName());
System.out.println("next sib: " + post.child(0).nextElementSibling().tagName());
}
}
Output:
id attr: post
classes: [article, featured]
hasClass: true
data-id: 42
dataset: {id=42}
missing: ''
text(): Use jsoup & CSS selectors. It is easy.
ownText(): Use & CSS selectors. It is easy.
wholeText():Use jsoup & CSS selectors.
It is easy.
html(): Use <a href="/docs/jsoup">jsoup</a> & CSS selectors.
<br>
It is easy.
outerHtml():<h2>Parsing HTML</h2>
href: /docs/jsoup
abs:href: https://example.com/docs/jsoup
absUrl src: https://example.com/blog/img/logo.png
children: 3
parent: p
next sib: p
Some things to know:
text()gives the text of the element and all its descendants. Entities are decoded (&becomes&), whitespace is collapsed, and<br>becomes a space. Use this for most data extraction.ownText()skips text inside child elements. That's why "jsoup" is missing above.wholeText()keeps the original whitespace and turns<br>into a newline.html()returns the inner HTML andouterHtml()includes the element itself. Both are re-serialized from the tree, so they will not match the original source byte for byte. Pretty-printing also normalizes whitespace, which is whyParsing HTMLcame out asParsing HTML.attr(name)returns"", notnull, when the attribute is missing. UsehasAttr(name)to tell the two cases apart.absUrl("href"), or theabs:prefix inattr("abs:href"), resolves a relative URL against the document's base URI. That only works if you passed a base URI toparse(), or if the page has a<base href>tag. Without one you get"".
Selecting elements with CSS selectors
select(css) returns Elements, which extends ArrayList<Element> and adds helper methods. selectFirst(css) returns the first match or null. jsoup supports the usual CSS selectors and adds some of its own, such as :contains and :matches:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class Selectors {
public static void main(String[] args) {
String html = """
<ul id="menu">
<li><a href="/">Home</a></li>
<li class="active"><a href="/docs">Docs</a></li>
<li><a href="https://github.com/jhy/jsoup" rel="external">GitHub</a></li>
<li><a href="mailto:me@example.com">Contact</a></li>
</ul>
<div class="card"><h3>Java</h3><span class="price">$10</span></div>
<div class="card"><h3>Kotlin</h3></div>
<div class="card sale"><h3>Go</h3><span class="price">$7</span></div>
""";
Document doc = Jsoup.parse(html);
print(doc, "#menu > li.active a");
print(doc, "a[href^=http]");
print(doc, "a[href~=(?i)^mailto:]");
print(doc, "li:nth-child(odd) a");
print(doc, "li:last-child");
print(doc, "a:contains(git)");
print(doc, "div.card:has(.price) h3");
print(doc, "div.card:not(:has(.price)) h3");
print(doc, "h3:matchesOwn(^[JK])");
print(doc, ".card.sale, li.active");
// selectFirst returns null when nothing matches
Element missing = doc.selectFirst("table");
System.out.println("selectFirst(table) = " + missing);
// expectFirst throws instead
try {
doc.expectFirst("table");
} catch (IllegalArgumentException e) {
System.out.println("expectFirst: " + e.getMessage());
}
// Elements is a List<Element> with helpers
Elements links = doc.select("#menu a");
System.out.println("eachText: " + links.eachText());
System.out.println("eachAttr: " + links.eachAttr("href"));
}
static void print(Document doc, String css) {
System.out.printf("%-32s -> %s%n", css, doc.select(css).eachText());
}
}
Output:
#menu > li.active a -> [Docs]
a[href^=http] -> [GitHub]
a[href~=(?i)^mailto:] -> [Contact]
li:nth-child(odd) a -> [Home, GitHub]
li:last-child -> [Contact]
a:contains(git) -> [GitHub]
div.card:has(.price) h3 -> [Java, Go]
div.card:not(:has(.price)) h3 -> [Kotlin]
h3:matchesOwn(^[JK]) -> [Java, Kotlin]
.card.sale, li.active -> [Docs, Go $7]
selectFirst(table) = null
expectFirst: No elements matched the query 'table' in the document.
eachText: [Home, Docs, GitHub, Contact]
eachAttr: [/, /docs, https://github.com/jhy/jsoup, mailto:me@example.com]
A quick reference:
| Selector | Matches |
|---|---|
tag, #id, .class |
By tag name, id or class |
a b, a > b, a + b, a ~ b |
Descendant, child, next sibling, any later sibling |
[attr], [attr=v] |
Has the attribute, or has it with an exact value |
[attr^=v], [attr$=v], [attr*=v] |
Value starts with, ends with, or contains v |
[attr~=regex] |
Value matches a regular expression |
:nth-child(an+b), :first-child, :last-child, :eq(n) |
By position (:eq counts from 0) |
:has(sel), :not(sel), :is(sel) |
Contains a match, doesn't match, matches any of |
:contains(text), :containsOwn(text) |
Text contains the string, ignoring case |
:matches(regex), :matchesOwn(regex) |
Text matches a regular expression |
Results always come back in document order, even when you combine selectors with a comma.
selectFirst returns null when nothing matches. If a missing element means the page has changed and your code can't go on, use expectFirst. It throws an exception with a clear message, so you don't get a NullPointerException later on.
Extracting a table into records
The most common job is to turn repeating markup into Java objects. Select the rows, map each one to a record, and convert values as you go:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import java.math.BigDecimal;
import java.util.List;
public class Table {
record Book(String title, String author, BigDecimal price, String url) {}
public static void main(String[] args) {
String html = """
<table id="books">
<thead><tr><th>Title</th><th>Author</th><th>Price</th></tr></thead>
<tbody>
<tr><td><a href="/b/1">Effective Java</a></td><td>Joshua Bloch</td><td>$45.99</td></tr>
<tr><td><a href="/b/2">Java Concurrency in Practice</a></td><td>Brian Goetz</td><td>$39.50</td></tr>
<tr class="ad"><td colspan="3">Sponsored</td></tr>
<tr><td><a href="/b/3">Modern Java in Action</a></td><td>Raoul-Gabriel Urma</td><td>$49.00</td></tr>
</tbody>
</table>
""";
Document doc = Jsoup.parse(html, "https://shop.example.com/");
List<Book> books = doc.select("#books tbody tr:not(.ad)").stream()
.map(Table::toBook)
.toList();
books.forEach(System.out::println);
BigDecimal total = books.stream().map(Book::price).reduce(BigDecimal.ZERO, BigDecimal::add);
System.out.println("Total: " + total);
}
static Book toBook(Element row) {
Element link = row.expectFirst("td:eq(0) a");
String author = row.child(1).text();
String price = row.child(2).text().replace("$", "");
return new Book(link.text(), author, new BigDecimal(price), link.absUrl("href"));
}
}
Output:
Book[title=Effective Java, author=Joshua Bloch, price=45.99, url=https://shop.example.com/b/1]
Book[title=Java Concurrency in Practice, author=Brian Goetz, price=39.50, url=https://shop.example.com/b/2]
Book[title=Modern Java in Action, author=Raoul-Gabriel Urma, price=49.00, url=https://shop.example.com/b/3]
Total: 134.49
tr:not(.ad) skips the ad row. Elements is a List, so stream() and the other collection methods work on it. Use BigDecimal for prices instead of double (see BigDecimal).
Selecting with XPath
When a CSS selector can't express what you need, such as "the <dd> right after the <dt> that says License", use selectXpath:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.TextNode;
public class XPath {
public static void main(String[] args) {
String html = """
<dl>
<dt>Version</dt><dd>1.23.2</dd>
<dt>License</dt><dd>MIT</dd>
<dt>Java</dt><dd>8+</dd>
</dl>
""";
Document doc = Jsoup.parse(html);
// Find the <dd> that follows the <dt> whose text is "License"
System.out.println(doc.selectXpath("//dt[text()='License']/following-sibling::dd[1]").text());
System.out.println(doc.selectXpath("//dd[position() > 1]").eachText());
// XPath can return text nodes when you ask for a node type
for (TextNode t : doc.selectXpath("//dt/text()", TextNode.class)) {
System.out.print(t.text() + " ");
}
System.out.println();
}
}
Output:
MIT
[MIT, 8+]
Version License Java
selectXpath runs on jsoup's own tree. You don't need to convert to a W3C DOM. The overload that takes a class can return other node types, such as TextNode.
Walking the tree
select covers most needs. When you have to visit every node, for example to build a table of contents or collect all text, there are three tools:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.nodes.Node;
import org.jsoup.nodes.TextNode;
import java.util.Map;
import java.util.TreeMap;
import java.util.stream.Collectors;
public class Traverse {
public static void main(String[] args) {
String html = """
<article>
<h1>Title</h1>
<p>Intro with <a href="#a">a link</a>.</p>
<h2>Part one</h2><p>Text <em>here</em>.</p>
<h2>Part two</h2><p>More <a href="#b">links</a>.</p>
</article>
""";
Document doc = Jsoup.parse(html);
Element article = doc.expectFirst("article");
// 1. Element.stream(): this element and all descendants, depth-first
Map<String, Long> counts = article.stream()
.collect(Collectors.groupingBy(Element::tagName, TreeMap::new, Collectors.counting()));
System.out.println(counts);
// 2. NodeVisitor sees every node, including text nodes, with its depth
article.traverse((Node node, int depth) -> {
if (node instanceof Element e && e.tagName().matches("h[1-6]")) {
int level = e.tagName().charAt(1) - '0';
System.out.println(" ".repeat(level - 1) + "- " + e.text());
}
});
// 3. textNodes / nodeStream for text only
String allText = article.nodeStream(TextNode.class)
.map(TextNode::text)
.map(String::strip)
.filter(s -> !s.isEmpty())
.collect(Collectors.joining("|"));
System.out.println(allText);
}
}
Output:
{a=2, article=1, em=1, h1=1, h2=2, p=3}
- Title
- Part one
- Part two
Title|Intro with|a link|.|Part one|Text|here|.|Part two|More|links|.
Element.stream()gives the element and all its descendant elements, depth-first, as aStream<Element>.traverse(NodeVisitor)callsheadwhen it enters a node andtailwhen it leaves it.tailhas a default implementation, so a lambda works. The visitor also gets text nodes, comments and data nodes, not just elements.nodeStream(TextNode.class)streams nodes of one type only.
The difference between Node and Element matters here. An Element is a tag. A Node can also be text (TextNode), a comment (Comment) or script/style content (DataNode). children() returns only child elements, while childNodes() returns all child nodes.
Modifying the document
A parsed document can be changed and written back out. You might strip scripts, rewrite links or add attributes:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
public class Modify {
public static void main(String[] args) {
String html = """
<div id="content">
<p>Read the <a href="/guide">guide</a>.</p>
<script>track();</script>
<p class="ad">Buy now!</p>
<img src="/a.png">
</div>
""";
Document doc = Jsoup.parse(html, "https://example.com/");
doc.select("script, .ad").remove(); // delete elements
doc.select("a[href]").forEach(a -> a.attr("href", a.absUrl("href"))
.attr("rel", "nofollow"));
doc.select("img:not([alt])").attr("alt", ""); // bulk attribute set
doc.select("p").first().addClass("lead");
Element content = doc.expectFirst("#content");
content.prependElement("h1").text("Welcome & hello"); // text is escaped
content.appendElement("footer").html("<small>(c) 2026</small>");
content.select("img").wrap("<figure></figure>");
System.out.println(content.outerHtml());
// Compact, single-line output
doc.outputSettings().prettyPrint(false);
System.out.println(content.outerHtml());
}
}
Output:
<div id="content">
<h1>Welcome & hello</h1>
<p class="lead">Read the <a href="https://example.com/guide" rel="nofollow">guide</a>.</p>
<figure>
<img src="/a.png" alt="">
</figure>
<footer>
<small>(c) 2026</small>
</footer>
</div>
<div id="content"><h1>Welcome & hello</h1>
<p class="lead">Read the <a href="https://example.com/guide" rel="nofollow">guide</a>.</p>
<figure><img src="/a.png" alt=""></figure>
<footer><small>(c) 2026</small></footer></div>
Methods on Elements such as remove(), attr(k, v), addClass() and wrap() apply to every matched element. text(s) escapes its argument, while html(s) parses it as markup, so never pass untrusted input to html(). To change how the output looks, use doc.outputSettings(). prettyPrint(false) keeps the original whitespace (the blank lines are where the removed <script> and .ad paragraph were), indentAmount(n) sets the indentation, and charset(...) and escapeMode(...) control how characters are escaped.
Sanitizing untrusted HTML
If you show user-supplied HTML, such as comments or rich-text editor content, you must remove scripts, event handlers and javascript: URLs, or you have an XSS hole. Jsoup.clean parses the input as a body fragment and keeps only what a Safelist allows:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.safety.Safelist;
public class Clean {
public static void main(String[] args) {
String comment = """
<p onclick="steal()">Nice <b>post</b>!<script>alert(1)</script>
<a href="javascript:alert(2)">click</a> <a href="https://example.com" target="_blank">site</a>
<img src="x.png" onerror="alert(3)"><iframe src="evil"></iframe>
""";
// Body fragment: parsed as the content of <body>, no <html>/<head> added
Document frag = Jsoup.parseBodyFragment("<li>one<li>two");
System.out.println(frag.body().html());
System.out.println("---");
System.out.println(Jsoup.clean(comment, Safelist.basic()));
System.out.println("---");
System.out.println(Jsoup.clean(comment, Safelist.none()));
System.out.println("---");
// Customize a Safelist
Safelist custom = Safelist.basic()
.addTags("img")
.addAttributes("img", "src", "alt")
.addProtocols("img", "src", "https");
System.out.println(Jsoup.clean(comment, "https://blog.example.com/", custom));
System.out.println("---");
System.out.println("valid? " + Jsoup.isValid(comment, Safelist.basic()));
System.out.println("valid? " + Jsoup.isValid("<p>Hi <b>there</b></p>", Safelist.basic()));
}
}
Output:
<li>one</li>
<li>two</li>
---
<p>Nice <b>post</b>! <a rel="nofollow">click</a> <a href="https://example.com" rel="nofollow">site</a> </p>
---
Nice post! click site
---
<p>Nice <b>post</b>! <a rel="nofollow">click</a> <a href="https://example.com" rel="nofollow">site</a> <img src="https://blog.example.com/x.png"></p>
---
valid? false
valid? true
The built-in safelists:
| Safelist | Allows |
|---|---|
none() |
Only text. All tags are removed |
simpleText() |
b, em, i, strong, u |
basic() |
Simple formatting, lists, quotes, and links with http, https, ftp or mailto. Adds rel="nofollow" to links |
basicWithImages() |
basic() plus img with http/https sources |
relaxed() |
Also tables, headings, div, span and more. Doesn't add nofollow |
Notes:
onclick,onerror,target,<script>and<iframe>were all removed. Thejavascript:link kept its text but lost itshref.- When a safelist restricts URL protocols, relative URLs are only kept if you pass a base URI. Then they're rewritten to absolute URLs, as happened to
x.png. Without a base URI, theimgwould lose itssrc. Jsoup.isValidtells you whether input would pass throughcleanunchanged. You can use it to reject bad input instead of silently cleaning it.- Sanitize when you output HTML, not only when you accept it. Escaping text for HTML in general is covered in Text and HTML templates.
Seeing parse errors
jsoup recovers from errors without telling you. For debugging, or to check markup quality, ask the parser to record what it fixed:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.parser.Parser;
public class Errors {
public static void main(String[] args) {
String html = "<p>Unclosed <b>bold<p>Next</i><div attr=\"x>";
Parser parser = Parser.htmlParser().setTrackErrors(10);
Document doc = Jsoup.parse(html, "", parser);
parser.getErrors().forEach(System.out::println);
System.out.println(doc.body().html());
}
}
Output:
<1:23>: Unexpected StartTag token [<p>] when in state [InBody]
<1:31>: Unexpected EndTag token [</i>] when in state [InBody]
<1:44>: Unexpectedly reached end of file (EOF) in input state [AttributeValue_doubleQuoted]
<1:44>: Unexpected EOF token [] when in state [InBody]
<p>Unclosed <b>bold</b></p>
<p><b>Next</b></p>
Each error has a line:column position. The <div with an unclosed attribute quote ran to the end of input and was dropped. Notice that <b> was reopened inside the second paragraph, which is what browsers do too.
HTML mode vs XML mode
The HTML parser lowercases tag names and applies HTML rules: <title> goes into <head>, and self-closing syntax like <x/> is ignored for tags that aren't void elements. For XML input, such as RSS feeds, or when you need case-sensitive names, use Parser.xmlParser():
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.parser.Parser;
public class XmlMode {
public static void main(String[] args) {
String xml = "<feed><Entry id='1'><title>A</title></Entry><Entry id='2'/></feed>";
Document asHtml = Jsoup.parse(xml);
System.out.println(asHtml.select("entry").size() + " " + asHtml.body().html().replace("\n", ""));
Document asXml = Jsoup.parse(xml, "", Parser.xmlParser());
System.out.println(asXml.select("Entry").eachAttr("id"));
System.out.println(asXml.html().replace("\n", ""));
}
}
Output:
2 <feed> <entry id="1"> <title>A</title> </entry> <entry id="2"></entry></feed>
[1, 2]
<feed><Entry id="1"><title>A</title></Entry><Entry id="2" /></feed>
The XML parser doesn't add <html>/<body>, keeps Entry capitalized, and understands <Entry id='2'/> as an empty element. Unlike a real XML parser, it still accepts malformed input. For strict, schema-aware processing use JAXP or XOM.
Parsing files and character encodings
HTML files on disk come in many encodings. Pass null as the charset and jsoup works it out: it checks for a byte order mark, then a <meta charset> or <meta http-equiv="Content-Type"> tag, and falls back to UTF-8:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import java.nio.charset.Charset;
import java.nio.file.Files;
import java.nio.file.Path;
public class FromFile {
public static void main(String[] args) throws Exception {
Path file = Files.createTempFile("page", ".html");
String html = "<html><head><meta charset=\"windows-1250\"><title>Zażółć gęślą jaźń</title></head></html>";
Files.write(file, html.getBytes(Charset.forName("windows-1250")));
// null charset: detect from BOM or <meta charset>, falling back to UTF-8
Document doc = Jsoup.parse(file, null);
System.out.println(doc.title());
System.out.println(doc.charset());
Files.delete(file);
}
}
Output (run with -Dstdout.encoding=UTF-8 so the Windows console prints Polish characters):
Zażółć gęślą jaźń
windows-1250
If you know the encoding, pass it, for example Jsoup.parse(path, "ISO-8859-1"). Reading the file into a String yourself with the wrong charset corrupts the text before jsoup sees it, so pass jsoup the bytes (Path, File or InputStream) whenever the encoding isn't known.
Parsing an HttpClient response
You don't have to use Jsoup.connect to fetch pages. If your application already uses java.net.http.HttpClient, request the body as an InputStream and give it to Jsoup.parse. Use the response URI as the base URI, because that's the URL after redirects:
import com.sun.net.httpserver.HttpServer;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import java.io.InputStream;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
public class FromHttp {
public static void main(String[] args) throws Exception {
// Local server so the example runs offline
HttpServer server = HttpServer.create(new InetSocketAddress("localhost", 0), 0);
server.createContext("/news/", ex -> {
byte[] body = """
<html><head><meta charset="utf-8"><title>News</title></head>
<body><a href="today.html">Today</a> <a href="../archive/">Archive</a></body></html>
""".getBytes(StandardCharsets.UTF_8);
ex.getResponseHeaders().set("Content-Type", "text/html; charset=utf-8");
ex.sendResponseHeaders(200, body.length);
ex.getResponseBody().write(body);
ex.close();
});
server.start();
URI uri = URI.create("http://localhost:" + server.getAddress().getPort() + "/news/");
HttpClient client = HttpClient.newHttpClient();
HttpResponse<InputStream> resp = client.send(
HttpRequest.newBuilder(uri).build(), HttpResponse.BodyHandlers.ofInputStream());
// Parse the byte stream; null charset = sniff from BOM / <meta>, default UTF-8.
// Pass the final URI as base so relative links resolve correctly.
try (InputStream in = resp.body()) {
Document doc = Jsoup.parse(in, null, resp.uri().toString());
System.out.println(doc.title());
doc.select("a[href]").forEach(a ->
System.out.println(a.text() + " -> " + a.absUrl("href")));
}
server.stop(0);
}
}
Output (the port changes from run to run):
News
Today -> http://localhost:57996/news/today.html
Archive -> http://localhost:57996/archive/
Streaming large documents
Jsoup.parse builds the whole tree in memory, which is fine for typical pages of a few MB. For very large files, such as data dumps or huge generated reports, StreamParser (jsoup 1.17+) parses incrementally. It hands you each element as soon as it's complete, and you can remove it once you've processed it:
import org.jsoup.nodes.Element;
import org.jsoup.parser.Parser;
import org.jsoup.parser.StreamParser;
import java.io.IOException;
import java.io.StringReader;
public class Streaming {
public static void main(String[] args) throws IOException {
// Imagine this is a multi-gigabyte file read with Files.newBufferedReader(path)
StringBuilder sb = new StringBuilder("<html><body><table>");
for (int i = 1; i <= 100_000; i++) {
sb.append("<tr class=row><td>").append(i).append("</td><td>item ").append(i).append("</td></tr>");
}
sb.append("</table></body></html>");
long sum = 0;
int rows = 0;
try (StreamParser streamer = new StreamParser(Parser.htmlParser())) {
streamer.parse(new StringReader(sb.toString()), "");
Element row;
while ((row = streamer.selectNext("tr.row")) != null) {
sum += Long.parseLong(row.child(0).text());
rows++;
row.remove(); // drop the processed row so memory stays flat
}
}
System.out.println("rows=" + rows + " sum=" + sum);
}
}
Output:
rows=100000 sum=5000050000
selectNext(css) returns the next complete element that matches, or null at the end. Calling row.remove() after each row keeps memory use flat no matter how big the input is. StreamParser also works with Parser.xmlParser(). Its stream() and iterator() methods give you every element as it is completed.
Tips
- Parse, don't regex. HTML is not a regular language. A regex that works on today's page will break on tomorrow's. Use selectors.
- Always pass a base URI if you need links. Otherwise
absUrlreturns an empty string. - Prefer stable hooks. Select on
id,data-*attributes or meaningful class names rather than long positional paths likediv > div:eq(3) > span. - Use
expectFirstfor required elements so that a change in page structure fails fast with a clear message. - jsoup doesn't run JavaScript. If the content is rendered on the client, you'll only see the empty shell. Look for the JSON API the page calls, or use a browser automation tool such as Playwright.
- Documents are not thread-safe. Parsing is cheap, so parse one
Documentper thread or per task instead of sharing one.