Protocol Buffers
suggest changeProtocol Buffers (protobuf) is Google's language-neutral format for serializing structured data. You describe your messages once in a .proto file. The protoc compiler then generates Java classes (and Go, Python, C++, ... classes from the same file) that read and write a compact binary encoding.
Compared to the alternatives:
- vs JSON (see JSON in Java): messages are typically 3-10x smaller and much faster to parse, and the schema is enforced by the compiler. The downside is that the bytes are not human readable and both sides need the schema.
- vs Java serialization (see Basic Serialization in Java): protobuf works across languages, has clear rules for evolving a schema, and has no deserialization gadget attacks.
Protobuf is the payload format of gRPC. It is also widely used for files, message queues (Kafka, Pub/Sub) and caches.
Project setup
You need the runtime library protobuf-java, plus protobuf-java-util if you want JSON conversion and the Timestamps/Durations helpers. You also need the protoc compiler. Build plugins download the right protoc binary for your OS, so you don't have to install it yourself. Keep the protoc version and the runtime version the same.
Gradle (build.gradle.kts). Put .proto files in src/main/proto:
plugins {
java
id("com.google.protobuf") version "0.10.0"
}
dependencies {
implementation("com.google.protobuf:protobuf-java:4.36.2")
implementation("com.google.protobuf:protobuf-java-util:4.36.2")
}
protobuf {
protoc {
artifact = "com.google.protobuf:protoc:4.36.2"
}
}
Maven with the io.github.ascopes:protobuf-maven-plugin. Put .proto files in src/main/protobuf:
<properties>
<protobuf.version>4.36.2</protobuf.version>
</properties>
<dependencies>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${protobuf.version}</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java-util</artifactId>
<version>${protobuf.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>io.github.ascopes</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>5.1.11</version>
<configuration>
<protoc kind="binary-maven">
<version>${protobuf.version}</version>
</protoc>
</configuration>
<executions>
<execution>
<goals>
<goal>generate</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
The generated sources go to build/generated/... (Gradle) or target/generated-sources/... (Maven) and are compiled together with your code. Don't commit them.
You can also run the compiler by hand:
protoc -I src/main/proto --java_out=build/gen src/main/proto/example/person.proto
Defining messages
src/main/proto/example/person.proto:
syntax = "proto3";
package example;
import "google/protobuf/timestamp.proto";
option java_package = "com.example.proto";
option java_multiple_files = true;
option java_outer_classname = "PersonProto";
message Person {
string name = 1;
int32 id = 2;
optional string email = 3;
repeated PhoneNumber phones = 4;
map<string, string> labels = 5;
google.protobuf.Timestamp created_at = 6;
Role role = 7;
oneof contact {
string slack_handle = 8;
string pager = 9;
}
}
message PhoneNumber {
string number = 1;
PhoneType type = 2;
}
enum PhoneType {
PHONE_TYPE_UNSPECIFIED = 0;
PHONE_TYPE_MOBILE = 1;
PHONE_TYPE_HOME = 2;
PHONE_TYPE_WORK = 3;
}
enum Role {
ROLE_UNSPECIFIED = 0;
ROLE_USER = 1;
ROLE_ADMIN = 2;
}
Key points:
- The number after
=is the field number. It is what goes on the wire, not the field name. Once a message is in use, never change a field's number or type. java_packagesets the Java package, which is usually different from the protopackage. Withjava_multiple_files = trueeach message and enum becomes a top-level class (Person,PhoneNumber,Role, ...). Without it, they are nested inside the outer classPersonProto.- Fields in snake_case get camelCase Java accessors:
created_atbecomesgetCreatedAt(). - In proto3 the first enum value must be
0. Name it..._UNSPECIFIEDso that "not set" is never mistaken for a real value. Prefix values with the enum name, because enum values share a scope with their siblings in the package.
Scalar types map to Java like this:
| proto | Java | Notes |
|---|---|---|
int32, sint32, sfixed32 |
int |
sint32 encodes negative numbers efficiently |
int64, sint64, sfixed64 |
long |
|
uint32, fixed32 |
int |
Java has no unsigned types; use Integer.toUnsignedLong |
uint64, fixed64 |
long |
use Long.toUnsignedString |
float, double |
float, double |
|
bool |
boolean |
|
string |
String |
must be valid UTF-8 |
bytes |
com.google.protobuf.ByteString |
immutable byte sequence |
Building messages
Generated messages are immutable. You create them with a builder:
import com.example.proto.Person;
import com.example.proto.PhoneNumber;
import com.example.proto.PhoneType;
import com.example.proto.Role;
public class BuildPerson {
public static void main(String[] args) {
Person alice = Person.newBuilder()
.setName("Alice")
.setId(42)
.setEmail("alice@example.com")
.addPhones(PhoneNumber.newBuilder()
.setNumber("555-1234")
.setType(PhoneType.PHONE_TYPE_MOBILE))
.setRole(Role.ROLE_ADMIN)
.build();
System.out.println(alice);
System.out.println(alice.getName() + " has " + alice.getPhonesCount() + " phone(s)");
System.out.println(alice.getPhones(0).getType());
}
}
name: "Alice"
id: 42
email: "alice@example.com"
phones {
number: "555-1234"
type: PHONE_TYPE_MOBILE
}
role: ROLE_ADMIN
Alice has 1 phone(s)
PHONE_TYPE_MOBILE
toString() prints the protobuf text format. It is good for logs and debugging, but don't parse it or depend on its exact layout.
Things to know about the generated API:
- Setters reject
nullwith aNullPointerException. To "unset" a field, callclearEmail()instead. addPhonesaccepts a message or a builder.addAllPhonestakes anIterable.getPhonesList()returns an unmodifiableList<PhoneNumber>.- Messages implement
equalsandhashCodeby value, so they are safe as map keys. - To "modify" a message, copy it into a builder with
toBuilder():
Person promoted = alice.toBuilder()
.setRole(Role.ROLE_USER)
.clearPhones()
.build();
System.out.println(alice.getPhonesCount()); // 1, the original is unchanged
System.out.println(promoted.getPhonesCount()); // 0
System.out.println(alice.equals(promoted)); // false
Serializing to bytes and back
import com.example.proto.Person;
import com.example.proto.PhoneNumber;
import com.example.proto.PhoneType;
import com.example.proto.Role;
import com.google.protobuf.InvalidProtocolBufferException;
import java.util.HexFormat;
public class RoundTrip {
public static void main(String[] args) throws InvalidProtocolBufferException {
Person alice = Person.newBuilder()
.setName("Alice")
.setId(42)
.setEmail("alice@example.com")
.addPhones(PhoneNumber.newBuilder()
.setNumber("555-1234")
.setType(PhoneType.PHONE_TYPE_MOBILE))
.setRole(Role.ROLE_ADMIN)
.build();
byte[] bytes = alice.toByteArray();
System.out.println(bytes.length + " bytes");
System.out.println(HexFormat.of().formatHex(bytes));
Person decoded = Person.parseFrom(bytes);
System.out.println(decoded.getEmail());
System.out.println(decoded.equals(alice));
}
}
44 bytes
0a05416c696365102a1a11616c696365406578616d706c652e636f6d220c0a083535352d3132333410013802
alice@example.com
true
The same data as compact JSON is 132 bytes. Besides byte[], there are overloads of parseFrom for ByteString, ByteBuffer, InputStream and CodedInputStream. For output there are toByteString(), writeTo(OutputStream) and getSerializedSize().
What the bytes mean
Each field is written as a tag followed by a value. The tag is a varint holding (field_number << 3) | wire_type. The common wire types are 0 (varint) and 2 (length-delimited: strings, bytes, nested messages, packed repeated numbers).
0a 05 41 6c 69 63 65 field 1 (name), wire type 2, length 5, "Alice"
10 2a field 2 (id), wire type 0, varint 42
1a 11 61 6c ... 6d field 3 (email), length 17, "alice@example.com"
22 0c 0a 08 35 ... 34 10 01 field 4 (phones), a 12-byte nested PhoneNumber
38 02 field 7 (role), varint 2 = ROLE_ADMIN
Varints store 7 bits per byte, least significant group first. The high bit means "more bytes follow". A Person with only id = 150 encodes as 10 96 01: 150 is 1 0010110 in binary, so the low 7 bits become 0x96 (0x16 with the continuation bit set) and the remaining bit becomes 0x01.
Note what is not in the output: field names, and created_at, labels and the oneof, which were never set. In proto3, a scalar field holding its default value (0, "", false, the first enum value) is not written at all. That is one reason protobuf is so compact.
Handling invalid input
parseFrom throws the checked InvalidProtocolBufferException for malformed or truncated input:
byte[] truncated = java.util.Arrays.copyOf(bytes, bytes.length - 5);
try {
Person.parseFrom(truncated);
} catch (InvalidProtocolBufferException e) {
System.out.println(e.getClass().getSimpleName());
}
InvalidProtocolBufferException
Garbage input does not always fail. Bytes that happen to form valid tags parse "successfully" into a message with odd fields or unknown fields. Protobuf has no magic number or checksum, so if you need to detect corruption, add one at a higher level.
Default values and field presence
Reading a field that was never set returns its default value, never null:
Person empty = Person.getDefaultInstance();
System.out.println("[" + empty.getName() + "]"); // []
System.out.println(empty.getId()); // 0
System.out.println(empty.getRole()); // ROLE_UNSPECIFIED
System.out.println(empty.getCreatedAt().getSeconds()); // 0
System.out.println(empty.toByteArray().length); // 0
For a plain proto3 scalar like name or id you cannot tell "not set" apart from "set to the default". If that difference matters, mark the field optional. You then get a has method, and the field is written even when it holds the default:
Person p = Person.newBuilder().setEmail("").build();
System.out.println(p.hasEmail()); // true, explicitly set to ""
System.out.println(p.toByteArray().length); // 2: tag + zero length
System.out.println(Person.getDefaultInstance().hasEmail()); // false
Message-typed fields (created_at) and oneof members always have has methods (hasCreatedAt(), hasSlackHandle()).
Repeated fields, maps and oneof
Person bob = Person.newBuilder()
.setName("Bob")
.addPhones(PhoneNumber.newBuilder().setNumber("555-0001").setType(PhoneType.PHONE_TYPE_HOME))
.addPhones(PhoneNumber.newBuilder().setNumber("555-0002").setType(PhoneType.PHONE_TYPE_WORK))
.putLabels("region", "eu")
.putLabels("team", "infra")
.setSlackHandle("@bob")
.build();
for (PhoneNumber phone : bob.getPhonesList()) {
System.out.println(phone.getType() + " " + phone.getNumber());
}
System.out.println(bob.getLabelsMap());
System.out.println(bob.getLabelsOrDefault("team", "none"));
System.out.println(bob.getLabelsOrDefault("owner", "none"));
System.out.println(bob.containsLabels("region"));
switch (bob.getContactCase()) {
case SLACK_HANDLE -> System.out.println("Slack: " + bob.getSlackHandle());
case PAGER -> System.out.println("Pager: " + bob.getPager());
case CONTACT_NOT_SET -> System.out.println("no contact");
}
// setting another member of the oneof clears the previous one
Person paged = bob.toBuilder().setPager("4242").build();
System.out.println(paged.getContactCase() + " " + paged.hasSlackHandle());
PHONE_TYPE_HOME 555-0001
PHONE_TYPE_WORK 555-0002
{region=eu, team=infra}
infra
none
true
Slack: @bob
PAGER false
getLabelsMap()returns an unmodifiableMap. Don't rely on the order of its entries: the wire format does not guarantee any order for maps.- A
oneofgenerates an enum (ContactCase) whose constants are named after the fields, plus aCONTACT_NOT_SETconstant. Aswitchon it is the idiomatic way to handle the alternatives.
Unknown enum values
Proto3 enums are open: a newer sender can send a number your generated code does not know. Instead of failing, the getter returns UNRECOGNIZED and the raw number is kept:
Person fromFuture = Person.newBuilder().setRoleValue(7).build();
System.out.println(fromFuture.getRole()); // UNRECOGNIZED
System.out.println(fromFuture.getRoleValue()); // 7
Always have a default branch when you switch on a protobuf enum. Calling getNumber() on UNRECOGNIZED throws IllegalArgumentException.
Well-known types: Timestamp and Duration
Protobuf ships common messages such as google.protobuf.Timestamp, Duration, Struct, Any and wrappers like StringValue. Their Java classes are in protobuf-java, and the com.google.protobuf.util package in protobuf-java-util adds helpers.
A Timestamp is seconds plus nanoseconds since the Unix epoch, in UTC. Converting to and from java.time.Instant is straightforward:
import com.google.protobuf.Duration;
import com.google.protobuf.Timestamp;
import com.google.protobuf.util.Durations;
import com.google.protobuf.util.Timestamps;
import java.time.Instant;
public class WellKnownTypes {
static Timestamp toProto(Instant i) {
return Timestamp.newBuilder()
.setSeconds(i.getEpochSecond())
.setNanos(i.getNano())
.build();
}
static Instant toInstant(Timestamp ts) {
return Instant.ofEpochSecond(ts.getSeconds(), ts.getNanos());
}
public static void main(String[] args) throws Exception {
Instant created = Instant.parse("2026-10-17T12:00:00Z");
Timestamp ts = toProto(created);
System.out.println(ts.getSeconds());
System.out.println(Timestamps.toString(ts));
System.out.println(toInstant(ts).equals(created));
Timestamp parsed = Timestamps.parse("2026-10-17T12:00:00.250Z");
System.out.println(Timestamps.toMillis(parsed));
Duration timeout = Durations.fromSeconds(90);
System.out.println(Durations.toString(timeout));
System.out.println(Timestamps.toString(Timestamps.add(ts, timeout)));
}
}
1792238400
2026-10-17T12:00:00Z
true
1792238400250
90s
2026-10-17T12:01:30Z
Prefer Timestamp over an int64 created_at_millis field. The meaning is unambiguous, and the JSON form is a readable RFC 3339 string.
Converting to and from JSON
JsonFormat from protobuf-java-util implements the canonical proto3 JSON mapping. Use it for debugging endpoints, configuration files, or when talking to clients that cannot use protobuf:
import com.example.proto.Person;
import com.google.protobuf.util.JsonFormat;
import java.time.Instant;
public class ToJson {
public static void main(String[] args) throws Exception {
Person bob = Person.newBuilder()
.setName("Bob")
.putLabels("region", "eu")
.putLabels("team", "infra")
.setCreatedAt(com.google.protobuf.Timestamp.newBuilder()
.setSeconds(Instant.parse("2026-10-17T12:00:00Z").getEpochSecond()))
.setSlackHandle("@bob")
.build();
String json = JsonFormat.printer().print(bob);
System.out.println(json);
Person.Builder builder = Person.newBuilder();
JsonFormat.parser().merge(json, builder);
System.out.println(builder.build().equals(bob));
}
}
{
"name": "Bob",
"labels": {
"region": "eu",
"team": "infra"
},
"createdAt": "2026-10-17T12:00:00Z",
"slackHandle": "@bob"
}
true
The mapping rules:
- Field names become lowerCamelCase. The parser accepts both
slackHandleandslack_handle. - Enums are written by name. The parser accepts names or numbers.
int64/uint64values are written as strings, because JavaScript numbers can't hold all 64-bit values.bytesare base64.TimestampandDurationuse their string forms ("90s").- Fields with default values are omitted, as in the binary format.
Useful printer and parser options:
JsonFormat.Printer compact = JsonFormat.printer()
.omittingInsignificantWhitespace() // single line
.preservingProtoFieldNames() // created_at instead of createdAt
.alwaysPrintFieldsWithNoPresence(); // also print "id": 0, "role": "ROLE_UNSPECIFIED", ...
JsonFormat.Parser lenient = JsonFormat.parser()
.ignoringUnknownFields(); // default: unknown JSON keys are an error
By default the parser rejects unknown fields with InvalidProtocolBufferException. The binary parser does the opposite and keeps them (see below). If clients might send JSON produced from a newer schema, use ignoringUnknownFields().
Writing many messages to a stream
A protobuf message does not record its own length, so if you simply concatenate messages you can't tell where one ends. Use the delimited variants. They prefix each message with its size as a varint:
import com.example.proto.Person;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.HexFormat;
public class Delimited {
public static void main(String[] args) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
String[] names = {"a", "b", "c"};
for (int i = 0; i < names.length; i++) {
Person.newBuilder().setName(names[i]).setId(i + 1).build().writeDelimitedTo(out);
}
byte[] data = out.toByteArray();
System.out.println(HexFormat.of().formatHex(data));
try (InputStream in = new ByteArrayInputStream(data)) {
Person p;
while ((p = Person.parseDelimitedFrom(in)) != null) { // null at end of stream
System.out.println(p.getId() + " " + p.getName());
}
}
}
}
050a01611001050a01621002050a01631003
1 a
2 b
3 c
This works with FileOutputStream/FileInputStream (wrap them in buffered streams) and with sockets. It is a simple way to store a log of records in a file. Don't call parseFrom(InputStream) in a loop on a shared stream: it reads until end of stream.
Evolving a schema
Protobuf's main strength is that old and new code can exchange messages. Say version 1 of a service uses UserV1 and version 2 adds a field:
message UserV1 { string name = 1; int32 id = 2; }
message UserV2 { string name = 1; int32 id = 2; string email = 3; }
(In real code both would be the same message User in two versions of the same file. Two names let us show it in one program.)
UserV2 v2 = UserV2.newBuilder().setName("Alice").setId(42).setEmail("alice@example.com").build();
// old code reads new data: field 3 is unknown but kept
UserV1 v1 = UserV1.parseFrom(v2.toByteArray());
System.out.println(v1.getName() + " " + v1.getId());
System.out.println(v1.getUnknownFields().asMap().keySet());
// old code modifies and re-serializes: the email survives the round trip
UserV1 updated = v1.toBuilder().setId(43).build();
UserV2 back = UserV2.parseFrom(updated.toByteArray());
System.out.println(back.getId() + " " + back.getEmail());
Alice 42
[3]
43 alice@example.com
New code reading old data simply sees default values for the new fields.
Rules for compatible changes:
- Adding fields is safe. Give them new numbers.
- Removing a field is safe if you reserve its number and name, so nobody reuses them by accident and misreads old data:
message User {
reserved 3, 15 to 20;
reserved "email";
string name = 1;
int32 id = 2;
}
- Renaming a field is safe for binary data (only numbers go on the wire) but breaks JSON, which uses names.
- Never change a field's number, and don't change its type except between compatible types (
int32,int64,uint32,boolare compatible with each other, but values can be truncated). - Moving a single field into a new
oneofis safe, but moving several existing fields into one is not. Changingrepeatedto singular (or back) loses data. - Field numbers 1-15 take one byte in the tag, 16-2047 take two. Use the small numbers for frequently set fields.
Tips
- Editions. Newer
.protofiles may start withedition = "2023";(or"2024") instead ofsyntax = "proto3";. Editions replace the proto2/proto3 split with per-feature settings. In editions, fields have explicit presence by default. The generated Java API is the same. - Don't use generated classes as your domain model throughout the app. Convert at the boundaries (RPC, storage). This keeps builders,
ByteStringand wire concerns out of your business logic, and lets the schema evolve independently. - Lite runtime. On Android, use
protobuf-javaliteand generate with theliteoption. It produces smaller code, but has no reflection,JsonFormator text format. - Large messages. Protobuf is designed for messages up to a few megabytes, and the default parse limit is 2 GB. For bigger data, stream many small delimited messages instead of one giant one.
- gRPC. Add
servicedefinitions to the.protofile and theprotoc-gen-grpc-javaplugin to get client stubs and server base classes. The messages are exactly the classes described here.