Protocol Buffers

suggest change

Protocol Buffers (protobuf) is Google's language-neutral format for serializing structured data. You describe your messages once in a .proto file. The protoc compiler then generates Java classes (and Go, Python, C++, ... classes from the same file) that read and write a compact binary encoding.

Compared to the alternatives:

Protobuf is the payload format of gRPC. It is also widely used for files, message queues (Kafka, Pub/Sub) and caches.

Project setup

You need the runtime library protobuf-java, plus protobuf-java-util if you want JSON conversion and the Timestamps/Durations helpers. You also need the protoc compiler. Build plugins download the right protoc binary for your OS, so you don't have to install it yourself. Keep the protoc version and the runtime version the same.

Gradle (build.gradle.kts). Put .proto files in src/main/proto:

plugins {
    java
    id("com.google.protobuf") version "0.10.0"
}

dependencies {
    implementation("com.google.protobuf:protobuf-java:4.36.2")
    implementation("com.google.protobuf:protobuf-java-util:4.36.2")
}

protobuf {
    protoc {
        artifact = "com.google.protobuf:protoc:4.36.2"
    }
}

Maven with the io.github.ascopes:protobuf-maven-plugin. Put .proto files in src/main/protobuf:

<properties>
  <protobuf.version>4.36.2</protobuf.version>
</properties>

<dependencies>
  <dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java</artifactId>
    <version>${protobuf.version}</version>
  </dependency>
  <dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java-util</artifactId>
    <version>${protobuf.version}</version>
  </dependency>
</dependencies>

<build>
  <plugins>
    <plugin>
      <groupId>io.github.ascopes</groupId>
      <artifactId>protobuf-maven-plugin</artifactId>
      <version>5.1.11</version>
      <configuration>
        <protoc kind="binary-maven">
          <version>${protobuf.version}</version>
        </protoc>
      </configuration>
      <executions>
        <execution>
          <goals>
            <goal>generate</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

The generated sources go to build/generated/... (Gradle) or target/generated-sources/... (Maven) and are compiled together with your code. Don't commit them.

You can also run the compiler by hand:

protoc -I src/main/proto --java_out=build/gen src/main/proto/example/person.proto

Defining messages

src/main/proto/example/person.proto:

syntax = "proto3";

package example;

import "google/protobuf/timestamp.proto";

option java_package = "com.example.proto";
option java_multiple_files = true;
option java_outer_classname = "PersonProto";

message Person {
  string name = 1;
  int32 id = 2;
  optional string email = 3;
  repeated PhoneNumber phones = 4;
  map<string, string> labels = 5;
  google.protobuf.Timestamp created_at = 6;
  Role role = 7;

  oneof contact {
    string slack_handle = 8;
    string pager = 9;
  }
}

message PhoneNumber {
  string number = 1;
  PhoneType type = 2;
}

enum PhoneType {
  PHONE_TYPE_UNSPECIFIED = 0;
  PHONE_TYPE_MOBILE = 1;
  PHONE_TYPE_HOME = 2;
  PHONE_TYPE_WORK = 3;
}

enum Role {
  ROLE_UNSPECIFIED = 0;
  ROLE_USER = 1;
  ROLE_ADMIN = 2;
}

Key points:

Scalar types map to Java like this:

proto Java Notes
int32, sint32, sfixed32 int sint32 encodes negative numbers efficiently
int64, sint64, sfixed64 long
uint32, fixed32 int Java has no unsigned types; use Integer.toUnsignedLong
uint64, fixed64 long use Long.toUnsignedString
float, double float, double
bool boolean
string String must be valid UTF-8
bytes com.google.protobuf.ByteString immutable byte sequence

Building messages

Generated messages are immutable. You create them with a builder:

import com.example.proto.Person;
import com.example.proto.PhoneNumber;
import com.example.proto.PhoneType;
import com.example.proto.Role;

public class BuildPerson {
    public static void main(String[] args) {
        Person alice = Person.newBuilder()
            .setName("Alice")
            .setId(42)
            .setEmail("alice@example.com")
            .addPhones(PhoneNumber.newBuilder()
                .setNumber("555-1234")
                .setType(PhoneType.PHONE_TYPE_MOBILE))
            .setRole(Role.ROLE_ADMIN)
            .build();

        System.out.println(alice);
        System.out.println(alice.getName() + " has " + alice.getPhonesCount() + " phone(s)");
        System.out.println(alice.getPhones(0).getType());
    }
}
name: "Alice"
id: 42
email: "alice@example.com"
phones {
  number: "555-1234"
  type: PHONE_TYPE_MOBILE
}
role: ROLE_ADMIN

Alice has 1 phone(s)
PHONE_TYPE_MOBILE

toString() prints the protobuf text format. It is good for logs and debugging, but don't parse it or depend on its exact layout.

Things to know about the generated API:

Person promoted = alice.toBuilder()
    .setRole(Role.ROLE_USER)
    .clearPhones()
    .build();

System.out.println(alice.getPhonesCount());    // 1, the original is unchanged
System.out.println(promoted.getPhonesCount()); // 0
System.out.println(alice.equals(promoted));    // false

Serializing to bytes and back

import com.example.proto.Person;
import com.example.proto.PhoneNumber;
import com.example.proto.PhoneType;
import com.example.proto.Role;
import com.google.protobuf.InvalidProtocolBufferException;
import java.util.HexFormat;

public class RoundTrip {
    public static void main(String[] args) throws InvalidProtocolBufferException {
        Person alice = Person.newBuilder()
            .setName("Alice")
            .setId(42)
            .setEmail("alice@example.com")
            .addPhones(PhoneNumber.newBuilder()
                .setNumber("555-1234")
                .setType(PhoneType.PHONE_TYPE_MOBILE))
            .setRole(Role.ROLE_ADMIN)
            .build();

        byte[] bytes = alice.toByteArray();
        System.out.println(bytes.length + " bytes");
        System.out.println(HexFormat.of().formatHex(bytes));

        Person decoded = Person.parseFrom(bytes);
        System.out.println(decoded.getEmail());
        System.out.println(decoded.equals(alice));
    }
}
44 bytes
0a05416c696365102a1a11616c696365406578616d706c652e636f6d220c0a083535352d3132333410013802
alice@example.com
true

The same data as compact JSON is 132 bytes. Besides byte[], there are overloads of parseFrom for ByteString, ByteBuffer, InputStream and CodedInputStream. For output there are toByteString(), writeTo(OutputStream) and getSerializedSize().

What the bytes mean

Each field is written as a tag followed by a value. The tag is a varint holding (field_number << 3) | wire_type. The common wire types are 0 (varint) and 2 (length-delimited: strings, bytes, nested messages, packed repeated numbers).

0a 05 41 6c 69 63 65          field 1 (name), wire type 2, length 5, "Alice"
10 2a                         field 2 (id), wire type 0, varint 42
1a 11 61 6c ... 6d            field 3 (email), length 17, "alice@example.com"
22 0c 0a 08 35 ... 34 10 01   field 4 (phones), a 12-byte nested PhoneNumber
38 02                         field 7 (role), varint 2 = ROLE_ADMIN

Varints store 7 bits per byte, least significant group first. The high bit means "more bytes follow". A Person with only id = 150 encodes as 10 96 01: 150 is 1 0010110 in binary, so the low 7 bits become 0x96 (0x16 with the continuation bit set) and the remaining bit becomes 0x01.

Note what is not in the output: field names, and created_at, labels and the oneof, which were never set. In proto3, a scalar field holding its default value (0, "", false, the first enum value) is not written at all. That is one reason protobuf is so compact.

Handling invalid input

parseFrom throws the checked InvalidProtocolBufferException for malformed or truncated input:

byte[] truncated = java.util.Arrays.copyOf(bytes, bytes.length - 5);
try {
    Person.parseFrom(truncated);
} catch (InvalidProtocolBufferException e) {
    System.out.println(e.getClass().getSimpleName());
}
InvalidProtocolBufferException

Garbage input does not always fail. Bytes that happen to form valid tags parse "successfully" into a message with odd fields or unknown fields. Protobuf has no magic number or checksum, so if you need to detect corruption, add one at a higher level.

Default values and field presence

Reading a field that was never set returns its default value, never null:

Person empty = Person.getDefaultInstance();
System.out.println("[" + empty.getName() + "]");          // []
System.out.println(empty.getId());                        // 0
System.out.println(empty.getRole());                      // ROLE_UNSPECIFIED
System.out.println(empty.getCreatedAt().getSeconds());    // 0
System.out.println(empty.toByteArray().length);           // 0

For a plain proto3 scalar like name or id you cannot tell "not set" apart from "set to the default". If that difference matters, mark the field optional. You then get a has method, and the field is written even when it holds the default:

Person p = Person.newBuilder().setEmail("").build();
System.out.println(p.hasEmail());              // true, explicitly set to ""
System.out.println(p.toByteArray().length);    // 2: tag + zero length

System.out.println(Person.getDefaultInstance().hasEmail());  // false

Message-typed fields (created_at) and oneof members always have has methods (hasCreatedAt(), hasSlackHandle()).

Repeated fields, maps and oneof

Person bob = Person.newBuilder()
    .setName("Bob")
    .addPhones(PhoneNumber.newBuilder().setNumber("555-0001").setType(PhoneType.PHONE_TYPE_HOME))
    .addPhones(PhoneNumber.newBuilder().setNumber("555-0002").setType(PhoneType.PHONE_TYPE_WORK))
    .putLabels("region", "eu")
    .putLabels("team", "infra")
    .setSlackHandle("@bob")
    .build();

for (PhoneNumber phone : bob.getPhonesList()) {
    System.out.println(phone.getType() + " " + phone.getNumber());
}

System.out.println(bob.getLabelsMap());
System.out.println(bob.getLabelsOrDefault("team", "none"));
System.out.println(bob.getLabelsOrDefault("owner", "none"));
System.out.println(bob.containsLabels("region"));

switch (bob.getContactCase()) {
    case SLACK_HANDLE -> System.out.println("Slack: " + bob.getSlackHandle());
    case PAGER -> System.out.println("Pager: " + bob.getPager());
    case CONTACT_NOT_SET -> System.out.println("no contact");
}

// setting another member of the oneof clears the previous one
Person paged = bob.toBuilder().setPager("4242").build();
System.out.println(paged.getContactCase() + " " + paged.hasSlackHandle());
PHONE_TYPE_HOME 555-0001
PHONE_TYPE_WORK 555-0002
{region=eu, team=infra}
infra
none
true
Slack: @bob
PAGER false

Unknown enum values

Proto3 enums are open: a newer sender can send a number your generated code does not know. Instead of failing, the getter returns UNRECOGNIZED and the raw number is kept:

Person fromFuture = Person.newBuilder().setRoleValue(7).build();
System.out.println(fromFuture.getRole());       // UNRECOGNIZED
System.out.println(fromFuture.getRoleValue());  // 7

Always have a default branch when you switch on a protobuf enum. Calling getNumber() on UNRECOGNIZED throws IllegalArgumentException.

Well-known types: Timestamp and Duration

Protobuf ships common messages such as google.protobuf.Timestamp, Duration, Struct, Any and wrappers like StringValue. Their Java classes are in protobuf-java, and the com.google.protobuf.util package in protobuf-java-util adds helpers.

A Timestamp is seconds plus nanoseconds since the Unix epoch, in UTC. Converting to and from java.time.Instant is straightforward:

import com.google.protobuf.Duration;
import com.google.protobuf.Timestamp;
import com.google.protobuf.util.Durations;
import com.google.protobuf.util.Timestamps;
import java.time.Instant;

public class WellKnownTypes {
    static Timestamp toProto(Instant i) {
        return Timestamp.newBuilder()
            .setSeconds(i.getEpochSecond())
            .setNanos(i.getNano())
            .build();
    }

    static Instant toInstant(Timestamp ts) {
        return Instant.ofEpochSecond(ts.getSeconds(), ts.getNanos());
    }

    public static void main(String[] args) throws Exception {
        Instant created = Instant.parse("2026-10-17T12:00:00Z");
        Timestamp ts = toProto(created);
        System.out.println(ts.getSeconds());
        System.out.println(Timestamps.toString(ts));
        System.out.println(toInstant(ts).equals(created));

        Timestamp parsed = Timestamps.parse("2026-10-17T12:00:00.250Z");
        System.out.println(Timestamps.toMillis(parsed));

        Duration timeout = Durations.fromSeconds(90);
        System.out.println(Durations.toString(timeout));
        System.out.println(Timestamps.toString(Timestamps.add(ts, timeout)));
    }
}
1792238400
2026-10-17T12:00:00Z
true
1792238400250
90s
2026-10-17T12:01:30Z

Prefer Timestamp over an int64 created_at_millis field. The meaning is unambiguous, and the JSON form is a readable RFC 3339 string.

Converting to and from JSON

JsonFormat from protobuf-java-util implements the canonical proto3 JSON mapping. Use it for debugging endpoints, configuration files, or when talking to clients that cannot use protobuf:

import com.example.proto.Person;
import com.google.protobuf.util.JsonFormat;
import java.time.Instant;

public class ToJson {
    public static void main(String[] args) throws Exception {
        Person bob = Person.newBuilder()
            .setName("Bob")
            .putLabels("region", "eu")
            .putLabels("team", "infra")
            .setCreatedAt(com.google.protobuf.Timestamp.newBuilder()
                .setSeconds(Instant.parse("2026-10-17T12:00:00Z").getEpochSecond()))
            .setSlackHandle("@bob")
            .build();

        String json = JsonFormat.printer().print(bob);
        System.out.println(json);

        Person.Builder builder = Person.newBuilder();
        JsonFormat.parser().merge(json, builder);
        System.out.println(builder.build().equals(bob));
    }
}
{
  "name": "Bob",
  "labels": {
    "region": "eu",
    "team": "infra"
  },
  "createdAt": "2026-10-17T12:00:00Z",
  "slackHandle": "@bob"
}
true

The mapping rules:

Useful printer and parser options:

JsonFormat.Printer compact = JsonFormat.printer()
    .omittingInsignificantWhitespace()   // single line
    .preservingProtoFieldNames()         // created_at instead of createdAt
    .alwaysPrintFieldsWithNoPresence();  // also print "id": 0, "role": "ROLE_UNSPECIFIED", ...

JsonFormat.Parser lenient = JsonFormat.parser()
    .ignoringUnknownFields();            // default: unknown JSON keys are an error

By default the parser rejects unknown fields with InvalidProtocolBufferException. The binary parser does the opposite and keeps them (see below). If clients might send JSON produced from a newer schema, use ignoringUnknownFields().

Writing many messages to a stream

A protobuf message does not record its own length, so if you simply concatenate messages you can't tell where one ends. Use the delimited variants. They prefix each message with its size as a varint:

import com.example.proto.Person;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.HexFormat;

public class Delimited {
    public static void main(String[] args) throws IOException {
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        String[] names = {"a", "b", "c"};
        for (int i = 0; i < names.length; i++) {
            Person.newBuilder().setName(names[i]).setId(i + 1).build().writeDelimitedTo(out);
        }
        byte[] data = out.toByteArray();
        System.out.println(HexFormat.of().formatHex(data));

        try (InputStream in = new ByteArrayInputStream(data)) {
            Person p;
            while ((p = Person.parseDelimitedFrom(in)) != null) {   // null at end of stream
                System.out.println(p.getId() + " " + p.getName());
            }
        }
    }
}
050a01611001050a01621002050a01631003
1 a
2 b
3 c

This works with FileOutputStream/FileInputStream (wrap them in buffered streams) and with sockets. It is a simple way to store a log of records in a file. Don't call parseFrom(InputStream) in a loop on a shared stream: it reads until end of stream.

Evolving a schema

Protobuf's main strength is that old and new code can exchange messages. Say version 1 of a service uses UserV1 and version 2 adds a field:

message UserV1 { string name = 1; int32 id = 2; }
message UserV2 { string name = 1; int32 id = 2; string email = 3; }

(In real code both would be the same message User in two versions of the same file. Two names let us show it in one program.)

UserV2 v2 = UserV2.newBuilder().setName("Alice").setId(42).setEmail("alice@example.com").build();

// old code reads new data: field 3 is unknown but kept
UserV1 v1 = UserV1.parseFrom(v2.toByteArray());
System.out.println(v1.getName() + " " + v1.getId());
System.out.println(v1.getUnknownFields().asMap().keySet());

// old code modifies and re-serializes: the email survives the round trip
UserV1 updated = v1.toBuilder().setId(43).build();
UserV2 back = UserV2.parseFrom(updated.toByteArray());
System.out.println(back.getId() + " " + back.getEmail());
Alice 42
[3]
43 alice@example.com

New code reading old data simply sees default values for the new fields.

Rules for compatible changes:

message User {
  reserved 3, 15 to 20;
  reserved "email";
  string name = 1;
  int32 id = 2;
}

Tips

Feedback about page:

Feedback:
Optional: your email if you want me to get back to you:



Table Of Contents