Testing in CI

suggest change

Continuous integration (CI) means a server builds the project and runs its tests on every push and pull request. This catches code that only works on the author's machine before it is merged.

A Java project is ready for CI when:

This article covers running JUnit tests with Maven and Gradle the way a CI server does, splitting fast and slow tests, a GitHub Actions workflow with a JDK and OS matrix, test reports, making tests CI-proof, parallel execution and handling flaky tests.

All examples were run on JDK 25 with JUnit 6.1, Maven 3.9 (Surefire/Failsafe 3.6) and Gradle 9.8. For writing the tests themselves see Unit Testing, Parameterized tests and Mocks and test doubles.

The example project

A small Maven project with one class under test:

// src/main/java/com/example/shop/PriceCalculator.java
package com.example.shop;

import java.math.BigDecimal;
import java.math.RoundingMode;

public final class PriceCalculator {
    private PriceCalculator() {
    }

    /** Applies a percentage discount and rounds to cents. */
    public static BigDecimal discounted(BigDecimal price, int percent) {
        if (percent < 0 || percent > 100) {
            throw new IllegalArgumentException("percent out of range: " + percent);
        }
        BigDecimal factor = BigDecimal.valueOf(100 - percent, 2);
        return price.multiply(factor).setScale(2, RoundingMode.HALF_UP);
    }
}
// src/test/java/com/example/shop/PriceCalculatorTest.java
package com.example.shop;

import org.junit.jupiter.api.Test;

import java.math.BigDecimal;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;

class PriceCalculatorTest {

    @Test
    void appliesDiscount() {
        assertEquals(new BigDecimal("80.00"),
                PriceCalculator.discounted(new BigDecimal("100"), 20));
    }

    @Test
    void roundsToCents() {
        assertEquals(new BigDecimal("6.66"),
                PriceCalculator.discounted(new BigDecimal("9.99"), 33));
    }

    @Test
    void rejectsInvalidPercent() {
        assertThrows(IllegalArgumentException.class,
                () -> PriceCalculator.discounted(BigDecimal.TEN, 120));
    }
}

The pom.xml pins every plugin version. Without explicit versions, the build depends on whatever defaults the installed Maven version has, and the CI server may have a different version than your laptop.

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example</groupId>
  <artifactId>shop</artifactId>
  <version>1.0-SNAPSHOT</version>

  <properties>
    <maven.compiler.release>21</maven.compiler.release>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.junit</groupId>
        <artifactId>junit-bom</artifactId>
        <version>6.1.3</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter</artifactId>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.16.0</version>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.6.0</version>
      </plugin>
    </plugins>
  </build>
</project>

Running tests the way CI does

On CI, run Maven in batch mode:

$ mvn -B -ntp verify

The second test above has a wrong expectation: 9.99 × 0.67 is 6.6933, which rounds to 6.69. Maven reports the failure and exits with status 1:

[INFO] --- surefire:3.6.0:test (default-test) @ shop ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.shop.PriceCalculatorTest
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.136 s <<< FAILURE! -- in com.example.shop.PriceCalculatorTest
[ERROR] com.example.shop.PriceCalculatorTest.roundsToCents -- Time elapsed: 0.012 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <6.66> but was: <6.69>
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1199)
	at com.example.shop.PriceCalculatorTest.roundsToCents(PriceCalculatorTest.java:20)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   PriceCalculatorTest.roundsToCents:20 expected: <6.66> but was: <6.69>
[INFO]
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

That exit status is all a CI server needs to mark the job as failed. Never add -DskipTests or -Dmaven.test.failure.ignore=true to a CI command to "get the build green". The build is green, but nothing was checked.

Use the Maven or Gradle wrapper

The wrapper is a small script committed to the repository. It downloads and runs the exact build tool version the project specifies, so CI, your laptop and your colleagues all use the same version:

$ mvn wrapper:wrapper -Dmaven=3.9.16    # creates mvnw, mvnw.cmd, .mvn/wrapper/
$ ./mvnw -B -ntp verify

$ gradle wrapper                        # creates gradlew, gradlew.bat, gradle/wrapper/
$ ./gradlew build

Commit the generated files. On Linux and macOS CI runners, make sure mvnw / gradlew are committed as executable (git update-index --chmod=+x mvnw), or the job fails with Permission denied.

Running a subset of tests

These are handy for reproducing a CI failure locally:

$ mvn test -Dtest=PriceCalculatorTest                 # one class
$ mvn test -Dtest='PriceCalculatorTest#roundsToCents' # one method
$ mvn test -Dtest='*Calculator*'                      # pattern
$ mvn test -Dsurefire.skipAfterFailureCount=1         # stop at the first failure

$ ./gradlew test --tests PriceCalculatorTest
$ ./gradlew test --tests 'com.example.shop.PriceCalculatorTest.roundsToCents'

Unit tests and integration tests

Integration tests start servers, talk to databases or touch the network. They are slower, so most projects run them separately from unit tests. In Maven the convention is:

Add Failsafe to the <plugins> section:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <version>3.6.0</version>
  <executions>
    <execution>
      <goals>
        <goal>integration-test</goal>
        <goal>verify</goal>
      </goals>
    </execution>
  </executions>
</plugin>

An integration test that starts a real HTTP server. It binds to port 0, so the operating system picks a free port. A hard-coded port like 8080 fails as soon as two CI jobs share a machine, or something else is already listening there.

// src/test/java/com/example/shop/PriceServerIT.java
package com.example.shop;

import com.sun.net.httpserver.HttpServer;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.Timeout;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;

import static org.junit.jupiter.api.Assertions.assertEquals;

class PriceServerIT {
    static HttpServer server;
    static URI base;

    @BeforeAll
    static void start() throws IOException {
        // Port 0: the OS picks a free port, so parallel CI jobs never collide.
        server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
        server.createContext("/price", exchange -> {
            byte[] body = "80.00".getBytes(StandardCharsets.UTF_8);
            exchange.sendResponseHeaders(200, body.length);
            exchange.getResponseBody().write(body);
            exchange.close();
        });
        server.start();
        base = URI.create("http://127.0.0.1:" + server.getAddress().getPort());
    }

    @AfterAll
    static void stop() {
        server.stop(0);
    }

    @Test
    @Timeout(5)
    void returnsPrice() throws Exception {
        try (HttpClient client = HttpClient.newHttpClient()) {
            HttpResponse<String> resp = client.send(
                    HttpRequest.newBuilder(base.resolve("/price")).build(),
                    HttpResponse.BodyHandlers.ofString());
            assertEquals(200, resp.statusCode());
            assertEquals("80.00", resp.body());
        }
    }
}

@Timeout(5) fails the test after 5 seconds instead of letting a hung connection block the CI job until the job-level timeout (6 hours by default on GitHub Actions).

With the wrong expectation in roundsToCents fixed, mvn -B -ntp verify runs both kinds of tests:

[INFO] --- surefire:3.6.0:test (default-test) @ shop ---
[INFO] Running com.example.shop.PriceCalculatorTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.060 s -- in com.example.shop.PriceCalculatorTest
[INFO] Results:
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] --- jar:3.5.0:jar (default-jar) @ shop ---
[INFO] --- failsafe:3.6.0:integration-test (default) @ shop ---
[INFO] Running com.example.shop.PriceServerIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.373 s -- in com.example.shop.PriceServerIT
[INFO] Results:
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] --- failsafe:3.6.0:verify (default) @ shop ---
[INFO] BUILD SUCCESS

mvn test gives fast feedback with unit tests only. mvn verify is the full CI run.

For integration tests against real databases or message brokers, Testcontainers starts them in Docker from inside the test. GitHub-hosted Linux runners have Docker installed, so these tests run on CI without extra setup.

Tagging slow tests

Naming conventions split tests by kind. Tags split them by any property you like: slow, needs network, needs a GPU.

// src/test/java/com/example/shop/CatalogImportTest.java
package com.example.shop;

import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;

import static org.junit.jupiter.api.Assertions.assertEquals;

class CatalogImportTest {

    @Test
    @Tag("slow")
    void importsLargeCatalog(@TempDir Path dir) throws IOException {
        Path file = dir.resolve("catalog.csv");
        List<String> lines = new ArrayList<>();
        for (int i = 0; i < 200_000; i++) {
            lines.add("sku-" + i + ",9.99");
        }
        Files.write(file, lines);
        assertEquals(200_000, Files.readAllLines(file).size());
    }
}

@TempDir gives each test a fresh directory that JUnit deletes afterwards. Never write to a fixed path like /tmp/test.csv: two jobs on the same runner will overwrite each other's files.

To skip slow tests by default, set Surefire's excludedGroups as a property, not inside <configuration>. A value in <configuration> can't be overridden from the command line, a property can:

<properties>
  <!-- Skip @Tag("slow") by default; override with -DexcludedGroups= -->
  <excludedGroups>slow</excludedGroups>
</properties>
$ mvn -B -ntp test                                  # skips slow tests
$ mvn -B -ntp test -Dgroups=slow -DexcludedGroups=  # only slow tests
[INFO] Running com.example.shop.CatalogImportTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.308 s -- in com.example.shop.CatalogImportTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Tag expressions can be combined: -Dgroups='slow & !network'.

The same with Gradle

// build.gradle.kts
plugins {
    java
}

repositories {
    mavenCentral()
}

tasks.withType<JavaCompile>().configureEach {
    options.release = 21
}

dependencies {
    testImplementation(platform("org.junit:junit-bom:6.1.3"))
    testImplementation("org.junit.jupiter:junit-jupiter")
    testRuntimeOnly("org.junit.platform:junit-platform-launcher")
}

tasks.test {
    useJUnitPlatform {
        excludeTags("slow")
    }
    // Print each failure in full in the CI log, not just "1 test failed".
    testLogging {
        events("passed", "skipped", "failed")
        exceptionFormat = org.gradle.api.tasks.testing.logging.TestExceptionFormat.FULL
    }
}

tasks.register<Test>("slowTest") {
    description = "Runs tests tagged @Tag(\"slow\")."
    group = "verification"
    testClassesDirs = sourceSets.test.get().output.classesDirs
    classpath = sourceSets.test.get().runtimeClasspath
    useJUnitPlatform { includeTags("slow") }
}

By default Gradle only prints a summary of failed tests to the console and points to an HTML report. The CI log is often all you have, so the testLogging block is worth adding. With the failing assertion, ./gradlew test prints:

> Task :test FAILED

PriceCalculatorTest > appliesDiscount() PASSED

PriceCalculatorTest > rejectsInvalidPercent() PASSED

PriceCalculatorTest > roundsToCents() FAILED
    org.opentest4j.AssertionFailedError: expected: <6.66> but was: <6.69>
        at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1199)
        at app//com.example.shop.PriceCalculatorTest.roundsToCents(PriceCalculatorTest.java:20)

PriceServerIT > returnsPrice() PASSED

4 tests completed, 1 failed

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///home/runner/work/shop/shop/build/reports/tests/test/index.html

Gradle has no Failsafe-style naming convention: PriceServerIT ran in the normal test task. To separate integration tests, give them their own source set with the JVM Test Suite plugin, or filter by tag as with slowTest above.

On CI, run ./gradlew build (compile, test, check, assemble) or ./gradlew check. Gradle writes JUnit XML to build/test-results/test/ and HTML to build/reports/tests/test/.

A GitHub Actions workflow

Save this as .github/workflows/ci.yml:

name: CI

on:
  push:
    branches: [main]
  pull_request:

# Cancel the previous run when a new commit is pushed to the same branch/PR.
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

permissions:
  contents: read
  checks: write   # for the test report action

jobs:
  test:
    runs-on: ${{ matrix.os }}
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest]
        java: ['21', '25']
        include:
          - os: windows-latest
            java: '25'
    steps:
      - uses: actions/checkout@v7

      - uses: actions/setup-java@v6
        with:
          distribution: temurin
          java-version: ${{ matrix.java }}
          cache: maven

      - name: Build and test
        run: ./mvnw -B -ntp verify
        shell: bash

      - name: Publish test report
        if: ${{ !cancelled() }}
        uses: mikepenz/action-junit-report@v6
        with:
          report_paths: '**/target/*-reports/TEST-*.xml'
          check_name: Tests (${{ matrix.os }}, JDK ${{ matrix.java }})

      - name: Upload raw reports
        if: failure()
        uses: actions/upload-artifact@v7
        with:
          name: test-reports-${{ matrix.os }}-${{ matrix.java }}
          path: |
            **/target/surefire-reports/
            **/target/failsafe-reports/
          retention-days: 7

The important details:

For Gradle, replace the build step with ./gradlew build and the report path with **/build/test-results/**/TEST-*.xml.

GitHub Actions sets the environment variable CI=true, and so do GitLab CI, CircleCI, Travis CI, Bitbucket Pipelines and most other services. Tests and build scripts can use it to detect CI, as shown later.

Other CI services

The build command and reports are the same everywhere. Only the YAML around them changes. GitLab CI, for example:

# .gitlab-ci.yml
test:
  image: eclipse-temurin:25-jdk
  variables:
    MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
  cache:
    key:
      files: [pom.xml]
    paths: [.m2/repository]
  script:
    - ./mvnw -B -ntp verify
  artifacts:
    when: always
    reports:
      junit:
        - target/surefire-reports/TEST-*.xml
        - target/failsafe-reports/TEST-*.xml

artifacts:reports:junit makes GitLab show test results in the merge request.

The JUnit XML report

Surefire writes an XML file and a text summary per test class to target/surefire-reports/ (Failsafe uses target/failsafe-reports/):

TEST-com.example.shop.CatalogImportTest.xml
TEST-com.example.shop.PriceCalculatorTest.xml
com.example.shop.CatalogImportTest.txt
com.example.shop.PriceCalculatorTest.txt
<testsuite name="com.example.shop.CatalogImportTest" time="0.308"
           tests="1" errors="0" skipped="0" failures="0" flakes="0">
  <properties>...all system properties of the test JVM...</properties>
  <testcase name="importsLargeCatalog" classname="com.example.shop.CatalogImportTest" time="0.308"/>
</testsuite>

This format, originally from Ant's JUnit task, is the common format of CI test reporting. Jenkins, GitLab, Azure DevOps, TeamCity, CircleCI and the GitHub report actions all read it. Anything printed to System.out during a failing test goes into the report too, so log useful context in tests.

Making tests CI-proof

A test that passes locally and fails on CI almost always depends on something about the environment. The usual suspects:

Depends on Typical symptom Fix
Default time zone Dates off by hours, wrong day Pass ZoneId or a Clock explicitly
Default locale 12,50 instead of 12.50, wrong month names Pass Locale explicitly (Locale.ROOT for machine-readable text)
Default charset / line separator Broken non-ASCII text, \r\n vs \n on Windows Specify StandardCharsets.UTF_8, compare with lines()
Fixed ports and paths BindException, files from another job Port 0, @TempDir
Test execution order Passes alone, fails in the full suite No shared mutable state, random order
Timing Fails under load Wait for a condition, not a fixed sleep
Current date Fails on Feb 29 or on Jan 1 Inject a Clock

Time zone and locale

This code has two hidden dependencies:

package com.example.shop;

import java.math.BigDecimal;
import java.time.Instant;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.util.Locale;

public final class Receipt {
    private static final DateTimeFormatter DATE =
            DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm");

    // Buggy: depends on the machine's default time zone and locale.
    public static String lineDefault(Instant when, BigDecimal total) {
        return DATE.withZone(ZoneId.systemDefault()).format(when)
                + String.format("  %.2f", total);
    }

    // Fixed: zone and locale are explicit.
    public static String line(Instant when, BigDecimal total, ZoneId zone) {
        return DATE.withZone(zone).format(when)
                + String.format(Locale.ROOT, "  %.2f", total);
    }
}

A test for lineDefault, written by someone working in UTC:

@Test
void lineDefault() {
    assertEquals("2026-03-01 23:30  12.50",
            Receipt.lineDefault(Instant.parse("2026-03-01T23:30:00Z"), new BigDecimal("12.5")));
}

Run on a machine in Central Europe, the time is an hour later and on the next day:

org.opentest4j.AssertionFailedError: expected: <2026-03-01 23:30  12.50> but was: <2026-03-02 00:30  12.50>

Run in UTC but with a German locale (-Duser.timezone=UTC -Duser.language=de -Duser.country=DE), the decimal separator changes:

org.opentest4j.AssertionFailedError: expected: <2026-03-01 23:30  12.50> but was: <2026-03-01 23:30  12,50>

The real fix is in the code: pass the zone and locale explicitly, as line() does, or inject a java.time.Clock (see Dates and Time (java.time.*) and Locale). As a second line of defense, pin the defaults for the test JVM so every machine runs tests with the same settings:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.6.0</version>
  <configuration>
    <argLine>-Duser.timezone=UTC -Duser.language=en -Duser.country=US</argLine>
  </configuration>
</plugin>

Add the same <configuration> to Failsafe. In Gradle use jvmArgs("-Duser.timezone=UTC", "-Duser.language=en", "-Duser.country=US") in tasks.test. If you use JaCoCo, write <argLine>@{argLine} -Duser.timezone=UTC ...</argLine> so the coverage agent isn't dropped (see Code coverage).

Pinning hides bugs that your users in other time zones will hit. Some teams do the opposite and run a CI job with an unusual zone and locale, such as -Duser.timezone=Pacific/Chatham -Duser.language=tr, to find these bugs on purpose.

Test order

JUnit doesn't promise any order of test methods. The default order is deterministic but intentionally not obvious, so a hidden dependency between tests can stay hidden for years until a JUnit upgrade or a renamed method changes the order. This class only passes when cartStartsEmpty runs first:

package com.example.shop;

import org.junit.jupiter.api.Test;

import java.util.ArrayList;
import java.util.List;

import static org.junit.jupiter.api.Assertions.assertEquals;

class OrderTest {
    // Shared mutable state: the hidden dependency between tests.
    static final List<String> cart = new ArrayList<>();

    @Test
    void addItem() {
        cart.add("book");
        assertEquals(1, cart.size());
    }

    @Test
    void cartStartsEmpty() {
        assertEquals(0, cart.size());
    }
}

Make JUnit shuffle methods and classes with src/test/resources/junit-platform.properties:

junit.jupiter.testmethod.order.default=org.junit.jupiter.api.MethodOrderer$Random
junit.jupiter.testclass.order.default=org.junit.jupiter.api.ClassOrderer$Random

Each run now uses a different order. When one fails, you need to reproduce it, so set the seed explicitly. JUnit reads configuration parameters from system properties too:

$ mvn -B -ntp -q test -Djunit.jupiter.execution.order.random.seed=123456789
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.124 s <<< FAILURE! -- in com.example.shop.OrderTest
[ERROR]   OrderTest.cartStartsEmpty:22 expected: <0> but was: <1>

$ mvn -B -ntp -q test -Djunit.jupiter.execution.order.random.seed=987654321
$

On GitHub Actions a good choice is -Djunit.jupiter.execution.order.random.seed=${{ github.run_id }}: the order changes on every run, and the seed is visible in the logged command.

Running only on CI, or never on CI

Some tests need resources only CI has (credentials in secrets, a big machine). Others need something CI doesn't have (a display, a local license). JUnit's conditions handle both:

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.DisabledIfEnvironmentVariable;
import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;

import static org.junit.jupiter.api.Assertions.assertTrue;

class EnvTest {
    @Test
    void alwaysRuns() {
        assertTrue(Runtime.getRuntime().availableProcessors() > 0);
    }

    @Test
    @EnabledIfEnvironmentVariable(named = "CI", matches = "true")
    void onlyOnCi() {
        System.out.println("running on CI");
    }

    @Test
    @DisabledIfEnvironmentVariable(named = "CI", matches = "true",
            disabledReason = "needs a desktop session")
    void notOnCi() {
        System.out.println("running locally");
    }
}

Running it with the JUnit Console Launcher (junit-platform-console-standalone, a single jar that runs tests without Maven or Gradle) and CI=true:

$ javac -cp junit-platform-console-standalone-6.1.3.jar -d out EnvTest.java
$ CI=true java -jar junit-platform-console-standalone-6.1.3.jar execute \
      --class-path out --select-class EnvTest \
      --details=tree --disable-banner --disable-ansi-colors \
      --reports-dir=reports
running on CI
╷
├─ JUnit Platform Suite ✔
├─ JUnit Jupiter ✔
│  └─ EnvTest ✔
│     ├─ onlyOnCi() ✔
│     ├─ alwaysRuns() ✔
│     └─ notOnCi() ↷ Environment variable [CI] with value [true] matches regular expression [true] ==> needs a desktop session
└─ JUnit Vintage ✔

Test run finished after 229 ms
[         4 containers found      ]
[         0 containers skipped    ]
[         4 containers started    ]
[         0 containers aborted    ]
[         4 containers successful ]
[         0 containers failed     ]
[         3 tests found           ]
[         1 tests skipped         ]
[         2 tests started         ]
[         0 tests aborted         ]
[         2 tests successful      ]
[         0 tests failed          ]

The launcher exits with status 1 if any test failed, and --reports-dir writes JUnit XML, so it works for CI pipelines without a build tool. Skipped tests show the reason, so nobody wonders why a test "disappeared" on CI.

Use these conditions sparingly. A test that never runs on CI is a test nobody runs.

Parallel execution

CI runners have several cores. JUnit can run tests in parallel within one JVM. Enable it in junit-platform.properties:

junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.execution.parallel.config.strategy=dynamic
junit.jupiter.execution.parallel.config.dynamic.factor=1
junit.jupiter.execution.parallel.config.executor-service=WORKER_THREAD_POOL

dynamic with factor 1 uses one thread per available core. WORKER_THREAD_POOL selects the newer thread-pool executor. Without that line, JUnit 6.1 uses the older ForkJoinPool executor and prints a notice suggesting the switch. Four tests that each sleep 500 ms:

c on junit-1-worker-2
d on junit-1-worker-4
b on junit-1-worker-3
a on junit-1-worker-1

Test run finished after 712 ms

They finish in about 700 ms instead of 2 seconds. Parallel tests must not share mutable state. Mark tests that touch global state, such as system properties, environment, or a shared database table, with @ResourceLock("name") so tests holding the same lock don't overlap, or @Isolated to run a class alone.

The build tools can also run several test JVMs at once: Surefire's -DforkCount=1C (one fork per core) and Gradle's maxParallelForks = Runtime.getRuntime().availableProcessors() in tasks.test. Separate JVMs isolate static state better, at the cost of JVM startup time.

Flaky tests

A flaky test sometimes passes and sometimes fails with the same code. Typical causes are races, timeouts that are too tight for a busy CI machine, and the order and environment dependencies covered above. Flaky tests are worse than no tests: people learn to click "re-run" and stop believing red builds.

Both build tools can retry failed tests and report them as flaky instead of failed. That keeps the build usable while you fix the test, but the retries must be visible. A test that only fails on the first attempt:

@Test
void sometimesFails() {
    // Simulates a race: fails on the first attempt only.
    assertTrue(attempts.incrementAndGet() > 1, "timed out waiting for cache");
}

With Surefire's rerunFailingTestsCount:

$ mvn -B -ntp test -Dsurefire.rerunFailingTestsCount=2
[INFO] Results:
[WARNING] Flakes:
[WARNING] com.example.shop.FlakyTest.sometimesFails
[ERROR]   Run 1: FlakyTest.sometimesFails:15 timed out waiting for cache ==> expected: <true> but was: <false>
[INFO]   Run 2: PASS
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

The XML report records it with flakes="1" and a <flakyFailure> element containing the first failure's stack trace, and report actions can list flaky tests separately.

For Gradle, use the official org.gradle.test-retry plugin. This configuration only retries on CI, so flakiness stays visible during local development:

plugins {
    java
    id("org.gradle.test-retry") version "1.6.6"
}

tasks.test {
    retry {
        // Retry only on CI, so flaky tests stay visible locally.
        if (System.getenv("CI") == "true") {
            maxRetries = 2
            maxFailures = 10
        }
        failOnPassedAfterRetry = false
    }
}

maxFailures disables retries when many tests fail at once: that's a real breakage, and retrying it would only waste time. Gradle retries failed tests in a new JVM, so a static counter would start from zero again. For this run the example test remembers its first attempt in a file under build/ instead:

> Task :test
FlakyTest > sometimesFails() FAILED
    org.opentest4j.AssertionFailedError: timed out waiting for cache ==> expected: <true> but was: <false>
        at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:232)
        at app//com.example.shop.FlakyTest.sometimesFails(FlakyTest.java:18)
PriceCalculatorTest > appliesDiscount() PASSED
PriceCalculatorTest > rejectsInvalidPercent() PASSED
PriceCalculatorTest > roundsToCents() PASSED
PriceServerIT > returnsPrice() PASSED
ReceiptTest > line() PASSED
FlakyTest > sometimesFails() PASSED
7 tests completed, 1 failed
BUILD SUCCESSFUL in 4s

Treat every flake as a bug with an owner. Common fixes:

Summary

Feedback about page:

Feedback:
Optional: your email if you want me to get back to you:



Table Of Contents