Testing in CI
suggest changeContinuous integration (CI) means a server builds the project and runs its tests on every push and pull request. This catches code that only works on the author's machine before it is merged.
A Java project is ready for CI when:
- one command builds the code and runs all tests, with no IDE and no manual steps
- that command exits with a non-zero status when a test fails, so the CI job turns red
- tests are deterministic: they don't depend on the machine's time zone, locale, free ports, test order or timing
- the build writes machine-readable reports (JUnit XML) that the CI server can display
This article covers running JUnit tests with Maven and Gradle the way a CI server does, splitting fast and slow tests, a GitHub Actions workflow with a JDK and OS matrix, test reports, making tests CI-proof, parallel execution and handling flaky tests.
All examples were run on JDK 25 with JUnit 6.1, Maven 3.9 (Surefire/Failsafe 3.6) and Gradle 9.8. For writing the tests themselves see Unit Testing, Parameterized tests and Mocks and test doubles.
The example project
A small Maven project with one class under test:
// src/main/java/com/example/shop/PriceCalculator.java
package com.example.shop;
import java.math.BigDecimal;
import java.math.RoundingMode;
public final class PriceCalculator {
private PriceCalculator() {
}
/** Applies a percentage discount and rounds to cents. */
public static BigDecimal discounted(BigDecimal price, int percent) {
if (percent < 0 || percent > 100) {
throw new IllegalArgumentException("percent out of range: " + percent);
}
BigDecimal factor = BigDecimal.valueOf(100 - percent, 2);
return price.multiply(factor).setScale(2, RoundingMode.HALF_UP);
}
}
// src/test/java/com/example/shop/PriceCalculatorTest.java
package com.example.shop;
import org.junit.jupiter.api.Test;
import java.math.BigDecimal;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
class PriceCalculatorTest {
@Test
void appliesDiscount() {
assertEquals(new BigDecimal("80.00"),
PriceCalculator.discounted(new BigDecimal("100"), 20));
}
@Test
void roundsToCents() {
assertEquals(new BigDecimal("6.66"),
PriceCalculator.discounted(new BigDecimal("9.99"), 33));
}
@Test
void rejectsInvalidPercent() {
assertThrows(IllegalArgumentException.class,
() -> PriceCalculator.discounted(BigDecimal.TEN, 120));
}
}
The pom.xml pins every plugin version. Without explicit versions, the build depends on whatever defaults the installed Maven version has, and the CI server may have a different version than your laptop.
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>shop</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.release>21</maven.compiler.release>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.junit</groupId>
<artifactId>junit-bom</artifactId>
<version>6.1.3</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.16.0</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.6.0</version>
</plugin>
</plugins>
</build>
</project>
Running tests the way CI does
On CI, run Maven in batch mode:
$ mvn -B -ntp verify
-B(--batch-mode) disables interactive prompts and ANSI colors, which would show up as garbage in CI logs.-ntp(--no-transfer-progress) hides the thousands of "Downloading..." lines.verifycompiles, runs unit tests, packages the jar and runs integration tests. Use it rather thantestso CI checks everything a release build does.
The second test above has a wrong expectation: 9.99 × 0.67 is 6.6933, which rounds to 6.69. Maven reports the failure and exits with status 1:
[INFO] --- surefire:3.6.0:test (default-test) @ shop ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.shop.PriceCalculatorTest
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.136 s <<< FAILURE! -- in com.example.shop.PriceCalculatorTest
[ERROR] com.example.shop.PriceCalculatorTest.roundsToCents -- Time elapsed: 0.012 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <6.66> but was: <6.69>
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1199)
at com.example.shop.PriceCalculatorTest.roundsToCents(PriceCalculatorTest.java:20)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] PriceCalculatorTest.roundsToCents:20 expected: <6.66> but was: <6.69>
[INFO]
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
That exit status is all a CI server needs to mark the job as failed. Never add -DskipTests or -Dmaven.test.failure.ignore=true to a CI command to "get the build green". The build is green, but nothing was checked.
Use the Maven or Gradle wrapper
The wrapper is a small script committed to the repository. It downloads and runs the exact build tool version the project specifies, so CI, your laptop and your colleagues all use the same version:
$ mvn wrapper:wrapper -Dmaven=3.9.16 # creates mvnw, mvnw.cmd, .mvn/wrapper/
$ ./mvnw -B -ntp verify
$ gradle wrapper # creates gradlew, gradlew.bat, gradle/wrapper/
$ ./gradlew build
Commit the generated files. On Linux and macOS CI runners, make sure mvnw / gradlew are committed as executable (git update-index --chmod=+x mvnw), or the job fails with Permission denied.
Running a subset of tests
These are handy for reproducing a CI failure locally:
$ mvn test -Dtest=PriceCalculatorTest # one class
$ mvn test -Dtest='PriceCalculatorTest#roundsToCents' # one method
$ mvn test -Dtest='*Calculator*' # pattern
$ mvn test -Dsurefire.skipAfterFailureCount=1 # stop at the first failure
$ ./gradlew test --tests PriceCalculatorTest
$ ./gradlew test --tests 'com.example.shop.PriceCalculatorTest.roundsToCents'
Unit tests and integration tests
Integration tests start servers, talk to databases or touch the network. They are slower, so most projects run them separately from unit tests. In Maven the convention is:
- Surefire runs
*Test.java,Test*.java,*Tests.javaand*TestCase.javain thetestphase. - Failsafe runs
*IT.java,IT*.javaand*ITCase.javain theintegration-testphase and fails the build in theverifyphase. The split means thepost-integration-testphase (stopping a server, removing a container) still runs when tests fail.
Add Failsafe to the <plugins> section:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>3.6.0</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
An integration test that starts a real HTTP server. It binds to port 0, so the operating system picks a free port. A hard-coded port like 8080 fails as soon as two CI jobs share a machine, or something else is already listening there.
// src/test/java/com/example/shop/PriceServerIT.java
package com.example.shop;
import com.sun.net.httpserver.HttpServer;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.Timeout;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import static org.junit.jupiter.api.Assertions.assertEquals;
class PriceServerIT {
static HttpServer server;
static URI base;
@BeforeAll
static void start() throws IOException {
// Port 0: the OS picks a free port, so parallel CI jobs never collide.
server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/price", exchange -> {
byte[] body = "80.00".getBytes(StandardCharsets.UTF_8);
exchange.sendResponseHeaders(200, body.length);
exchange.getResponseBody().write(body);
exchange.close();
});
server.start();
base = URI.create("http://127.0.0.1:" + server.getAddress().getPort());
}
@AfterAll
static void stop() {
server.stop(0);
}
@Test
@Timeout(5)
void returnsPrice() throws Exception {
try (HttpClient client = HttpClient.newHttpClient()) {
HttpResponse<String> resp = client.send(
HttpRequest.newBuilder(base.resolve("/price")).build(),
HttpResponse.BodyHandlers.ofString());
assertEquals(200, resp.statusCode());
assertEquals("80.00", resp.body());
}
}
}
@Timeout(5) fails the test after 5 seconds instead of letting a hung connection block the CI job until the job-level timeout (6 hours by default on GitHub Actions).
With the wrong expectation in roundsToCents fixed, mvn -B -ntp verify runs both kinds of tests:
[INFO] --- surefire:3.6.0:test (default-test) @ shop ---
[INFO] Running com.example.shop.PriceCalculatorTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.060 s -- in com.example.shop.PriceCalculatorTest
[INFO] Results:
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] --- jar:3.5.0:jar (default-jar) @ shop ---
[INFO] --- failsafe:3.6.0:integration-test (default) @ shop ---
[INFO] Running com.example.shop.PriceServerIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.373 s -- in com.example.shop.PriceServerIT
[INFO] Results:
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] --- failsafe:3.6.0:verify (default) @ shop ---
[INFO] BUILD SUCCESS
mvn test gives fast feedback with unit tests only. mvn verify is the full CI run.
For integration tests against real databases or message brokers, Testcontainers starts them in Docker from inside the test. GitHub-hosted Linux runners have Docker installed, so these tests run on CI without extra setup.
Tagging slow tests
Naming conventions split tests by kind. Tags split them by any property you like: slow, needs network, needs a GPU.
// src/test/java/com/example/shop/CatalogImportTest.java
package com.example.shop;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
class CatalogImportTest {
@Test
@Tag("slow")
void importsLargeCatalog(@TempDir Path dir) throws IOException {
Path file = dir.resolve("catalog.csv");
List<String> lines = new ArrayList<>();
for (int i = 0; i < 200_000; i++) {
lines.add("sku-" + i + ",9.99");
}
Files.write(file, lines);
assertEquals(200_000, Files.readAllLines(file).size());
}
}
@TempDir gives each test a fresh directory that JUnit deletes afterwards. Never write to a fixed path like /tmp/test.csv: two jobs on the same runner will overwrite each other's files.
To skip slow tests by default, set Surefire's excludedGroups as a property, not inside <configuration>. A value in <configuration> can't be overridden from the command line, a property can:
<properties>
<!-- Skip @Tag("slow") by default; override with -DexcludedGroups= -->
<excludedGroups>slow</excludedGroups>
</properties>
$ mvn -B -ntp test # skips slow tests
$ mvn -B -ntp test -Dgroups=slow -DexcludedGroups= # only slow tests
[INFO] Running com.example.shop.CatalogImportTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.308 s -- in com.example.shop.CatalogImportTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
Tag expressions can be combined: -Dgroups='slow & !network'.
The same with Gradle
// build.gradle.kts
plugins {
java
}
repositories {
mavenCentral()
}
tasks.withType<JavaCompile>().configureEach {
options.release = 21
}
dependencies {
testImplementation(platform("org.junit:junit-bom:6.1.3"))
testImplementation("org.junit.jupiter:junit-jupiter")
testRuntimeOnly("org.junit.platform:junit-platform-launcher")
}
tasks.test {
useJUnitPlatform {
excludeTags("slow")
}
// Print each failure in full in the CI log, not just "1 test failed".
testLogging {
events("passed", "skipped", "failed")
exceptionFormat = org.gradle.api.tasks.testing.logging.TestExceptionFormat.FULL
}
}
tasks.register<Test>("slowTest") {
description = "Runs tests tagged @Tag(\"slow\")."
group = "verification"
testClassesDirs = sourceSets.test.get().output.classesDirs
classpath = sourceSets.test.get().runtimeClasspath
useJUnitPlatform { includeTags("slow") }
}
By default Gradle only prints a summary of failed tests to the console and points to an HTML report. The CI log is often all you have, so the testLogging block is worth adding. With the failing assertion, ./gradlew test prints:
> Task :test FAILED
PriceCalculatorTest > appliesDiscount() PASSED
PriceCalculatorTest > rejectsInvalidPercent() PASSED
PriceCalculatorTest > roundsToCents() FAILED
org.opentest4j.AssertionFailedError: expected: <6.66> but was: <6.69>
at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1199)
at app//com.example.shop.PriceCalculatorTest.roundsToCents(PriceCalculatorTest.java:20)
PriceServerIT > returnsPrice() PASSED
4 tests completed, 1 failed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///home/runner/work/shop/shop/build/reports/tests/test/index.html
Gradle has no Failsafe-style naming convention: PriceServerIT ran in the normal test task. To separate integration tests, give them their own source set with the JVM Test Suite plugin, or filter by tag as with slowTest above.
On CI, run ./gradlew build (compile, test, check, assemble) or ./gradlew check. Gradle writes JUnit XML to build/test-results/test/ and HTML to build/reports/tests/test/.
A GitHub Actions workflow
Save this as .github/workflows/ci.yml:
name: CI
on:
push:
branches: [main]
pull_request:
# Cancel the previous run when a new commit is pushed to the same branch/PR.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
checks: write # for the test report action
jobs:
test:
runs-on: ${{ matrix.os }}
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
java: ['21', '25']
include:
- os: windows-latest
java: '25'
steps:
- uses: actions/checkout@v7
- uses: actions/setup-java@v6
with:
distribution: temurin
java-version: ${{ matrix.java }}
cache: maven
- name: Build and test
run: ./mvnw -B -ntp verify
shell: bash
- name: Publish test report
if: ${{ !cancelled() }}
uses: mikepenz/action-junit-report@v6
with:
report_paths: '**/target/*-reports/TEST-*.xml'
check_name: Tests (${{ matrix.os }}, JDK ${{ matrix.java }})
- name: Upload raw reports
if: failure()
uses: actions/upload-artifact@v7
with:
name: test-reports-${{ matrix.os }}-${{ matrix.java }}
path: |
**/target/surefire-reports/
**/target/failsafe-reports/
retention-days: 7
The important details:
cache: maveninsetup-javacaches~/.m2/repository, keyed on a hash of thepom.xmlfiles. Without it every run downloads all dependencies. Usecache: gradlefor Gradle, orgradle/actions/setup-gradle@v6, which also caches the Gradle build cache.- The matrix runs the tests on each supported JDK and on Windows.
fail-fast: falselets the other combinations finish when one fails, so you can see whether a failure is specific to one JDK or one OS. shell: bashmakes./mvnwwork on the Windows runner too (the default shell there is PowerShell).timeout-minutesstops a deadlocked test suite from burning runner minutes for hours.if: ${{ !cancelled() }}runs the report step even when the build step failed, which is exactly when you need the report.if: failure()uploads the raw XML only for failed runs.- Report actions like
mikepenz/action-junit-reportread the JUnit XML files and show failed tests as a check with annotations on the lines that failed, so you don't have to dig through the log.
For Gradle, replace the build step with ./gradlew build and the report path with **/build/test-results/**/TEST-*.xml.
GitHub Actions sets the environment variable CI=true, and so do GitLab CI, CircleCI, Travis CI, Bitbucket Pipelines and most other services. Tests and build scripts can use it to detect CI, as shown later.
Other CI services
The build command and reports are the same everywhere. Only the YAML around them changes. GitLab CI, for example:
# .gitlab-ci.yml
test:
image: eclipse-temurin:25-jdk
variables:
MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
cache:
key:
files: [pom.xml]
paths: [.m2/repository]
script:
- ./mvnw -B -ntp verify
artifacts:
when: always
reports:
junit:
- target/surefire-reports/TEST-*.xml
- target/failsafe-reports/TEST-*.xml
artifacts:reports:junit makes GitLab show test results in the merge request.
The JUnit XML report
Surefire writes an XML file and a text summary per test class to target/surefire-reports/ (Failsafe uses target/failsafe-reports/):
TEST-com.example.shop.CatalogImportTest.xml
TEST-com.example.shop.PriceCalculatorTest.xml
com.example.shop.CatalogImportTest.txt
com.example.shop.PriceCalculatorTest.txt
<testsuite name="com.example.shop.CatalogImportTest" time="0.308"
tests="1" errors="0" skipped="0" failures="0" flakes="0">
<properties>...all system properties of the test JVM...</properties>
<testcase name="importsLargeCatalog" classname="com.example.shop.CatalogImportTest" time="0.308"/>
</testsuite>
This format, originally from Ant's JUnit task, is the common format of CI test reporting. Jenkins, GitLab, Azure DevOps, TeamCity, CircleCI and the GitHub report actions all read it. Anything printed to System.out during a failing test goes into the report too, so log useful context in tests.
Making tests CI-proof
A test that passes locally and fails on CI almost always depends on something about the environment. The usual suspects:
| Depends on | Typical symptom | Fix |
|---|---|---|
| Default time zone | Dates off by hours, wrong day | Pass ZoneId or a Clock explicitly |
| Default locale | 12,50 instead of 12.50, wrong month names |
Pass Locale explicitly (Locale.ROOT for machine-readable text) |
| Default charset / line separator | Broken non-ASCII text, \r\n vs \n on Windows |
Specify StandardCharsets.UTF_8, compare with lines() |
| Fixed ports and paths | BindException, files from another job |
Port 0, @TempDir |
| Test execution order | Passes alone, fails in the full suite | No shared mutable state, random order |
| Timing | Fails under load | Wait for a condition, not a fixed sleep |
| Current date | Fails on Feb 29 or on Jan 1 | Inject a Clock |
Time zone and locale
This code has two hidden dependencies:
package com.example.shop;
import java.math.BigDecimal;
import java.time.Instant;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.util.Locale;
public final class Receipt {
private static final DateTimeFormatter DATE =
DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm");
// Buggy: depends on the machine's default time zone and locale.
public static String lineDefault(Instant when, BigDecimal total) {
return DATE.withZone(ZoneId.systemDefault()).format(when)
+ String.format(" %.2f", total);
}
// Fixed: zone and locale are explicit.
public static String line(Instant when, BigDecimal total, ZoneId zone) {
return DATE.withZone(zone).format(when)
+ String.format(Locale.ROOT, " %.2f", total);
}
}
A test for lineDefault, written by someone working in UTC:
@Test
void lineDefault() {
assertEquals("2026-03-01 23:30 12.50",
Receipt.lineDefault(Instant.parse("2026-03-01T23:30:00Z"), new BigDecimal("12.5")));
}
Run on a machine in Central Europe, the time is an hour later and on the next day:
org.opentest4j.AssertionFailedError: expected: <2026-03-01 23:30 12.50> but was: <2026-03-02 00:30 12.50>
Run in UTC but with a German locale (-Duser.timezone=UTC -Duser.language=de -Duser.country=DE), the decimal separator changes:
org.opentest4j.AssertionFailedError: expected: <2026-03-01 23:30 12.50> but was: <2026-03-01 23:30 12,50>
The real fix is in the code: pass the zone and locale explicitly, as line() does, or inject a java.time.Clock (see Dates and Time (java.time.*) and Locale). As a second line of defense, pin the defaults for the test JVM so every machine runs tests with the same settings:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.6.0</version>
<configuration>
<argLine>-Duser.timezone=UTC -Duser.language=en -Duser.country=US</argLine>
</configuration>
</plugin>
Add the same <configuration> to Failsafe. In Gradle use jvmArgs("-Duser.timezone=UTC", "-Duser.language=en", "-Duser.country=US") in tasks.test. If you use JaCoCo, write <argLine>@{argLine} -Duser.timezone=UTC ...</argLine> so the coverage agent isn't dropped (see Code coverage).
Pinning hides bugs that your users in other time zones will hit. Some teams do the opposite and run a CI job with an unusual zone and locale, such as -Duser.timezone=Pacific/Chatham -Duser.language=tr, to find these bugs on purpose.
Test order
JUnit doesn't promise any order of test methods. The default order is deterministic but intentionally not obvious, so a hidden dependency between tests can stay hidden for years until a JUnit upgrade or a renamed method changes the order. This class only passes when cartStartsEmpty runs first:
package com.example.shop;
import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
class OrderTest {
// Shared mutable state: the hidden dependency between tests.
static final List<String> cart = new ArrayList<>();
@Test
void addItem() {
cart.add("book");
assertEquals(1, cart.size());
}
@Test
void cartStartsEmpty() {
assertEquals(0, cart.size());
}
}
Make JUnit shuffle methods and classes with src/test/resources/junit-platform.properties:
junit.jupiter.testmethod.order.default=org.junit.jupiter.api.MethodOrderer$Random
junit.jupiter.testclass.order.default=org.junit.jupiter.api.ClassOrderer$Random
Each run now uses a different order. When one fails, you need to reproduce it, so set the seed explicitly. JUnit reads configuration parameters from system properties too:
$ mvn -B -ntp -q test -Djunit.jupiter.execution.order.random.seed=123456789
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.124 s <<< FAILURE! -- in com.example.shop.OrderTest
[ERROR] OrderTest.cartStartsEmpty:22 expected: <0> but was: <1>
$ mvn -B -ntp -q test -Djunit.jupiter.execution.order.random.seed=987654321
$
On GitHub Actions a good choice is -Djunit.jupiter.execution.order.random.seed=${{ github.run_id }}: the order changes on every run, and the seed is visible in the logged command.
Running only on CI, or never on CI
Some tests need resources only CI has (credentials in secrets, a big machine). Others need something CI doesn't have (a display, a local license). JUnit's conditions handle both:
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.DisabledIfEnvironmentVariable;
import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;
import static org.junit.jupiter.api.Assertions.assertTrue;
class EnvTest {
@Test
void alwaysRuns() {
assertTrue(Runtime.getRuntime().availableProcessors() > 0);
}
@Test
@EnabledIfEnvironmentVariable(named = "CI", matches = "true")
void onlyOnCi() {
System.out.println("running on CI");
}
@Test
@DisabledIfEnvironmentVariable(named = "CI", matches = "true",
disabledReason = "needs a desktop session")
void notOnCi() {
System.out.println("running locally");
}
}
Running it with the JUnit Console Launcher (junit-platform-console-standalone, a single jar that runs tests without Maven or Gradle) and CI=true:
$ javac -cp junit-platform-console-standalone-6.1.3.jar -d out EnvTest.java
$ CI=true java -jar junit-platform-console-standalone-6.1.3.jar execute \
--class-path out --select-class EnvTest \
--details=tree --disable-banner --disable-ansi-colors \
--reports-dir=reports
running on CI
╷
├─ JUnit Platform Suite ✔
├─ JUnit Jupiter ✔
│ └─ EnvTest ✔
│ ├─ onlyOnCi() ✔
│ ├─ alwaysRuns() ✔
│ └─ notOnCi() ↷ Environment variable [CI] with value [true] matches regular expression [true] ==> needs a desktop session
└─ JUnit Vintage ✔
Test run finished after 229 ms
[ 4 containers found ]
[ 0 containers skipped ]
[ 4 containers started ]
[ 0 containers aborted ]
[ 4 containers successful ]
[ 0 containers failed ]
[ 3 tests found ]
[ 1 tests skipped ]
[ 2 tests started ]
[ 0 tests aborted ]
[ 2 tests successful ]
[ 0 tests failed ]
The launcher exits with status 1 if any test failed, and --reports-dir writes JUnit XML, so it works for CI pipelines without a build tool. Skipped tests show the reason, so nobody wonders why a test "disappeared" on CI.
Use these conditions sparingly. A test that never runs on CI is a test nobody runs.
Parallel execution
CI runners have several cores. JUnit can run tests in parallel within one JVM. Enable it in junit-platform.properties:
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.execution.parallel.config.strategy=dynamic
junit.jupiter.execution.parallel.config.dynamic.factor=1
junit.jupiter.execution.parallel.config.executor-service=WORKER_THREAD_POOL
dynamic with factor 1 uses one thread per available core. WORKER_THREAD_POOL selects the newer thread-pool executor. Without that line, JUnit 6.1 uses the older ForkJoinPool executor and prints a notice suggesting the switch. Four tests that each sleep 500 ms:
c on junit-1-worker-2
d on junit-1-worker-4
b on junit-1-worker-3
a on junit-1-worker-1
Test run finished after 712 ms
They finish in about 700 ms instead of 2 seconds. Parallel tests must not share mutable state. Mark tests that touch global state, such as system properties, environment, or a shared database table, with @ResourceLock("name") so tests holding the same lock don't overlap, or @Isolated to run a class alone.
The build tools can also run several test JVMs at once: Surefire's -DforkCount=1C (one fork per core) and Gradle's maxParallelForks = Runtime.getRuntime().availableProcessors() in tasks.test. Separate JVMs isolate static state better, at the cost of JVM startup time.
Flaky tests
A flaky test sometimes passes and sometimes fails with the same code. Typical causes are races, timeouts that are too tight for a busy CI machine, and the order and environment dependencies covered above. Flaky tests are worse than no tests: people learn to click "re-run" and stop believing red builds.
Both build tools can retry failed tests and report them as flaky instead of failed. That keeps the build usable while you fix the test, but the retries must be visible. A test that only fails on the first attempt:
@Test
void sometimesFails() {
// Simulates a race: fails on the first attempt only.
assertTrue(attempts.incrementAndGet() > 1, "timed out waiting for cache");
}
With Surefire's rerunFailingTestsCount:
$ mvn -B -ntp test -Dsurefire.rerunFailingTestsCount=2
[INFO] Results:
[WARNING] Flakes:
[WARNING] com.example.shop.FlakyTest.sometimesFails
[ERROR] Run 1: FlakyTest.sometimesFails:15 timed out waiting for cache ==> expected: <true> but was: <false>
[INFO] Run 2: PASS
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
The XML report records it with flakes="1" and a <flakyFailure> element containing the first failure's stack trace, and report actions can list flaky tests separately.
For Gradle, use the official org.gradle.test-retry plugin. This configuration only retries on CI, so flakiness stays visible during local development:
plugins {
java
id("org.gradle.test-retry") version "1.6.6"
}
tasks.test {
retry {
// Retry only on CI, so flaky tests stay visible locally.
if (System.getenv("CI") == "true") {
maxRetries = 2
maxFailures = 10
}
failOnPassedAfterRetry = false
}
}
maxFailures disables retries when many tests fail at once: that's a real breakage, and retrying it would only waste time. Gradle retries failed tests in a new JVM, so a static counter would start from zero again. For this run the example test remembers its first attempt in a file under build/ instead:
> Task :test
FlakyTest > sometimesFails() FAILED
org.opentest4j.AssertionFailedError: timed out waiting for cache ==> expected: <true> but was: <false>
at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:232)
at app//com.example.shop.FlakyTest.sometimesFails(FlakyTest.java:18)
PriceCalculatorTest > appliesDiscount() PASSED
PriceCalculatorTest > rejectsInvalidPercent() PASSED
PriceCalculatorTest > roundsToCents() PASSED
PriceServerIT > returnsPrice() PASSED
ReceiptTest > line() PASSED
FlakyTest > sometimesFails() PASSED
7 tests completed, 1 failed
BUILD SUCCESSFUL in 4s
Treat every flake as a bug with an owner. Common fixes:
- replace
Thread.sleep(500)with polling for the condition until a deadline (or a library like Awaitility) - give asynchronous operations generous timeouts (seconds, not milliseconds): CI machines are shared and slower than your laptop
- remove shared static state, or reset it in
@BeforeEach - use
@TempDirand port 0 instead of fixed paths and ports
Summary
- CI runs one command,
./mvnw -B -ntp verifyor./gradlew build, that fails with a non-zero exit code when any test fails. Pin plugin versions and commit the wrapper. - Keep unit tests fast. Run integration tests with Failsafe (
*IT) or a separate Gradle task, and use tags for slow tests. - Cache dependencies, use a JDK/OS matrix with
fail-fast: false, set a job timeout, and publish JUnit XML reports even when the build fails. - Make tests independent of time zone, locale, ports, paths and order: pass
ZoneId,ClockandLocaleexplicitly, use port 0 and@TempDir, and shuffle test order with a known seed. - Use
@Timeoutso a hung test fails quickly instead of hanging the job. - Retry flaky tests only as a stopgap, keep the retries visible in reports, and fix the root cause.