Case Study13 min read
When ubuntu-latest broke our deploys : 1,200+ repositories fixed in under 7 hours
A Java 25 warning ended up inside a Maven groupId, blocking a production hotfix. Two version changes fixed it. Rolling them out was another story 🔥
The previous post described the auto-updater : one command that opens the same pull request across hundreds of repositories. This one is what it looks like on a bad day.
In short :
- Cause : GitHub’s
ubuntu-latestmoved to Ubuntu 26.04, whose default Java 25 prints new JVM warnings (JEP 472, JEP 498) on stderr. A deploy script read Maven’s output with2>&1, and got them as its groupId. - Fix : Maven 3.9.9 → 3.9.16, and the Artifact Registry wagon 2.2.4 → 2.2.5. No warnings on Java 11, 21, or 25.
- Rollout : the auto-updater, 1,200+ repositories in under 7 hours, all 1,292 the next day, with a build storm on the way.
- Lesson : parse stdout only. And the fix is urgent, the fleet is not : unblock the teams that have to deploy today, and leave the fleet-wide run for later.
On a Thursday morning, a teammate had a hotfix to ship to production for Mexico 🇲🇽. On the way, its pre-production deploy failed with an error nobody had seen before. She re-ran it, it failed again, and a few minutes later she sent me the link : does this error ring a bell ? The afternoon before, the same release had deployed green, with the same script. What had changed was the machine it ran on : GitHub’s ubuntu-latest now meant Ubuntu 26.04, and that runner image defaults to Java 25. GitHub’s announcement says Java 17 stays the default. The image’s software list says Java 25, and so did our runner. A user had asked which one was true a week earlier, in the announcement’s comments.
The failing script comes from the templates of three types of data pipelines, and 1,118 repositories generated from those templates carry it. Every one of them would fail the same way on its next deploy on the new image.
Eleven minutes after her message, I had the cause, and 1 hour 35 minutes after it, a validated fix. Rolling it out took the rest of the day, and a little of the next.
Timeline of the incident
The error
The deploy step downloads the pipeline’s jar from a private Maven repository with the download-maven-plugin. It failed inside the plugin :
[ERROR] Failed to execute goal com.googlecode.maven-download-plugin:download-maven-plugin:1.13.0:artifact … IllegalArgumentException: checksum location must be relativeNo checksum is configured anywhere. The green deploy of the day before ran the same release, script and plugin version, on ubuntu-24.04.
The cause : stdout and stderr in one variable
The script does not hardcode the artifact coordinates. It asks Maven for them, then strips the whitespace :
./mvnw -B -ntp -f tmp/pom.xml help:evaluate \ -Dexpression=project.groupId -q -DforceStdout > "$out_log" 2>&1groupId=$(cat "$out_log" | tr -d '\r\n[:space:]')-q -DforceStdout prints the value and nothing else, on stdout. The 2>&1 sends stderr to the same file. For years, stderr was empty, so nobody noticed.
Java 24 changed that. Two JEPs make the JVM print a warning the first time a library uses an API that is on its way out : JEP 472 for restricted methods (native code loading), JEP 498 for the memory-access methods of sun.misc.Unsafe. Maven 3.9.9 trips both : its console library loads a native library, and the Guava inside it uses Unsafe. With our Maven 3.9.9 setup on Java 25, the command started with :
WARNING: A restricted method in java.lang.System has been called…WARNING: sun.misc.Unsafe::objectFieldOffset has been called by ……Glue that to the real value and remove every space, and the “groupId” becomes :
WARNING:Arestrictedmethodinjava.lang.Systemhasbeencalled…WARNING: looks like a URI scheme, and a location with a scheme is absolute : hence checksum location must be relative.
The step never asked for Java 25. It used whatever JDK the image provided, and the image changed.
Seven fixes, tested the way CI runs
I tried each option with Java 25, an empty local Maven repository (a cold runner), and the script’s original 2>&1 capture, as on CI. Some warnings only appear when Maven downloads something, so a warm laptop gives false greens.
| Option | Result |
|---|---|
| stderr to its own file | ✅ works, by editing 1,118 scripts |
Pin ubuntu-24.04 | ⏸️ hides it until the next image |
JVM flags in .mvn/jvm.config | ⚠️ works on Java 25, breaks Java 21 |
Flags in JAVA_TOOL_OPTIONS | ❌ the JVM announces JAVA_TOOL_OPTIONS on stderr |
| Maven 3.9.10 alone | ❌ Maven’s Guice warns about Unsafe |
| Maven 3.9.16, wagon 2.2.4 | ❌ the wagon’s Guava 33.2.1 warns |
| Maven 3.9.16, wagon 2.2.5 | ✅ no warnings on Java 11, 21, or 25 |
The script change sends stderr to its own file (2> "$err_log"). The flags were --enable-native-access=ALL-UNNAMED and --sun-misc-unsafe-memory-access=allow, and Java 21 only starts with them if -XX:+IgnoreUnrecognizedVMOptions comes too. In JAVA_TOOL_OPTIONS, the JVM prints Picked up JAVA_TOOL_OPTIONS on stderr, and the script captures that line too : the same bug. Maven 3.9.10 silences the native-access warning, but Maven’s own Guice still warns about Unsafe. Wagon 2.2.4 brings Guava 33.2.1, which warns as soon as it downloads. Wagon 2.2.5 brings Guava 33.4.6.
I picked the last line because it changes only two version strings, in two files every repository already has, with no script logic and no JVM flags to explain :
.mvn/wrapper/maven-wrapper.properties: the Maven distribution,3.9.9→3.9.16.mvn/extensions.xml: the wagon that talks to the private repository,2.2.4→2.2.5
The 2>&1 is still in the script. Nothing writes to stderr any more, so it is harmless today and a trap for the next tool that prints a warning. Moving stderr to its own file belongs in the template, as a separate change.
The updater : two lines, no checks
Before writing anything, I looked at the fleet. The CI builds use the same Maven wrapper as the deploys, so the bump had to cover every template that uses Maven, not only the three pipeline ones. Among the repositories generated from those six templates, 1,292 had exactly the same setup : Maven 3.9.9, the same wrapper type, wagon 2.2.4. A handful of others have no Maven wrapper at all.
My first updater was careful. It checked the current version before replacing it, warned about unexpected values, refused files with a checksum. Then I deleted all of it, because the launcher already has the one check it needs : no diff, no pull request. An updater that sets the target value unconditionally is idempotent for free.
readonly MAVEN_VERSION='3.9.16'readonly WAGON_VERSION='2.2.5'
if [ -f .mvn/wrapper/maven-wrapper.properties ]; then perl -pi -e "s#/apache-maven/[^/]+/apache-maven-[^/]+-bin\.zip#/apache-maven/${MAVEN_VERSION}/apache-maven-${MAVEN_VERSION}-bin.zip# if /^distributionUrl=/" \ .mvn/wrapper/maven-wrapper.propertiesfi
if [ -f .mvn/extensions.xml ]; then perl -0pi -e "s#(<artifactId>\s*artifactregistry-maven-wagon\s*</artifactId>\s*<version>)[^<]*(</version>)#\${1}${WAGON_VERSION}\${2}#" \ .mvn/extensions.xmlfiperl -pi because it behaves the same on a Mac and on a runner, unlike sed -i. Only the version strings change : indentation, attribute order and the missing newline at the end of the file stay as each repository has them.
The same pull request bumped the six templates too. Since the last post, most updaters re-render a repository from its template, so without that bump, the next re-render would have silently put Maven 3.9.9 back.
For a dry run, I fetched both files from every repository and ran the updater twice on them : exactly two changed lines per repository on the first run, nothing on the second. My first idempotence check compared tar archives and reported every repository as changed : perl -i rewrites the file’s modification time, and a tar archive records it. Comparing the contents found no change.
One message to 75 engineers
The pull requests would take hours to reach every repository, but a team blocked on a deploy that morning would not wait for a bot. So right after the updater merged, one message went to the 75 data engineers and tech leads who own these repositories. Translated, it said :
GitHub switched the
ubuntu-latestrunners to Ubuntu 26.04, whose default JDK is 25. With this JDK, Maven 3.9.9 and the Artifact Registry wagon 2.2.4 print JVM warnings on stderr. Some scripts, like the deploy script, use Mavenhelp:evaluateto get the groupId or the artifactId.🤖 The fix : an auto-updater will open a pull request on every repository. There are more than 1,000 of them, so it may take a while because of GitHub’s rate limit on pull requests.
✋ If you are blocked until then, you can make the change by hand, it’s only two lines :
.mvn/wrapper/maven-wrapper.properties: replace 3.9.9 with 3.9.16.mvn/extensions.xml: replace 2.2.4 with 2.2.5
It came with a screenshot of the two-line diff :
The rollout was a load test
GitHub, which had started all this by changing ubuntu-latest, had a bad afternoon too : two incidents, one right after the other. The first delayed GitHub Actions workflow results for an hour. The second made search results fail on and off until the evening. Neither explains the first problem of the afternoon.
The message warned about GitHub’s rate limit on pull requests, the hidden one from the last post. The pull requests themselves opened fine : more than a thousand in under two and a half hours. But each one started a CI build, and the builds hit a different limit. In the afternoon, red builds started to show this :
[ERROR] Failed to read artifact descriptor for …:jar:10.0.1[ERROR] Caused by: … Could not transfer artifact …:pom:10.0.1 from/to artifact-registry-release (artifactregistry://…): Received an error from the remote server.The pom existed. It resolved fine on my machine, with the same Maven, the same wagon and an empty local repository. The wagon’s own code explains the message : a 401 or 403 becomes Permission denied, a 404 becomes does not exist, and any other HTTP error becomes Received an error from the remote server. The registry had refused a request for a file it had. The wagon does not log the status code, but under load, throttling is the likely one.
The failing repository’s last green build had taken under 30 seconds. The pull request’s build took 6 minutes 54 seconds, and logged more than two thousand lines like this one :
[INFO] Artifact … is present in the local repository, but cached from a remote repository ID that is unavailable in current build context, verifying that is downloadable from […]The cache key had not changed, but the cache was gone : GitHub evicts caches that have not been used for seven days, and the repository’s CI had not run for nine. The build fell back to a shared baseline cache, built elsewhere, whose artifacts are recorded as coming from other repository ids. Maven does not trust them, so it re-checks every one against the remote repositories. That meant thousands of requests per build, in hundreds of builds running at the same time.
I sampled the failing builds : half had the registry error, and the rest were unrelated failures, including repositories without a pom.xml and Testcontainers issues.
Merged, then re-run on main
The companion merge script from the last post approves and merges every auto-updater pull request waiting for my review. It does not look at CI. 307 pull requests were merged red, on a bet : the change was two version strings, tested on Java 11, 21, and 25, and a merge to main only builds and tests, it publishes and deploys nothing. The red, I assumed, was the registry.
Every merge starts a new build on main, and the merges were spread over the afternoon. The next morning, most of them were green on main, with the same two version strings. I was right, this time, by luck. The rest were mostly the unrelated failures from the sample, and a few dozen still showed the registry error.
Those were worth a re-run, because a red main build saves no cache : the next pull request on that repository would start cold again, and hit the registry again. I re-ran the failed jobs in small batches, five repositories at a time and three minutes apart. None failed on the registry again : most turned green, and the others exposed unrelated problems.
The 32 that never got a pull request
The day after, counting for this post, I compared the fleet with the inventory taken before the rollout. 32 repositories were still on Maven 3.9.9, all batch pipelines, and none of them had a pull request, open or closed.
The auto-updater lists its targets by GitHub topic, then splits them into chunks of 100. Behind the topic filter, gh repo list runs a GitHub search, read 100 results per page. For the batch pipelines, the listing returned 100 repositories. There are 133. The run processed one chunk, raised no error and finished green, so nothing looked wrong. The scan had run during the second GitHub incident, the one breaking search results, and the listing stopped after its first page. The same command returns all 133 today.
A re-run with the 32 names given explicitly opened the 32 pull requests, and ten minutes later they were merged. 1,292 out of 1,292, 29 hours after the first red deploy.
What I keep
- Capture stdout only when you parse it.
2>&1into a value works until a tool starts talking. When it does, the error appears three steps later, in someone else’s code. ubuntu-latestis a dependency. A step that runs Java should say which Java.- On a uniform fleet, upgrade instead of patching. The script fix worked too, but the version bump was two lines, idempotent, and the same everywhere, which is what made a same-day rollout possible.
What I would do differently
The hotfix that started all this reached production that afternoon. It only needed the two version changes in its own repository and a new release containing them. The other teams that needed to deploy had the same workaround in the message, and the auto-updater can target one repository as easily as the whole fleet. I could have unblocked the urgent deploys first, by hand or with the updater pointed at their repositories, then run it on the fleet at a quieter time.
The fix had to ship that day. I treated the fleet rollout as part of the same emergency, and it was not.
The series so far
Post one : 1,273 repositories moved to one platform. Post two : the machine that keeps them consistent. This post : the same machine on an incident day, with a fix of two version strings, a fleet-wide build storm, and 32 repositories it skipped in silence.
