Opsian talks to Aleksey Shipilëv about JDK updates

Richard Warburton & Sadiq Jaffer

Opsian’s Richard Warburton had the pleasure of sitting down with Aleksey Shipilëv and getting his thoughts on a range of topics. In Part 1 of the interview, Aleksey and Richard discuss the current state of JDK updates.

Richard: For those who don't know who you are, would you like to introduce yourself or tell us a little bit about yourself?

Aleksey: Hi, I am Aleksey Shipilëv. I have been doing Java and performance work for about 15 years now. I can probably list the projects that I’ve been involved with in the OpenJDK itself: the various tooling like JMH, JCStress and JOL. I did a few benchmarks on the side, general performance work and some feature JEPs. After joining Red Hat I’ve also done some GC work with Epsilon and Shenandoah and I am helping with OpenJDK update releases for Red Hat.

Richard: Right. And would you say you spend most of your time these days working on Shenandoah?

Aleksey: I would say it is half and half: I spend half of my time working on Shenandoah and half on generic JVM work. The last few months have been busy with update releases with the initial flurry of patches after Red Hat took over the 8u and 11u maintenance. We are hopeful that it will recede in throughput now, so I would spend maybe a day in the week, or something like that. But yes, so far I have spent lots of time dealing with update releases.

Richard: So, that's perhaps a good place to start then because I think there's quite a lot of confusion in the Java community as a whole about what's actually happening with updates. There's a lot of ‘no free updates, no security’, a lot of concern and things out there. I know there's a few people who have put out more detailed documentation or information - perhaps you'd like to explain what you're doing and what Red Hat doing around the update releases?

Aleksey: I think there is confusion around what OpenJDK is. There is OpenJDK “the project” and there is OpenJDK “the distribution”. The confusing part about the distribution is that there is no golden binary for OpenJDK. OpenJDK site is deliberately free of binaries, which means you have to rely on downstream distributions to actually produce the build for you. [Editor: Red Hat started to publish vanilla upstream builds at https://adoptopenjdk.net/upstream.html some time after this interview took place.]

In the Sun/Oracle days, it was usually Sun/Oracle producing the proprietary downstream JDK distributions based on OpenJDK sources. Recently, Oracle had decided to do their own proprietary builds only with the commercial support attached. They graciously publish the OpenJDK builds as well on their https://jdk.java.net/ site.

Being a downstream vendor like everyone else, Oracle is free to choose what builds to publish. And they choose to publish only the latest OpenJDK 12 builds at this point. If you want 11u or 8u from Oracle, you are supposed to go for their commercial offering. But this is the Oracle part, and there are also other players in the ecosystem which take OpenJDK as the project, that is the source repositories that you have in public, and build their 8u and 11u binaries from that. There are many of them, including but not limited to Red Hat, SAP, AdoptOpenJDK, etc.

Now, the question is what do those binaries contain? There are two answers to that. First, there are obviously open shared source repositories for 8u and 11u which you can build from. Second, some vendors choose to apply some additional patches on top of that because they, for example, cannot wait for the patch to appear upstream to provide it to their users. However, I think there is the internal process pressure to share these patches upstream because it helps to load balance the work on update releases.

When you hear the news that Red Hat "takes over" the ownership of the 8u and 11u releases -- that one is about Andrew Haley being the formal project lead in those OpenJDK update projects. It is more about the formal role that makes Red Hat formally responsible for those source trees and the processes around it. The actual work happens beyond those formalities. If you look at what is being done there for the current 8u212 and 11.0.3, you would see that while Red Hat does the significant part of backporting patches, there are also other teams and other companies involved, notably SAP, and very active. Other downstream vendors, like Amazon and Google, are also active in the project.

There is the shared playing ground where vendors can collaborate on the update releases and then they build from that upstream repository. So their own 8u and 11u binaries work routinely converges at OpenJDK. Some might want to paint it as the free-for-all jungle when you have multiple vendors and you don’t know which vendor ships what, but the reality is that most vendors actually work through this upstream 8u and 11u process, and thus basically agree on what their sources are.

I guess this answers the question about free updates and free bug fixes. The sources are updated as the part of OpenJDK update releases process. Choose the vendor that gives you free builds (most of them are) from those sources, and you get free updates.

Richard: Right. So, the practical implication there for a Java developer is they can still take a Java application and have it run on different vendors JVMs. And they’re free to pick whoever offers the best support contract and process and things like that?

Aleksey: Yeah, the specification… I mean, there is the required testing that every release has to pass to be called Java. Typically, vendors run more tests on top of that. I would expect the same level of compatibility across the JDK releases.

If you look very carefully at the patch lists that vendors publish or how they describe their deviation from upstream, those patches end up being minor ones. For example, things that help them to build in their environment, provide optional CA certificates, or do some other one-off thing that is not a compatibility fix. Some of them are bug or performance fixes that are already in the update releases process, but are not yet pushed to the vanilla upstream source that you can build from right now.

Richard: Interesting, interesting. Well, it's a bit of a changing ecosystem for Java developers and they kind of need to figure out how to adapt and what the best thing is to move.

Aleksey: Yeah, I mean, for some folks. Others, like many Linux-based folks, are used to having OpenJDK binaries from their distros from time immemorial because that was the easiest option for them. Linux distros usually built OpenJDK for you. It's mostly a cultural shift, I guess. Before that, only Linux folks were used to this as it was an optional step to go and download the Oracle binary from their site...

Richard: Right yes, you can just use the distro package.

Aleksey: Yes. For everyone else, going to Oracle was the first option, right? If you wanted Java on your Windows desktop, for instance. So, what we see right now is the shift from this single-vendor mindset back to the Linux mindset where you select a provider that gives you a distribution for the product, under the conditions you like: platforms they build for, frequency/promptness of releases, how support is structured, etc. If you don't trust any of existing vendors, you can even build OpenJDK yourself and run with it.

Richard: And how would you comment upon, for example, existing Linux distros and the extent of compatibility testing that they provide? I mean, you mentioned that Red Hat obviously provided have a lot of access to testing, and do you think all distros achieve that?

Aleksey: I don't know what distros do exactly, but most of them build from the same source trees. So, if you are a user of a distro then it’s probably in your interest to go and ask them about it. In the words of Andrew Haley: if you cannot trust the package that your distro builds then maybe you should not be using that distro to begin with.

Richard: Possibly. Yeah.

Aleksey: So, yes, if you have doubts about what ships in your operation system distribution then go and ask the maintainers what is in there. I would say that since there's a shared code base for most OpenJDK builders, then even if you don't test at all and you just publish the binary and let users install it, there is a very high chance that the untested release would not be broken. Vendors who do testing would actually fix the bugs and publish the fixes into upstream source trees.

Richard: Yeah.

Aleksey: There might be some wrinkles if your build toolchain is slightly different from anyone else, these things happen. I would say that Red Hat’s RPM was basically tested as is: we build the OpenJDK as an RPM and then that RPM is tested. Therefore, we are sure that the binaries we have built are actually passing the tests. I don’t think that’s a unique situation. I think that's the same proposition for any binary vendor: you build something, you verify that binary works, and then you publish it…

Richard: Yep.

Aleksey: So ask your vendor. The differences should be minor.


This concludes part one of our interview with Aleksey. We’re still preparing part two, where the interview moves on to concurrent Garbage Collectors and Shenandoah in particular. That will be live in the next week, so don't forget to check back then.

We’d like to thank Aleksey for taking some time out and sitting down with us to share his thoughts.

Want to improve scalability? Opsian is a magnifying glass for your production JVMs

Related articles