Tagged: jar hell

[LinkSet] Compatibility

Theoretical part

 

Articles from Oracle

http://JEP 223: New Version-String Scheme

 

Six kinds of compatibility

Each Joda-Time relase has descriptions of incompatible changes categorized by six kinds:

Compatibility between 2.8 and 2.9
———————————
Build system — Yes
Binary compatible — Yes
Source compatible — Yes
Serialization compatible — Yes
Data compatible — Yes
— DateTimeZone data updated to version 2015g
Semantic compatible — Yes

When binary compatibilities are broken then Majour Version changed. For example v2.0 changelist

Explanation from  Stephen Colebourne, author of Joda-Time:

Build system

Not part of compatibility, just a fact about the build system in use

Example:  in v2.2

 - Ant build removed. Build only on Maven now.

Also, I think, it may be changing in artifact coordinates: groupId, artifactId or even changed repository. Maybe some changes in manifest, for example required dependencies. Or added OSGI manifest info. Maybe some classes was repackaged. Or old artifact was built with ANT and doesn’t contains a Maven’s pom.xml manifest like horrible Xerex library  and then was mavenized.

It also may be recompilation with optimizations or without debug information.

But I think that changing in packaging may be a separate kind of compatibility change.

Binary compatible and Source compatible

Whether the whole API is binary compatible or source compatible. See this document

Serialization compatible

Whether the serialization format is compatible with the previous version.

Data compatible

Whether the data, time-zone data in this case, has changed.

For example: some time zone was changed or even removed. If your database stores old timezone id and application trying to create a date object with this timezone id you’ll get an exception.

Semantic compatible

Whether there is some other semantic change. This is relevant when the classes and methods have not changed, but the meaning of a method has.

See section Serialization incompitables below

For example:  v2.0 has a lot of semantic changes

Previously, DateTimeZone.forID matched time zone names case-insensitively, now it is case-sensitive

I think it always hard to separate semantic changes when dealing with bugs. For example, JDK has bug when using null as key in Map

And that was a question is this a bug or feature.

Another one example, is Apache Commons Lang library. In version 3 the methods StringUtils.isAlpha(), isNumeric() and isAlphanumeric() now all return false when passed an empty String. Previously they returned true.

Semantic looks like correlated with behavior compatibility. When changed not signatures but they flow or contract.

Also it’s often situation when behavior was just undefined by contract. For example old class Vector doesn’t declare behavior about removing elements during iteration.  That’s why was created new class ArrayList and Vector was accepted as Superseded.
So, deprecation, as any changes in contract may be also some kind of semantic change.

 

Serialization incompitables

Compatible incopatibility

Also I think that it should be some kind of «reverse compatibility» that means a contract usage. For example, I saw an issue when subclass of HashMap doesn’t follow a contract. This change was compatible in all ways but all clients become incompatible. How to predict it, I don’t know.

Each of developer has experience when you can’t change something in your API because there is some clients that abuse it or implemented incorrectly. So it should have it’s own classification and developers should think about this kind of reverse incompatibility.

Complexity compatibility

I mean the complexity of program in wide sense: algorithm speed, it’s derivate and consumption of resources (memory, i/o, CPU, disk space), and even source code beautification and structural changes like refactoring. You know,  some algorithms may use more memory but use low processor.

For example, in some early versions of 64 bit JDK your application may fail with OutOfMemory error. This was change absolutely compatible in all categories mentioned before.

Another one sample is when new version of program is working more slowly than previous.

It may not change a contract and flow or anything else may it’s also may be changes that requires an attention.

 

Tool for checking API and BPI

Service provider interface (SPI)

https://stackoverflow.com/questions/2954372/difference-between-spi-and-api
https://en.wikipedia.org/wiki/Service_provider_interface

Also interesting

JAR Hell

[LinkSet] Dependency duplicates in Maven and Jar Hell with Java Classloader

Theoretical part:
Java Classloader

Maven: Introduction to the Dependency Mechanism

What is JAR hell? (Or is it classpath hell? Or dependency hell?)

Jar Hell made Easy — Demystifying the classpath with jHades
See JHades documentation, it very useful to find overlapping jars.

Another tool for dealing with Jar Hell is Weblogic Classloader Analysis Tool.

One of the most problematic dependency is Xeres Xerces Hell. It’s a good example how not to do library.

This presentation is a great resource about Jar Hell and the different type of classpath related exceptions:

But little bit boring.

Maven Dependency Wrangling

dealing with dependency chain issues in maven

Maven Enforcer plugin has extra rule
Ban Duplicate Classes
Also it should be very useful if you have legacy project that still runs under Java 6 or 7 you should avoid dependencies compiled with never Java 8.
You can use enforceBytecodeVersion

Please also make sure that you also specified requireJavaVersion (to compile), requireUpperBoundDeps, requireReleaseDeps, requirePluginVersions and other useful standard rules .

Also if you have a submodules in project it will be also useful ono-extra-enforcer-rules

So, your enforcer rules may looks like:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-enforcer-plugin</artifactId>
    <version>1.4.1</version>
    <executions>
        <execution>
            <id>enforce</id>
            <goals>
                <goal>enforce</goal>
            </goals>
            <configuration>
                <rules>
                    <bannedPlugins>
                        <!-- will only display a warning but does not fail the build. -->
                        <level>WARN</level>
                        <excludes>
                            <exclude>org.apache.maven.plugins:maven-verifier-plugin</exclude>
                        </excludes>
                        <message>Please consider using the maven-invoker-plugin (http://maven.apache.org/plugins/maven-invoker-plugin/)!</message>
                    </bannedPlugins>
                    <requireMavenVersion>
                        <version>3.0.5</version>
                    </requireMavenVersion>
                    <requireJavaVersion>
                        <version>1.8</version>
                    </requireJavaVersion>
                    <requireReleaseDeps>
                        <onlyWhenRelease>true</onlyWhenRelease>
                        <message>No Snapshots Allowed!</message>
                    </requireReleaseDeps>
                    <requireUpperBoundDeps>
                        <!-- 'uniqueVersions' (default:false) can be set to true if you want to compare the timestamped SNAPSHOTs  -->
                        <!-- <uniqueVersions>true</uniqueVersions> -->
                    </requireUpperBoundDeps>
                    <reactorModuleConvergence>
                        <message>The reactor is not valid</message>
                        <ignoreModuleDependencies>true</ignoreModuleDependencies>
                    </reactorModuleConvergence>
                    <requirePluginVersions>
                        <message>Best Practice is to always define plugin versions!</message>
                        <banLatest>true</banLatest>
                        <banRelease>true</banRelease>
                        <banSnapshots>true</banSnapshots>
                        <phases>clean,deploy,site</phases>
                        <additionalPlugins>
                            <additionalPlugin>org.apache.maven.plugins:maven-eclipse-plugin</additionalPlugin>
                            <additionalPlugin>org.apache.maven.plugins:maven-reactor-plugin</additionalPlugin>
                        </additionalPlugins>
                        <unCheckedPluginList>org.apache.maven.plugins:maven-enforcer-plugin,org.apache.maven.plugins:maven-idea-plugin</unCheckedPluginList>
                    </requirePluginVersions>
                    <enforceBytecodeVersion>
                        <maxJdkVersion>1.6</maxJdkVersion>
                        <excludes>
                            <exclude>org.mindrot:jbcrypt</exclude>
                        </excludes>
                    </enforceBytecodeVersion>
                    <banDuplicateClasses>
                        <ignoreClasses>
                            <!-- example of ignoring one specific class -->
                            <ignoreClass>com.xyz.i18n.Messages</ignoreClass>

                            <!-- example of ignoring with wildcards -->
                            <ignoreClass>org.apache.commons.logging.*</ignoreClass>
                        </ignoreClasses>
                        <findAllDuplicates>true</findAllDuplicates>
                    </banDuplicateClasses>
                    <banCircularDependencies/>
                    <ForbidOverridingManagedDependenciesRule>
                        <excludes>
                            <!-- guava in parent is too old, so allow to override it -->
                            <exclude>com.google.guava:guava</exclude>
                        </excludes>
                    </ForbidOverridingManagedDependenciesRule>
                    <ForbidOverridingManagedPluginsRule/>
                    <ForbidDependencyManagementInSubModulesRule/>
                    <ManageAllModulesRule/>
                </rules>
            </configuration>
        </execution>
    </executions>
    <dependencies>
        <dependency>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>extra-enforcer-rules</artifactId>
            <version>1.0-beta-3</version>
        </dependency>
        <dependency>
            <groupId>net.oneandone.maven</groupId>
            <artifactId>ono-extra-enforcer-rules</artifactId>
            <version>0.1.1</version>
        </dependency>
    </dependencies>
</plugin>

Two attempts to find duplicated classes with Maven
Remove duplicate classes the agile way: Maven Duplicate Finder Plugin

Finding Duplicate Class Definitions Using Maven

Both of this plugins are discussed here:
Figuring out duplicate class definitions using the Analyze goal

Also maven-shade-plugin does check for overlapping classes during packaging of uber-jar.

Resolving conflicts using the dependency:tree -Dverbose
It shows which dependencies are duplicated (omitted for duplicate), with are evicted with newer version (version managed from 1.6) but it doesn’t show which dependencies was excluded.

Another one good thing that worst to do is enable Failing the build on dependency analysis warnings. Note: it binded to verify phase that runed after package.

JDEPS Java Dependency Analysis Tool from JDK 8

Also some related articles

Versions compitable

For example changelog of Joda-Time v2.9

Compatibility with 2.8

Build system — Yes
Binary compatible — Yes
Source compatible — Yes
Serialization compatible — Yes
Data compatible — Yes
— DateTimeZone data updated to version 2015g
Semantic compatible — Yes

See another [Linkset] Compatibility

Jigsaw

It is possible a situation when some two libraries wanting to use the same dependency but of different versions. Unfortunately, in this cases we can’t manage this and should use -nodep versions.
Finally this problem will be resolved in JDK 9 Jigsaw: a jar can be declared as a module and it will run in it’s own isolated class loader, that reads class files from other similar module class loaders in an OSGI sort of way.
This will allow multiple versions of the same Jar to coexist in the same application if needed.

Working with deprecation

Upgrading of dependncies may require to remove some old codebase that depends on them.
This is also should be done in right way, so here some links that may helps:
* JEP 277
* Dr. Deprecator Prescriptions: important things that you should know about obsolete Java API

Speed up maven build

It’s also related topic. The main reason why I decided to add it here is because usually during speeding up build you will find a lot of problems with dependency graph.
It will helps you to make yoir project more modulized. Also for example paralell build may fails if your tests are in conflict (shares same resources, for example integration tests may use the same port).

Dependency analyzers

Also useful

* jApiCmp japicmp is a tool to compare two versions of a jar archive
* Java API Compliance Checker: A Perl script that uses javap to compare two jar archives. This approach cannot compare annotations and you need to have Perl installed.
* Clirr: A tool written in Java that compares two libraries for binary compatibility. Tracking of API changes is implemented only partially, tracking of annotations is not supported. Development has stopped around 2005.
* JDiff: A Javadoc doclet that generates an HTML report of all API changes. The source code for both versions has to be available, the differences are not distinguished between binary incompatible or not. Comparison of annotations is not supported.
* revapi: An API analysis and change tracking tool that was started about the same time as japicmp. It ships with a maven plugin and an Ant task, but the maven plugin currently (version 0.4.1) only reports changes on the command line.

[Linkset] Migration to Java 8

Notes from Oracle http://www.oracle.com/technetwork/java/javase/8-compatibility-guide-2156366.html

Also possible problems:

  • switching to 64bit JVM can require more memory (actually as I know it was only in few builds of Oracle JDK 6 and all next versions comes with enabled flag -XX:+UseCompressedOops)
  • changes in garbage collection can lead to unexpected pauses
  • inclusion of XML parsers into JDK proper requires changes to web application packaging or configuration
  • memory and runtime characterics of String#substring completely change in «minor» JDK revision
  • sorting a collection with a custom (incorrectly implemented) comparator suddenly throws exceptions it did not throw before  (Java error: Comparison method violates its general contract)
  • Calling Thread#stop(Throwable) (which was never a good idea and has been deprecated for a very long time) throws a UnsupportedOperationException since Java 8
  • Updated Unicode support changing sorting and casing behavior for some strings
  • Changes in generics compilation
  • inability to extend BitSet and implement Set<Integer> due to new default methods
  • Bigin rounding behavior (affected early builds of JDK7 and JDK8)

http://stackoverflow.com/questions/28228450/java-is-backward-compatible-but-why-we-need-to-upgrade-many-libraries-when-we-u

Also I tried to mark all this issues on StackOverflow with tags migration+java-8 and migration+java-7

Most valuable article http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
Key points: you need to update ASM lib and use cglib-nodep where possible.

Managing compatibility

See linkset [LinkSet] Compatibility
Tool for checking API and BPI http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
http://wiki.apidesign.org/wiki/SignatureTests
https://stackoverflow.com/questions/2040693/how-to-identify-a-missing-method-binary-compatibility-in-a-jar-statically
http://clirr.sourceforge.net/
http://www.sab39.org/Software/Japitools/42/
http://tattletale.jboss.org/

How Spring achieves compatibility with Java 6, 7 and 8

And don’t forget to enable -parameters option https://julien.ponge.org/blog/java8-parameter-names-and-jdk-dogfooding/

https://stackoverflow.com/questions/8111310/java-7-jdk-7-garbage-collection-and-documentation

http://allegrotech.io/How-to-migrate-to-Java-8.html

https://www.techempower.com/blog/2013/03/26/everything-about-java-8/

http://zeroturnaround.com/rebellabs/what-migrating-to-java-8-will-do-to-your-codebase-a-practical-example/

In some cases it possible speed downgrade
https://stackoverflow.com/questions/23756966/why-is-stringbuilderappendint-faster-in-java-7-than-in-java-8

Also from my experience of migration from JDK6 to JDK8:

1 . APT (Annotations Processing Tool) was removed. We used Enunciate for Web Services and it uses APT. So now we waiting next version of Enunciate that will use JSR 269 instead of APT.

And actually there was a lot of API that was removed from Java and in first link Oracle told about it just see «Features Removed from Java SE 8» and «Features Removed from JDK 8».

2 . java.util.Locale class was updated and it causes error in our application.
We have some users with ISO country code CS i.e. Serbia and Montenegro. In 2004 Montenegro separated from Serbia and this code CS was removed in new version of Locale.getISOCountries().
Since our application had checks on this code we got a lot of exceptions.

3 . maven-tomcat-plugin hasn’t yet version for Tomcat 8. You can vote or contribute MTOMCAT-234. And in Tomcat 8 in some configs we needed to change path starting with root /

4 . HashMap descendants should keep contract between put() and putAll()
All classes that extends HashMap and overrides `put() method may have errors if not overrides putAll() method.
You can see interest bug in Spring framework for details https://jira.spring.io/browse/SPR-7969.

5 . Clashing Getters.
Also we had an error when our domain classes contains two getters for the same property.
For example, class MerchantData contains two methods getParentMerchant() and isParentMerchant()

public void setParentMerchant(MerchantData parentMerchant) {
this.parentMerchant = parentMerchant;
}

public MerchantData getParentMerchant() {
return parentMerchant;
}

public boolean isParentMerchant() {
return subMerchants != null &amp;&amp; !subMerchants.isEmpty();
}

According to JavaBeans specification all getters of boolean properties should be named isSomething().
Hibernate tries to determine the type of property parentMerchant and it looks that type is Boolean.
Then he tries to get id of parentMerchant ant fail with error:

Caused by: org.hibernate.PropertyAccessException: IllegalArgumentException occurred calling getter of com.example.merchant.dao.MerchantData.id
at org.hibernate.property.BasicPropertyAccessor$BasicGetter.get(BasicPropertyAccessor.java:171)

To fix that we need to rename method isParentMerchant() or get rid of it.
This kind of issues are happened only when we run project on JDK8.
So, please take it into account when you create a new getters.

I asked guys from JetBrains to make inspection for IntelliJ that reports about this potential bugs, please vote.
Also I created plugin https://github.com/stokito/IntellijClashingGettersInspection

Something similar Why did PropertyDescriptor behavior change from Java 1.6 to 1.7?

6 . Groovy code may fails with java.lang.IncompatibleClassChangeError during constructing exceptions that was compiled in Java 6. Good article How to fix IncompatibleClassChangerError with groovy on jdk7
It’s very often error in Grails applications

7 . Maven is not working in Java 8 when JavaDoc tags are incomplete
You can easily find all this incomplete tags in IntelliJ by running inspection Ctrl+Alt+Shift+i «Declaration has JavaDoc problems»
Also there was few other errors in maven plugins, most of them already fixed but check them anyway
https://cwiki.apache.org/confluence/display/MAVEN/Java+8+Upgrade

8 . ActiveMQ still has a bug that occurs only in JDK 8 AMQ-5356 Unable to see message contents from the web UI

9 . You really should resolve your JAR HELL before upgrading.

Also known bug:
Xstream no-args constructor error

Tutorial: Migrating to Java 8

Migration plan:

  1. Make build stable: migrate to the latest Maven, introduce Continuous Integration.
  2. Resolve JAR hell, explicitly define all libraries and plugins versions.
  3. Improve test coverage, take care to rounding of BigDecimal, working with dates, localization (country codes and time zones),  Comparators, XML parsing,  Reflection and serialization. Integration tests for all parts of application that using external JAR’s or classes for example database  (JDBC drivers).
  4. Fix basic and common errors: clashing geters, missing @Override annotations.
  5. Upgrade all bytecode-level manipulation libraries (CGLIB, ASM, JavaAssist)
  6. Try to compile with old JDK (6, 7) but try to run with new JDK (8, 9).
  7. Run all regression tests and fix all bugs that you’ll find.
  8. Upgrade other core libraries (Hibernate, Spring, JDBC divers) and try to remove obsolete (com.sun package)
  9. Test again and fix new founded bugs.
  10. Then try to compile with new JDK (8,9) but with target language level 6. But still run with new one.
  11. Test again and fix new founded bugs.
  12. Then try to compile with new JDK (8,9) and with last target language level (8, 9). And run with last.
  13. Test again and fix new founded bugs. Migration finished