Tagged: OpenJDK

sjavac: Smart javac wrapper

I found one interesting tool that is not fairly known as it might be.
It called sjavac — Smart javac wrapper.
There is not so many information in Internet about it. But it sources contains well description. I trying to get all information in one article.

From JEP 139: Enhance javac to Improve Build Speed

The top level goals are:

  • Increase build speed by having javac use all cores and reuse javac in a server process.
  • Simplify work for developers by having javac build incrementally.

This project is part of a larger effort to improve the build infrastructure of the JDK.

The improvements to javac will be internal and not available through the public api of the javac launcher. Instead an internal wrapper program called smart-javac (or sjavac for short) will house the new functionality. Eventually, when the javac wrapper features have stabilized, they can be proposed in a future JEP to be moved to the public api of javac. This would allow all Java developers to take advantage of the improvements.

More description from source code OpenJDK com.sun.tools.sjavac.Main class:

This is a smart javac wrapper primarily used when building the OpenJDK,
though other projects are welcome to use it too. But please be aware
that it is not an official api and will change in the future.
(We really mean it!)


Create a state file, containing information about the build, so
that incremental builds only rebuild what is necessary. Also the
state file can be used by make/ant to detect when to trigger
a call to the smart javac wrapper.

This file is called bin/javac_state (assuming that you specified "-d bin")
Thus the simplest makefile is:

SJAVAC=java -cp .../tools.jar com.sun.tools.sjavac.Main
SRCS=$(shell find src -name "*.java")
bin/javac_state : $(SRCS)
                  $(SJAVAC) src -d bin

This makefile will run very fast and detect properly when Java code needs to
be recompiled. The smart javac wrapper will then use the information in java_state
to do an efficient incremental compile.

Previously it was near enough impossible to write an efficient makefile for Java
with support for incremental builds and dependency tracking.

Separate java sources to be compiled from java
sources used only for linking. The options:

"dir" points to root dir with sources to be compiled
"-sourcepath dir" points to root dir with sources used only for linking
"-classpath dir" points to dir with classes used only for linking (as before)

Use all cores for compilation by default.
"-j 4" limit the number of cores to 4.
For the moment, the sjavac server additionally limits the number of cores to three.
This will improve in the future when more sharing is performed between concurrent JavaCompilers.

** Basic translation support from other sources to java, and then compilation of the generated java.
This functionality might be moved into annotation processors instead.
Again this is driven by the OpenJDK sources where properties and a few other types of files
are converted into Java sources regularily. The javac_state embraces copy and tr, and perform
incremental recompiles and copying for these as well. META-INF will be a special copy rule
that will copy any files found below any META-INF dir in src to the bin/META-INF dir.

-copy .gif
-copy META-INF
-tr .prop=com.sun.tools.javac.smart.CompileProperties
-tr .propp=com.sun.tools.javac.smart.CompileProperties,java.util.ListResourceBundle
-tr .proppp=com.sun.tools.javac.smart.CompileProperties,sun.util.resources.LocaleNamesBundle

** Control which classes in the src,sourcepath and classpath that javac is allowed to see.
Again, this is necessary to deal with the source code structure of the OpenJDK which is
intricate (read messy).

"-i tools." to include the tools package and all its subpackages in the build.
"-x tools.net.aviancarrier." to exclude the aviancarrier package and all its sources and subpackages.
"-x tools.net.drums" to exclude the drums package only, keep its subpackages.
"-xf tools/net/Bar.java" // Do not compile this file...
"-xf *Bor.java" // Do not compile Bor.java wherever it is found, BUT do compile ABor.java!
"-if tools/net/Bor.java" // Only compile this file...odd, but sometimes used.

The smart javac wrapper is driven by the modification time on the source files and compared
to the modification times written into the javac_state file.

It does not compare the modification time of the source with the modification time of the artifact.
However it will detect if the modification time of an artifact has changed compared to the java_state,
and this will trigger a delete of the artifact and a subsequent recompile of the source.

The smart javac wrapper is not a generic makefile/ant system. Its purpose is to compile java source
as the final step before the output dir is finalized and immediately jared, or jmodded. The output
dir should be considered opaque. Do not write into the outputdir yourself!
Any artifacts found in the outputdir that javac_state does not know of, will be deleted!
This can however be prevented, using the switch --permit-unidentified-artifacts
This switch is necessary when build the OpenJDK because its makefiles still write directly to
the output classes dirs.

Any makefile/ant rules that want to put contents into the outputdir should put the content
in one of several source roots. Static content that is under version control, can be put in the same source
code tree as the Java sources. Dynamic content that is generated by make/ant on the fly, should
be put in a separate gensrc_stuff root. The smart javac wrapper call will then take the arguments:
"gensrc_stuff src -d bin"

The command line:

java -cp tools.jar com.sun.tools.sjavac.Main \
    -i "com.bar.*" -x "com.bar.foo.*" \
    first_root \
    -i "com.bar.foo.*" \
    second_root \
    -x "org.net.*" \
    -sourcepath link_root_sources \
    -classpath link_root_classes \
    -d bin

Will compile all sources for package com.bar and its subpackages, found below first_root,
except the package com.bar.foo (and its subpackages), for which the sources are picked
from second_root instead. It will link against classes in link_root_classes and against
sources in link_root_sources, but will not see (try to link against) sources matching org.net.*
but will link against org.net* classes (if they exist) in link_root_classes.

(If you want a set of complex filter rules to be applied to several source directories, without
having to repeat the the filter rules for each root. You can use the explicit -src option. For example:

sjavac -x "com.foo.*" -src root1:root2:root3  )

The resulting classes are written into bin.

See also

1. Also take a look into discussion Review request: 8004658: Add internal smart javac wrapper to solve JEP 139
2. Fredrik Öhrström created project The Smart Javac Wrapper JDK8 Backport with Extras.
3. JDK 9 will contain JEP 199: Smart Java Compilation, Phase Two that makes the sjavac tool available in the default JDK.
4. Also I would like to recommend you interest talk about incremental compilation (sorry, in Russian without subs)

Nikolay told about problems of incremental compilation and how different IDE do that.

I send a letter to Nikolay with info about sjavac and he answered:

Спасибо за информацию, я знаю про sjavac. К сожалению, он работает слишком грубо, там инкрементальность реализована на уровне package: если хоть один класс из одного пакета зависит от класса из другого пакета, то запоминается зависимость между пакетами и при изменении класса из зависимого пакета исходный пакет будет перекомпилирован целиком.
Это приведёт к тому, что при изменении класса будет перекомпилировано слишком много классов, многие из которых реально от изменённого класса не зависят. Так что в текущем состоянии для IDE это не подходит, но, правда, может быть использовано во всяких build tools, потому что лучше такая инкрементальная компиляция, чем никакой.

Translation in English:

Thanks for information, I know about sjavac.
Unfortunately, it works too dumb. Incrementational implemented on the package level: if at least one class from a package depends to class from another package, then remembered dependency between packages and on modification from dependant package source package will be totally recompiled.
It causes a lot of recompilation of not dependent classes. So it can’t be used for IDE, but can be used in build tools, because this incremental compilation is better that none.

Проблемы OpenJDK или в очередной раз про идиотизм Orcale

Ну для начала лекция для тех кому интересно въехать в курс дела:

У меня к OpenJDK есть такие претензии:
1. Название, я поначалу подумал «ну вот ето JDK а мне на сервер достаточно только JRE» и искал OpenJRE 🙂
2. Сайт: посмотрите на сайт Mono http://www.mono-project.com/ — красивый, всё ясно куда нажимать. А теперь посмотрите на это уродство http://openjdk.java.net/ . Я не дизайнер, и неприхотливый но даже меня тошнит. А ведь это же маркетинг!
4. Почему IDEA упорно не дружит с ним? Это меня настораживает — вы говорите что OpenJDK на 99% соответсвует обычной OracleJDK, но идея пишет что будут проблемы с графикой. Это правда? В чём причина? Напишите JetBrains чтобы они убрали это сообщение.
5. Где версия под Windows?
6. Отправлять патчик по мылу это прошлый век — перенесите проект на GitHub!

До сих пор у всех осталось отношение к OpenJDK как к кастрированому форку православной сановской Явы.
Боюсь чтоы избавится от этого его прийдётся переименовать и впихнуть в него что-то модное чтобы пресса рассказала о нём как о прорыве.

Обратите внимание что я говорю только о маркетиге, дитрибуции и организации проекта а не о технических деталях. В той или иной степени такие же проблемы у всех остальных проетков Оракла.