Bytedeco

Optimize and deploy models with JavaCPP and TVM

Sat, 12 Dec 2020 00:00:00 +0000

In the AI industry, the Java platform is used mainly when deploying to production. Consequently, it makes sense to provide easy access to all the tools and frameworks necessary to deploy deep learning models for Java. However, one of the most widely used frameworks, Apache TVM, did not have very good support for Java, until recently. With the JavaCPP Presets for TVM, we now have proper support for the Java platform.

Still, according to the GitHub repository of TVM, over 50% of the code is written in Python. This means that, for most use cases, we probably need to run some of those modules as part of the deployment pipeline. Moreover, there are usually preprocessing algorithms among other things that have been implemented only as Python scripts. For these reasons, we might as well make it easy as possible to execute this Python code from Java. This is where the JavaCPP Presets for CPython, LLVM, MKL, oneDNN (aka DNNL, MKL-DNN), OpenBLAS, NumPy, and SciPy come in. They are available as normal JAR files, which can be used as any other JAR files, but they can also be used to satisfy the dependencies of TVM—without having to install anything! For example, as shown in the following sample project for OpenCV, we can bundle everything with GraalVM Native Image as usual and it simply just works as expected with Quarkus:

https://github.com/bytedeco/sample-projects/tree/master/opencv-stitching-native

Importing and optimizing a model

For the moment though, let us return to the focus of this post: Importing, optimizing, and deploying deep learning models with TVM in Java. One of the most recent and important breakthrough in the industry is the transformer architecture and models based on it such as BERT, for which TVM has been shown to deliver the highest performance, at least on CPU: Speed up your BERT inference by 3x on CPUs using Apache TVM. We can embed, for example, the whole optimize_bert.py script as is in Java as demonstrated in this example, which also contains all the code found in this post:

https://github.com/bytedeco/javacpp-presets/blob/master/tvm/samples/DeployBERT.java

This script essentially imports a BERT model from GluonNLP and compares its performance with MXNet, but TVM also supports models in Keras, TensorFlow, ONNX, PyTorch, etc formats. In any case, to run that kind of script, we need to install all those Python packages first. This is where JavaCPP can help. With only these 3 lines of code, we are installing everything we need in JavaCPP’s cache, outside of any system or other user directories, so nothing gets broken, and the process is fully reproducible:

// Extract to JavaCPP's cache CPython and obtain the path to the executable file
String python = Loader.load(org.bytedeco.cpython.python.class);

// Install in JavaCPP's cache GluonNLP and MXNet to download and import BERT model
new ProcessBuilder(python, "-m", "pip", "install", "gluonnlp", "mxnet", "pytest").inheritIO().start().waitFor();

// Add TVM and its dependencies to Python path using C API to embed script in Java
Py_AddPath(org.bytedeco.tvm.presets.tvm.cachePackages());

That makes it possible to execute the example successfully with a single Maven command and nothing more than a functional installation of the JDK, with everything getting downloaded, mainly from the Maven Central Repository and PyPI:

mvn clean compile exec:java -Djavacpp.platform.host -Dexec.mainClass=DeployBERT

This outputs relevant information such as the following for an Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz with 12 cores:

MXNet latency for batch 1 and seq length 128: 57.88 ms
TVM latency for batch 1 and seq length 128: 22.45 ms
Running BERT runtime...
[ 0.06691778, 0.8569598 ]

Deploying the model

The above takes care of importing and optimizing models, but to achieve lower latency and/or higher throughput, we often need to perform inference without the additional overhead and restrictions of CPython. For this reason, after importing them, TVM can “export” models to native libraries. In this case, the DeployBERT example contains the following additional Python code to export the BERT model into a library named libbert.so:

# Export the model to a native library
with tvm.transform.PassContext(opt_level=3, required_pass=["FastMath"]):
    compiled_lib = relay.build(mod, "llvm -libs=mkl", params=params)
compiled_lib.export_library("lib/libbert.so")

Once we have this native library, we are freed from the shackles of CPython. We can load and run inference on the model, where performance matters most, using only native code, plus (in our case) Java code based on the C/C++ API of the TVM runtime, which for our libbert.so looks like this:

// load in the library
DLContext ctx = new DLContext().device_type(kDLCPU).device_id(0);
Module mod_factory = Module.LoadFromFile("lib/libbert.so");
// create the BERT runtime module
TVMValue values = new TVMValue(2);
IntPointer codes = new IntPointer(2);
TVMArgsSetter setter = new TVMArgsSetter(values, codes);
setter.apply(0, ctx);
TVMRetValue rv = new TVMRetValue();
mod_factory.GetFunction("default").CallPacked(new TVMArgs(values, codes, 1), rv);
Module gmod = rv.asModule();
PackedFunc set_input = gmod.GetFunction("set_input");
PackedFunc get_output = gmod.GetFunction("get_output");
PackedFunc run = gmod.GetFunction("run");

// Use the C++ API to create some random sequence
int batch = 1, seq_length = 128;
DLDataType dtype = new DLDataType().code((byte)kDLFloat).bits((byte)32).lanes((short)1);
NDArray inputs = NDArray.Empty(new long[]{batch, seq_length}, dtype, ctx);
NDArray token_types = NDArray.Empty(new long[]{batch, seq_length}, dtype, ctx);
NDArray valid_length = NDArray.Empty(new long[]{batch}, dtype, ctx);
NDArray output = NDArray.Empty(new long[]{batch, 2}, dtype, ctx);
FloatPointer inputs_data = new FloatPointer(inputs.accessDLTensor().data()).capacity(batch * seq_length);
FloatPointer token_types_data = new FloatPointer(token_types.accessDLTensor().data()).capacity(batch * seq_length);
FloatPointer valid_length_data = new FloatPointer(valid_length.accessDLTensor().data()).capacity(batch);
FloatPointer output_data = new FloatPointer(output.accessDLTensor().data()).capacity(batch * 2);
FloatIndexer inputs_idx = FloatIndexer.create(inputs_data, new long[]{batch, seq_length});
FloatIndexer token_types_idx = FloatIndexer.create(token_types_data, new long[]{batch, seq_length});
FloatIndexer valid_length_idx = FloatIndexer.create(valid_length_data, new long[]{batch});
FloatIndexer output_idx = FloatIndexer.create(output_data, new long[]{batch, 2});

Random random = new Random();
for (int i = 0; i < batch; ++i) {
    for (int j = 0; j < seq_length; ++j) {
        inputs_idx.put(i, j, random.nextInt(2000));
        token_types_idx.put(i, j, random.nextFloat());
    }
    valid_length_idx.put(i, seq_length);
}

// set the right input
setter.apply(0, new BytePointer("data0"));
setter.apply(1, inputs);
set_input.CallPacked(new TVMArgs(values, codes, 2), rv);
setter.apply(0, new BytePointer("data1"));
setter.apply(1, token_types);
set_input.CallPacked(new TVMArgs(values, codes, 2), rv);
setter.apply(0, new BytePointer("data2"));
setter.apply(1, valid_length);
set_input.CallPacked(new TVMArgs(values, codes, 2), rv);
// run the code
run.CallPacked(new TVMArgs(values, codes, 0), rv);
// get the output
setter.apply(0, 0);
setter.apply(1, output);
get_output.CallPacked(new TVMArgs(values, codes, 2), rv);

We can of course implement a high-level API on top of all this to simplify the usage, as with the official Java API called TVM4J, which also comes bundled with the presets, so that works out of the box too, but it is quite limited and not always practical. Whichever API we use though, it should be clear that all the hard work can be done from the Java platform, in a portable and reproducible fashion, without having to install or build anything manually on the system.

For CUDA on GPUs, we can also do something similar with the JavaCPP Presets for TensorRT, so be sure to check out this API as well along with the sample code written in Java:

https://github.com/bytedeco/javacpp-presets/blob/master/tensorrt/samples/SampleGoogleNet.java

If you are having any issues with the code or would like to talk more about this subject, please feel free to communicate via the mailing list from Google Groups, discussions on GitHub, or the chat room at Gitter. Happy New Year!

Beyond Java and C++

Thu, 11 Apr 2019 00:00:00 +0000

We are happy to announce the first release of Bytedeco in this new Japanese era! Version 1.5 is available for JavaCPP, JavaCPP Presets, JavaCV, ProCamCalib, and ProCamTracker. This release comes with new presets for NumPy (yes, the Python library, more on that below), NCCL, nGraph, Qt (providing an alternative to AWT, Swing, and JavaFX), and cpu_features, as well as updates for FFmpeg (now also including the ffmpeg and ffprobe programs themselves), libfreenect, HDF5, MKL, MKL-DNN, LLVM, Leptonica, ARPACK-NG, CUDA, cuDNN, MXNet, TensorFlow, TensorRT, ONNX, LiquidFun, and Skia. Many thanks to all contributors! Since the previous post just 3 months ago, according to git shortlog 1.4.4..1.5 --summary, they were:

Alex Merritt, Arnaud Jeansen, Greg Hart, HGuillemet, k3rnL, Samuel Audet, Simon Harris

However, as usual, this list does not contain anyone who very generously helped fix CI settings, test builds, debug code, file suggestions, make corrections, report bugs, propose ideas, update wiki pages, etc. One can find more detailed information about all these changes and fixes on the repositories listed above in the CHANGELOG.md files, among other places.

As hinted previously though, the most important update concerns the modularization of all the artifacts to comply with the Java Platform Module System (JPMS). In theory, we simply have to add irritating module-info.java files everywhere (we are grateful to ModiTect for the help with that), but one major restriction of JPMS is that we cannot split a Java package across multiple modules, so we had to update the packages of all the presets to comply, thus the minor version bump from 1.4.x to 1.5.x. In no small part thanks to Hervé Guillemet, we eventually managed to come up with a satisfactory naming scheme that not only complies with JPMS, but also produces top-level classes that are closer to typical Java APIs and that Javadoc can render more clearly into HTML. Instead of putting all classes in the org.bytedeco.javacpp package, all modules now have their own packages based on the names of their artifacts, for example, org.bytedeco.opencv in the case of OpenCV. Inside such a package we have top-level classes, further divided into subpackages when the artifact contains multiples native libraries, as well as a global subpackage containing, you guessed it, classes with native declarations for global C/C++ functions and variables. For most downstream projects, upgrading requires modifications to the import statements only. In the case of OpenCV and TensorFlow, for example, we would typically have statements like the following:

import static org.bytedeco.javacpp.opencv_core.*;
import static org.bytedeco.javacpp.opencv_imgproc.*;
import static org.bytedeco.javacpp.opencv_calib3d.*;
import static org.bytedeco.javacpp.opencv_objdetect.*
import static org.bytedeco.javacpp.tensorflow.*;

Given these exact statements, to migrate to the new API, the only change required would be to replace them with the following:

import org.bytedeco.opencv.opencv_core.*;
import org.bytedeco.opencv.opencv_imgproc.*;
import org.bytedeco.opencv.opencv_calib3d.*;
import org.bytedeco.opencv.opencv_objdetect.*;
import org.bytedeco.tensorflow.*;
import static org.bytedeco.opencv.global.opencv_core.*;
import static org.bytedeco.opencv.global.opencv_imgproc.*;
import static org.bytedeco.opencv.global.opencv_calib3d.*;
import static org.bytedeco.opencv.global.opencv_objdetect.*;
import static org.bytedeco.tensorflow.global.tensorflow.*;

To prevent conflicts with the old API, we also changed the groupId of all artifacts to org.bytedeco, so no more annoyingly long names containing org.bytedeco.javacpp-presets, which also reflects our evolving technical understanding of the world where the JVM cannot rule them all.

Java is able to abstract away the underlying operating system, and, along with tools such as Maven and now jlink to create compact self-contained standalone applications as per the opencv-stitching-jlink sample project, its abilities to manage the complexity of software in the ecosystem is unparalleled. However, the JVM itself is failing miserably at supporting other existing programming languages. We obviously need access to native libraries written in C/C++ for compute intensive algorithms, but even popular scripting languages such as JavaScript and Python have never been able to feel at home on the JVM either, notwithstanding their own arguably inferior native runtimes. Jython is stuck at Python 2.7, while Nashorn has been deprecated even before its proposed successor, Graal JavaScript, is ready to take its place. Likewise, Graal Python is truly awesome technology, but is still an “early-stage experimental implementation” that runs pretty much nothing useful yet, let alone CUDA or any kind of support for GPU no where to be found on any roadmap. Whatever the cause is for the lack of the appearance of a universal VM that a majority of users would be happy with, it is instead possible to have multiple runtimes executing in the same process to share data efficiently, an approach that is also future-proof given that Graal still needs to implement JNI and the C API of Python anyway to be compatible with tools such as JNA, JNR, or JavaCPP, and libraries like NumPy or TensorFlow. We show that an initial implementation can be easily achieved today in the case of CPython and the current JDK, starting with the JavaCPP Presets for NumPy, which leaves all the high-level dependency management to Java and Maven, where JavaCPP automatically extracts the Python package to its cache, along with MKL on supported platforms, in a portable and repeatable fashion, without the need of any other less portable tools including Docker. In effect, this makes Python and NumPy available as a sort of domain-specific scripting language to Java developers. We plan on making more Python packages available this way, continuing next with the JavaCPP Presets for TensorFlow.

If you would like to participate in this effort, please communicate via the mailing list from Google Groups, issues on GitHub, or the chat room at Gitter. In any case, stay tuned for more exciting developments later this year!

The importance of a distribution

Fri, 11 Jan 2019 00:00:00 +0000

Happy New Year! We were able to deliver yet another release on time. Version 1.4.4 is available for JavaCPP, JavaCPP Presets, JavaCV, ProCamCalib, and ProCamTracker. Many thanks to all contributors! Since the previous post, according to git shortlog 1.4.2..1.4.4 --summary, they were:

Alex Merritt, Aman Gupta, Ao Qi, bitstormGER, deinhofer, Deividi, eguid, EmergentOrder, Gertjan Al, HGuillemet, Jarek Sacha, Jeremy Apthorp, kigkrazy, lloydmeta, louxiu, nahojjjen, Nico Hezel, renderdude, Samuel Audet, Taha Emara, vimalaguti, vincent-grosbois, wumo, Yuta Okamoto, Zayin Krige

Many thanks as well to those not on this list but who helped fix CI settings, test builds, debug code, file suggestions, make corrections, report bugs, propose ideas, update wiki pages, etc. As usual this release fixes many issues and contains a lot of updates, all the details can be found in the CHANGELOG.md files, but one important change concerns MXNet, whose Scala API is now partially usable from Java. Consequently, the JavaCPP Presets for MXNet now also bundle the official Scala API, just as with the official Java APIs of OpenCV and TensorFlow. Given these developments, there is one question that keeps coming up over and over again: Why do we need a distribution like Bytedeco even if everyone shows up with Java APIs on their own?

Well, for one thing, the feature set exposed by these official APIs for Java is very limited, so many users still require access to the C/C++, Scala, or even Python APIs. Moreover, PyTorch is slated to become the first deep learning framework with a full-featured and easy-to-use C++ API, but with no plans in sight for Java, so once the C++ API becomes usable we intend on integrating it to Bytedeco as part of the JavaCPP Presets, see issue bytedeco/javacpp-presets#623.

Another reason could be that JavaCPP provides a set of basic classes as foundation for native libraries, such as PointerScope introduced in the previous post. Thanks to Hervé Guillemet and Sam Carlberg, they will also soon support the Java Platfom Module System (JPMS), including jlink, something that the upstream projects have no plans to provide for their Java APIs. (A preview version of OpenCV is already available on the jpms branch with corresponding snapshot artifacts.) Although Project Panama promises to offer most of the features that JavaCPP already provides today, delivery is expected only in a few more years, numbers showing faster-than-JNI performance for both JIT and AOT compilers are still lacking, development for C++ has yet to start, and it does not aspire to support platforms such as Android and iOS. Though, even in the case of plain Java SE, a loader for native libraries is required, something that community members such as Johan Vos understand, but there are no plans to integrate such functionality to OpenJDK. Unless a fork like Corretto happens to stimulate evolution towards the demands of contemporary software development, I am afraid the need for a third-party tool like JavaCPP is here to stay for the foreseeable future.

Nevertheless, assuming that Project Panama succeeds and renders JavaCPP obsolete—which would be awesome, although doubtful based on the discussion above, but just for the sake of the argument—the need for a distribution like Bytedeco will remain. The redistributable binaries for CUDA are over 2 GB for all supported platforms, compressed, and similarly for MKL, which is about 0.5 GB for all supported platforms. These include aggressively optimized implementations of BLAS, LAPACK, and FFTW, among many other things, achieving at least 10× speedups over anything one could possibly write in pure Java today. (Project Panama also promises to fix this for CPUs, eventually, but not for GPUs.) Their files are currently bundled by Bytedeco and shared by, for instance, Deeplearning4j, and the JavaCPP Presets for OpenCV, Caffe, MXNet, and TensorFlow. To offer a user-friendly experience, if each of these projects were to start bundling CUDA and MKL for all platforms on their own, an application depending on all those libraries may very well end up bundling over 10 GB of duplicate code! Projects competing against each other either downstream or upstream cannot offer such redistributables as shared resources: That is the role of a distribution. The goal is to make all those libraries, including the JDK itself, work well together.

Right now, unless we are misinformed, Bytedeco is the only such distribution available for Java, however small it may be, which is indeed quite sad, but I hope this post helps spur more cooperation or, at the very least, some competition. There is still much to be done before the community at large begins to recognize the clear necessity for Java distributions of native libraries that work not only with Java SE on Linux, Mac, and Windows, but also for mobiles and embedded platforms such as Android, iOS, and Raspberry Pi.

More concretely, to attain these goals, we will have to start putting more resources in priority into the following efforts:

Obtain more visibility, gain a wider sense of community, by
- Writing more blog posts, on a fancier web site,
- Meeting people in person at conferences, workshops, etc, and
- Participating in upstream projects, driving acceptance by their developers and users, by
  - Reporting bugs and helping to fix them, and even by
  - Contributing and maintaining JavaCPP-based wrappers upstream, among other things;
Aim for fully automated bindings generation for the features of C++ most widely used by native libraries,
- In all probability with the help of libclang, see issue bytedeco/javacpp#51 and the samples in Java; and
Keep creating new presets for additional native libraries, plus maintain them up-to-date with upstream projects.

A lot of help is going to be required to realize all this, so please spread the word! To contribute yourself to one of the items listed above, please communicate via the mailing list from Google Groups, issues on GitHub, or the chat room at Gitter. We are looking forward to continue working with everyone on all these projects this year as well!

Bytedeco as a distribution

Tue, 17 Jul 2018 00:00:00 +0000

With version 1.4.2 now released, it’s time to publish a small blog post introducing the main innovations. As everyone probably knows, we’ve also released version 1.4.1 back in March. Thanks to the CI servers at AppVeyor and Travis CI, we are now able to produce full releases of all binaries without too much effort, so we will probably be maintaining a release cycle of about 3 months. This brings Bytedeco one step closer to a distribution such as Anaconda, Fedora, or Ubuntu, but cross-platform and for multiple languages (that is being able to access native libraries from other languages than C++), where the presets for OpenCV and TensorFlow, for example, now also bundle the official Java APIs, but still provide users with access to the C/C++ APIs as well as additional features for better integration. PointerScope is one such new feature that can help manage native memory, even when using multiple different APIs together.

Until now, to prevent garbage from hanging around too long in memory, we had to call manually Pointer.deallocate() on all native objects (either directly or with try-with-resources statements). We can now use a PointerScope to manage a group of such objects, which is similar to workspaces in Deeplearning4j (but without the workspace memory) or Scope from Panama, and as exemplified below for OpenCV and TensorFlow, a single PointerScope works even across libraries:

    static void checkStatus(Status s) {
        if (!s.ok()) {
            throw new RuntimeException(s.error_message().getString());
        }
    }

    static void detectObjects(File file, Session session) throws Exception {
        // load the image file
        Mat mat = imread(file.getCanonicalPath());
        if (mat == null || mat.empty()) return;
        cvtColor(mat, mat, CV_BGR2RGB);
        Tensor tensor = new Tensor(DT_UINT8,
            new TensorShape(1, mat.rows(), mat.cols(), mat.channels()),
            mat.arrayData().capacity(mat.arraySize()));

        // run inference
        String[] inputNames = {"image_tensor"};
        Tensor[] inputTensors = {tensor};
        String[] outputNames = {"detection_boxes", "detection_scores", "detection_classes", "num_detections"};
        TensorVector outputTensors = new TensorVector();
        checkStatus(session.Run(new StringTensorPairVector(inputNames, inputTensors),
                new StringVector(outputNames), new StringVector(), outputTensors));

        // do something with outputTensors...
    }

    public static void main(String args[]) throws Exception {
        // read the model
        GraphDef graph = new GraphDef();
        checkStatus(ReadBinaryProto(Env.Default(), "/path/to/frozen_inference_graph.pb", graph));

        // create the session
        Session session = new Session(new SessionOptions());
        checkStatus(session.Create(graph));

        // detect objects from all files in directory
        File dir = new File("/path/to/images/");
        for (File f : dir.listFiles()) {
            try (PointerScope scope = new PointerScope()) {
                detectObjects(f, session);
            }
            System.out.println(Pointer.physicalBytes());
        }
    }

When executing this code with opencv-platform, tensorflow-platform, and the model file from ssd_inception_v2_coco_2017_11_17.tar.gz, the output shows that memory usage rapidly stabilizes around 1 GB:

However, if we take the PointerScope out of the loop, it goes on to fill up all the memory that we give it, which can easily exceed 16 GB by default on today’s machines:

This is good not only to prevent wasting resources, but also to limit memory fragmentation. If you are interested in using TensorFlow this way, thanks to Nico Hezel, who also fixed the build for TensorFlow with CUDA support on Windows, additional samples are now available at:

https://github.com/bytedeco/javacpp-presets/tree/master/tensorflow/samples

Finally, other important changes include:

New artifacts for CUDA and cuDNN containing their redistributable libraries similarly to NVIDIA at Docker Hub, but that work also on Mac and Windows, to allow other libraries (Deeplearning4j, OpenCV, Caffe, TensorFlow, etc) to use GPUs without installing CUDA:
- https://github.com/bytedeco/javacpp-presets/tree/master/cuda
New presets for ARPACK-NG, CMINPACK, TensorRT, and MKL-DNN (which bundles a free version of MKL that other presets can use via OpenBLAS),
64-bit builds for Android (android-arm64 and android-x86_64) and iOS (ios-arm64 and ios-x86_64) that work with both Gluon VM and RoboVM,
New LeptonicaFrameConverter for Tesseract and JavaFXFrameConverter for JavaFX (thanks to Johan Vos), and
Support for audio frames in FFmpegFrameFilter bringing it on par with FFmpegFrameGrabber and FFmpegFrameRecorder.

Please find more information about these updates and other changes in the CHANGELOG.md files for JavaCPP, JavaCPP Presets, JavaCV, ProCamCalib, and ProCamTracker. Many thanks to all contributors! According to git shortlog 1.4..1.4.2 --summary they are:

agiasmothi, Alex Black, Aman Gupta, canelzio, delthas, gunther82, Jeremy Apthorp, Jeremy Laviole, Johan Vos, Kevin Watters, louxiu, Mariano Scazzariello, Michael Haslgrübler, mifritscher, Nico Hezel, SIY1121, Sören Brunk, Stanislav Chizhik, Vincent Baines (vb216)

For completeness, here are all contributors since the beginning according to git shortlog --summary:

Adam Gibson, agiasmothi, Agustin Alvarez, Alex Black, alicana, Aman Gupta, Amnon Owed, Amos Yuen, Andreas Eberle, Arseniy Pendryak, Ashley, beligum, Benjamin Hill, bennyveo, canelzio, Chetan Narsude, Chris Chow, Chris Nuernberger, Cyprien Noel (cypof), delthas, Djalma Antonio dos Passos Filho, Dmitriy Gerashenko, dragos-d, Edison Wang, Edu Garcia, egorapolonov, emilio1986, Fabrizio (Misto) Milo, Felix Andrews, Florian Enner, François Garillot, Fredrik Henricsson, Gabor Nagy, George Kankava, Gertjan Al, GroG (supertick), gunther82, HungHD, Jadon Fowler, Jarek Sacha (jarek), Jeremy Apthorp, Jeremy Laviole, Jeremy Martin, Jeroen Arens, jjYBdx4IL, Johan Vos, John Sichi, Jonathan Leitschuh, juergenuhl, kenji yoshida, Kevin Watters, leef, lfdversluis, Lloyd Chan (lloydmeta), louxiu, Luca Barbato, mabruce, Mariano Scazzariello, Martin, matrupihai, Maurice, Maurice Betzel (mbetzel), Michael Dietz, Michael Haslgrübler, Michael Tandy, Michal Conos, mifritscher, Miroslav Zoričák, mmanco, Mo Tao, mtandy, Nico Hezel, n-kai-cj, Oleg Kuliasov, Pachev Joseph, Paolo Bolettieri (paolobolettieri), Paul Gregoire, Paul Hammant, Paul Heideman, Pawel Hajduk, pchundi, Perry Nguyen, Philipp Moritz, Piotr, Richard Alam, Ryan Osial, Sam Carlberg, Samuel, Sean Carlisle, SIY1121, Sören Brunk, Stanislav Chizhik, Tatsuyuki Ishi, Teerapap Changwichukarn, The Gitter Badger, Tiago Daniel Jacobs, Tom Clark, Trejkaz (pen name), TwistedUmbrella, Vincent Baines (vb216), waldemarnt, Yizhi Liu, zhanhb, wmz7year, 李立强

This list does not include anyone who has helped in other ways, including but not limited to fixing CI settings, testing builds, debugging code, filing issues, making corrections, reporting bugs, proposing ideas, updating the wiki, etc, but please do continue to communicate with us via the mailing list from Google Groups, issues on GitHub, or the chat room at Gitter. Your input is valuable, we hope to continue hearing from all of you!

Java as a system programming language

Wed, 17 Jan 2018 00:00:00 +0000

Happy New Year! After spending over a year in (part-time) preparation for the next major release here at Bytedeco, version 1.4 has finally been released! A lot has been happening, so let me summarize the most important items. First, a million thanks to Vincent Baines for all the hard work getting builds to pass. We now have a proper continuous integration (CI) infrastructure based on AppVeyor and Travis CI testing builds for pull requests as well as publishing SNAPSHOT artifacts for all platforms at each commit to the source code repositories. More information about that on the builds page. Next, we have introduced the concept of “extension” to JavaCPP, letting us provide separate but optional CUDA builds for OpenCV, Caffe, and TensorFlow. To enable them, one simply needs to add to the class path the JAR files containing “-gpu” in their names, and JavaCPP will automatically pick them up based on their contents. On load error, it also gracefully fall backs on non-CUDA binaries. The list of currently available CUDA artifacts is given at the bottom of the download page. Finally, we have been busy closing the gap that prevents Java from being usable as a system programming language by introducing the JavaCPP Presets for Systems to access system APIs such as libc and Win32.

Before going into more details on the latter topic at hand, please find a complete list of all changes in the CHANGELOG.md files for JavaCPP, JavaCPP Presets, JavaCV, ProCamCalib, and ProCamTracker. Binaries can be obtained as usual from the Maven Central Repository. New presets that were contributed include libfreenect2, MKL, libpostal, The Arcade Learning Environment (ALE), LiquidFun, and Skia, in addition to the system APIs.

Although Java is still the most widely used programming language, according to the TIOBE Index, among many other sources, it is still not considered a system programming language and is conspicuously missing from the list of system programming languages on Wikipedia. However, things are starting to change. Beginning with Java 9, the JDK supports ahead-of-time (AOT) compilation of Java classes, something otherwise supported by Avian and many others for a while already, thus allowing developers to create native executable programs that can be integrated into operating systems. Finally, thanks to JNI, JavaCPP, and now the systems presets, we can benefit easily from all the features (and suffer from all the pitfalls) of C++ right from the Java platform.

Having access to systems APIs allows us to perform any operation supported by the underlying platform that is not otherwise mapped to a high-level Java API. Before the process API updates included in Java 9, it was not possible to query all children and descendants of a process, or to kill forcibly an arbitrary process on the system, but it could still have been done with native APIs. However, it is still not possible, for example, to set the priority of a process on the system without resorting to external tools. With the systems presets, we can accomplish this with just a few lines of code, such as the following to set the current process priority to the lowest level, code that can also be executed interactively in a JShell session:

    import org.bytedeco.javacpp.*;

    String platform = Loader.getPlatform();
    if (platform.startsWith("linux")) {
        linux.setpriority(linux.PRIO_PROCESS, linux.getpid(), 20);
    } else if (platform.startsWith("macosx")) {
        macosx.setpriority(macosx.PRIO_PROCESS, macosx.getpid(), 20);
    } else if (platform.startsWith("windows")) {
        windows.SetPriorityClass(windows.GetCurrentProcess(), windows.IDLE_PRIORITY_CLASS);
    }

We can perform the same calls with other tools such as JNA or JNR, but with JavaCPP we get a uniform layer that also supports C++ libraries, providing a level of integration as yet unmatched by any other solutions that we are aware of on any platform. In a nutshell, any language targeting Java bytecode is now in a good position to become a system programming language, on the same level as the Go language!

We hope that you are excited as we are in participating in this grand experiment, so please do not hesitate to contact us via the mailing list from Google Groups, issues on GitHub, or the chat room at Gitter, for any questions that you may have. We hope to hear from all of you soon!

Refining native integration

Wed, 07 Dec 2016 00:00:00 +0000

Over the past few months, I have been busy integrating JavaCPP to Deeplearning4j, mainly as part of ND4J and DataVec. Many bugs had to be fixed, we also needed to enhance native memory management, a few presets were created, but I actually spent most of my time trying to run Skymind in Japan. In any case, thanks to all the team, Deeplearning4j has come a long way since last year, so if you have not given it a try recently, make sure you do!

However, a release at Bytedeco was long overdue, so I am proud to announce the availability of version 1.3! The source code and the binaries can be obtained as usual from GitHub and the Maven Central Repository for JavaCPP, JavaCPP Presets, JavaCV, ProCamCalib, and ProCamTracker. This release comes with binaries for linux-armhf (for devices such as Raspberry Pi), thanks to Vince Baines for his continuous effort, as well as linux-ppc64le, built on SuperVessel Cloud, which features virtual machines that IBM offers for free to the community. Lloyd Chan has also been maintaining sbt-javacpp and sbt-javacv, while Andreas Eberle has generously provided Android builds for TensorFlow.

To manage native memory more automatically, JavaCPP now monitors physical memory usage, also known as “resident set size” on Linux, Mac OS X, etc or “working set size” on Windows, as reported by the kernel. The maximum value defaults to 2 times Runtime.maxMemory(), which can be specified with the usual -Xmx option on the command line, but we can also set it independently via the “org.bytedeco.javacpp.maxphysicalbytes” system property. When the whole process uses more physical memory than that amount, System.gc() followed by Thread.sleep(100) are called a few times in a row, an amount adjustable via the “org.bytedeco.javacpp.maxretries” system property, in an attempt to free memory. With this strategy, we are able to tame memory usage more accurately no matter how it is being allocated.

The presets that were created for this release are librealsense (a great contribution from Jeremy Laviole), HDF5 (that Deeplearning4j uses to import models from Keras, Theano, and TensorFlow), and OpenBLAS (which also dynamically binds to MKL if found on the system or in the class path). Additions planned for the near future include MAGMA to supplement functionality missing from cuSOLVER. To simplify further the user experience, we also plan to offer bundles containing all the binaries for CUDA and MKL, if it is determined that we are allowed to do so, which appears likely according to the EULAs that accompany their free downloads (CUDA and MKL). Traditionally, we have to spend time either installing them manually or figuring out a way to automate the process on a case-by-case basis, using when possible platform-specific package managers or containers such as Docker, probably along with some scripts. Having such bundles available on the Maven Central Repository would relieve developers and operators from this burden.

Other important changes include a HalfIndexer to process in Java 16-bit half-precision floating-point data from CUDA or other libraries, the adoption of a user defined directory (defaults to ~/.javacpp/cache/) where JavaCPP now caches native library files, instead of extracting them into a temporary directory, and the introduction of “platform artifacts” for the JavaCPP Presets and JavaCV. Each entry in the cache is a directory with the same name as the JAR file from which the files are extracted, including the subdirectories, for example, opencv-3.1.0-1.3-linux-x86_64.jar/org/bytedeco/javacpp/linux-x86_64/libopencv_core.so.3.1. This way, the files are given a (most of the time) unique, predetermined, but easy to remember path, preventing not only the build up of messy temporary files, but also allowing for faster startup times as well as easier integration with native tools, outside the scope of JavaCPP. The technique also works for files other than libraries. Right now, the implementation does not support the extraction of whole directories, but when that becomes possible, one will be able to bundle header files, among other native resources, and have them available for immediate consumption with such a simple call as Loader.cacheResource(opencv_core.class, "include"). As a matter of course, the cache functions just as smoothly with uber JARs. “Platform artifacts” can also come in handy. Users are invited to add dependencies on those artifacts suffixed with “-platform”, for example, javacv-platform, opencv-platform, ffmpeg-platform, etc, which in turn depend on binaries for all supported platforms. This new strategy was designed to work well with build systems other than Maven (sbt, Gradle, M2Eclipse, etc).

On a final note, before long, we hope to have build servers running allowing us to make releases available in a more timely fashion. Stay tuned for updates, but in the meantime, do not hesitate to contact us through the mailing list from Google Groups, issues on GitHub, or the chat room at Gitter, for any questions that you may have. Contributions are also very welcome!

Going deeper into deep learning

Sun, 15 May 2016 00:00:00 +0000

After more than half a year, we are finally making a release! You can obtain the new version 1.2 at the usual places on GitHub and the Maven Central Repository for JavaCPP, JavaCPP Presets, JavaCV, ProCamCalib, and ProCamTracker. For Scala users, Lloyd Chan has also contributed sbt-javacpp and sbt-javacv, offering them easy-to-use plugins for sbt. Thanks to Vince Baines, this release also contains a few binaries for the linux-armhf platform, which work on most Raspberry Pi devices, among others. We also hope to have continuous integration (CI) set up before long to provide a larger selection of prebuilt binaries on all platforms for non-release versions as well.

With regards to deep learning, we realized last year that JavaCPP had the characteristics of something that was in demand especially in that field, that is to say an easy way to access native libraries from an efficient platform like Java: Java meets Caffe, deep learning in perspective. Since then, Samuel has switched jobs and now works for Skymind, integrating JavaCPP, the JavaCPP Presets, and JavaCV into ND4J and Deeplearning4j, as well as pursuing other avenues, such as maintaining not only bindings for Caffe and cuDNN, but also for MXNet and TensorFlow, among others. Many thanks to Adam Gibson and Chris Nicholson for their trust! I am sure we will achieve great things together.

That said, to attain accuracies higher than traditional methods, deep learning requires a lot of data. That makes it a good candidate for data processing in big data applications. Naturally, they consume large amounts of memory, but Java cannot access arrays larger that can be indexed with a 32-bit int variable. This is an inherit limitation of the JVM. Given that Hadoop on Java is the de facto standard when it comes to big data applications, it makes sense to have a solution to that limitation. To support 64-bit indexing, we have thus extended the position, limit, and capacity fields of the Pointer class to long. Moreover, since standard NIO buffers do not support long indexing, we provide a new backend for the indexer package using sun.misc.Unsafe. Indexing memory with long variables represents a fundamental shift in the API, so it might break some existing code, but nothing too dramatic, we hope. On the brighter side, performance on 64-bit architectures is not affected.

Indexer also now implements AutoCloseable, to obtain the same benefit as with Pointer and try-with-resources constructs. For applications that cannot take advantage of this for memory management purposes, Pointer now also tracks the amount of memory allocated as reported by the capacity field. This does not work for most native libraries, more work would be required to query memory consumption from the operating system, but it works when allocating arrays of simple types with allocateArray(). Once memory consumption tracked this way reaches Pointer.maxBytes, the allocator will do its best to reclaim memory by calling System.gc(), waiting a bit, and retrying a few times in a loop.

That about covers the essential changes in this release, but other things have been fixed and updated too, so we invite you to check the changelogs and to contact us through the mailing list from Google Groups, issues on GitHub, or the chat room at Gitter, for any questions that you may have. Together, let’s make the future happen!

Moving to Java SE 7

Sun, 25 Oct 2015 00:00:00 +0000

A few months have already passed since the last release, encouraging us to create a new one, which we have just done! Version 1.1 of JavaCPP, JavaCPP Presets, JavaCV, ProCamCalib, and ProCamTracker are now available, and they depend on Java SE 7. We felt that since everyone is now comfortable with Java SE 8, it was time to drop compatibility with Java SE 6. :) Seriously, the main reason behind the change was to provide support for the try-with-resources construct, instead of relying solely on the garbage collector to imitate C++-like memory management semantics, aka “Resource Acquisition Is Initialization (RAII)”. This required making the Pointer class implement the AutoCloseable interface, a tiny but important interface. Many thanks to Adam Gibson and Cyprien Noel for having successfully convinced me to make this change. Android also provides that interface since version 4.0, covering more than 95% of all Android users at the time of this writing, according to Android Dashboards. So, the choice was not so hard to make after all, and JavaCPP now requires Android 4.0.

That said, what does try-with-resources gets us anyway? It provides a user-friendly way to release resources in a deterministic manner, something that a lot of C++ libraries take for granted. There is no guarantees that the garbage collector will ever run. It only cares about memory allocated on the Java heap, and if we allocate chunks of memory only from the native heap, it might decide to never run, a situation that can easily become problematic. Also, we cannot predict on which thread memory eventually gets deallocated. We cannot force any given thread used for an allocation to come back at some point for deallocation: It might disappear at any time without any warning. The point is, for more than a few reasons, people came up with this concept to provide services garbage collection was unable to provide. As a concrete example, in the traditional paradigm, there are occasions when we would write the following:

    MyPointer data = new MyPointer();
    try {
        // use data...
    } finally {
        data.deallocate();
    }

With try-with-resources, we can rewrite the logic above into a single statement like this:

    try (MyPointer data = new MyPointer()) {
        // use data ...
    }

A lot cleaner, and thus a lot less error-prone. Exceptions also behave better, but that is about the gist of it. By the way, this also works with native smart pointers such as boost::shared_ptr (Boost), std::shared_ptr (C++11), cv::Ptr (OpenCV), and others. JavaCPP provides a new transparent bridge to any such references by holding on to a set of two pointers for each native object: the actual pointer of the object to access the data and call functions, and an “owner pointer” corresponding to a smart pointer allocated on the heap, which gets accessed only during deallocation when applying the delete operator. Check the documentation of the @SharedPtr annotation and the associated C++ template SharedPtrAdapter for more information.

On top of that, Pointer now also checks for the “org.bytedeco.javacpp.nopointergc” system property, and if “true”, disables completely the garbage collector safety net, potentially saving a significant amount of time for some applications. To make sure that yours does not rely on the garbage collector for any deallocation, we have also added debug messages that are logged when a Pointer gets allocated, collected as garbage, and eventually deallocated. They can be captured by setting the “org.bytedeco.javacpp.logger” system property to “slf4j”, and by setting the log level to “debug”, for example, via the “org.slf4j.simpleLogger.defaultLogLevel” system property in the case of SimpleLogger from SL4J.

In closing, let us give credit where credit is due. Cyprien Noel was the one that not only brought up all these ideas, but helped to implement them, and tested them out for their own framework of Caffe-on-Spark. Discussions with Adam Gibson were also extremely fruitful, providing valuable feedback. They have since adopted JavaCPP for their own use as part of Deeplearning4j. Though grateful to them and everyone else who helped out, there is still much work to be done. If you are interested by where this is going, or would like to make your own recommendations and contributions, be sure to contact us via the mailing list from Google Groups, issues on GitHub, or the chat room at Gitter. Thank you for your interest, and enjoy!

Now offering software under the Apache License

Sun, 12 Jul 2015 00:00:00 +0000

To facilitate its adoption in commercial applications such as Deeplearning4j, the core software of JavaCPP, JavaCPP Presets, and JavaCV has been relicensed to offer users the Apache License, Version 2.0 as a new choice of license. No large technical changes has otherwise occurred, but we believe these core components have acquired a stable enough existence to merit an increment in their major version number to 1.0. Moreover, this way we keep a meaningful relationship between our version numbers and the ones of OpenCV, which has recently seen large changes in its API and has been updated to version 3.0.

In other news, we now provide presets for CUDA, mapping close to the entirety of its API, including cuDNN, for which no other wrappers that we are aware of currently exist. We hope that, along with the presets for Caffe, which are as far as we know also still a unique offering of ours, this will provide some useful tools to the new exciting world of deep learning.

As usual, if you have any questions, problems, or would like to contribute, but are unsure how to proceed, please feel free to share your concerns on the mailing list of JavaCPP, on the one for JavaCV, or via “issues” on GitHub. We are looking forward to work with you in the future. Have fun!

Introducing JavaCV frame converters

Sat, 04 Apr 2015 00:00:00 +0000

To supplement the long-standing functionality of CanvasFrame, FrameGrabber, FrameRecorder, and their implementation classes found in JavaCV, we are introducing the concept of FrameConverter starting with version 0.11, released today. Other more minor changes have been made to all the components since version 0.10, so be sure to check out the repositories on GitHub: JavaCPP, JavaCPP Presets, JavaCV, ProCamCalib, and ProCamTracker.

A FrameConverter, in short, allows the user to share easily the same audio samples or video image data among different APIs, but without coupling their applications to all of those APIs. For example, JavaCV currently ships with AndroidFrameConverter, Java2DFrameConverter, and OpenCVFrameConverter to let users represent image data as either android.graphics.Bitmap, java.awt.image.BufferedImage, IplImage, or Mat. The plain old data class adopted by JavaCV is Frame, which does not itself depend on either Android, FFmpeg, Java 2D, or OpenCV. A user could for example grab and record frames using FFmpegFrameGrabber and FFmpegFrameRecorder, without requiring anything more than FFmpeg. We could further display those frames with CanvasFrame, creating a dependency on Java 2D, or alternatively, on Android, if we transfer the frames to video memory with AndroidFrameConverter. To further process the data, one might start using OpenCVFrameConverter, automatically creating a dependency on OpenCV that was not present until that point. We intentionally designed the API to be as easy to use as possible, but at the same time as efficient as possible, eliminating data copies whenever the underlying APIs allow. A typical usage scenario might look like this:

   FFmpegFrameGrabber grabber = new FFmpegFrameGrabber("video.mp4");
   AndroidFrameConverter converterToBitmap = new AndroidFrameConverter();
   OpenCVFrameConverter.ToMat converterToMat = new OpenCVFrameConverter.ToMat();

   // Grab an image Frame from the video file
   Frame frame = grabber.grab();
   // Perform a shallow copy to represent frame as a Mat
   Mat mat = converterToMat.convert(frame);
   // Do some processing on mat with OpenCV
   Mat processedMat = ...
   // Convert processedMat back to a Frame
   frame = converterToMat.convert(processedMat);
   // Copy the data to a Bitmap for display or something
   Bitmap bitmap = converterToBitmap.convert(frame);

It is also possible to pass a FrameConverter in a generic way to prevent unnecessary processing of data, for example:

    <I> void process(FrameConverter<I> converter, I image) {
        if (someCondition) {
            Frame frame = converter.convert(image);
            // Process frame.image...
        }
    }

As a side note, to implement Java2DFrameConverter we used code previously found inside the opencv_core helper class. On Android, this was problematic because it would cause the log file to fill up with warnings about classes missing from the java.awt package. Moving that code out fixed this annoyance. As a less happy consequence, to recover the functionality that was lost, existing users need to refactor their code based on a combination of Java2DFrameConverter and OpenCVFrameConverter. To limit these kinds of breaking changes, the features are still very limited, but in a way, this is intentional. It allows users to focus on what is missing, rather than on figuring out how to use everything. We are aware of no precedent to a framework like JavaCV that has attempted to bridge the gap between multimedia and computer vision across platforms.

In any case, we hope that overall you find this “frame processing” technique practical for your own applications, and as usual, if you have any questions, problems, or would like to contribute, but are unsure how to proceed, please feel free to share your concerns on the mailing list or via “issues” on GitHub. Enjoy and thank you for your continued interest!