DevOps

How to SBT + Docker buildX + Multi-Arch

  • April 16th 2020
  • 7 min read

Table of contents

Introduction

Docker v19.03 added a new experimental feature. It is the multi-arch image builder based on the Docker BuildX project.

Also, I started to test Simplexspatial server in my Pine64 cluster, so I need to have the docker image published for two different platforms: amd64 and arm64.

In this post, I will explain how to create Docker images for different architectures using SBT Native Packager and Docker.

SBT configuration

In the SBT project, I’m using the sbt-native-packager SBT plugin. This allows to package the application in native formats, like msi for Windows, dmg for macOS, deb for Debian distros, rpm for RHEL distro, etc.

In this post, I will talk only about one distribution format: Docker. It will create a Docker image with all stuff necessary to execute the application from any system with Docker installed.

Two simple steps:

  1. Import the sbt plugin. To do it, add the plugin in the project/plugins.sbt:

    addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.7.0")

  2. Enable Docker and java application packaging in the build.sbt file.

    Plugins:

    • JavaAppPackaging It will generate a folder structure with the application, dependencies, and a bash script like BashStartScriptPlugin does. I prefer this approach than the painful fat jar.

    • DockerPlugin enables docker images generation.

       .enablePlugins(
     JavaAppPackaging,
     DockerPlugin // Enable the docker plugin
 )
 .settings(
     packageName in Docker := "simplexspatial/simplexspatial",
     packageDescription := "The Reactive Geospatial Server",
     dockerBaseImage := "adoptopenjdk:11-jre-hotspot", // arm compatible base image
     dockerUpdateLatest := true, // docker:publishLocal will replace the latest tagged image.
     dockerExposedPorts ++= Seq(2550, 9010, 6080, 7080, 8080),
     defaultLinuxInstallLocation in Docker := "/opt/simplexspatial",
     dockerExposedVolumes := Seq("/opt/simplexspatial/conf", "/opt/simplexspatial/logs")
 )

Generate unique arch Docker image, using sbt-native-packager

To generate the Docker image, it is as simple as to execute sbt docker:publishLocal. Using docker images to list images in your systems, you will find two new images with the name given using the packageName configuration property, and tagged with the version of your project and latest as well.

After building the docker image using sbt plugin, we can check if it is there:

$ docker images
REPOSITORY                      TAG                                        IMAGE ID            CREATED             SIZE
simplexspatial/simplexspatial   0.0.1-SNAPSHOT                             099c0b759140        13 seconds ago      313MB
simplexspatial/simplexspatial   latest                                     099c0b759140        13 seconds ago      313MB
..........

But wait! I want to execute the image in my Pine64 cluster and my Recycled homemade cluster.

So let’s check the architecture of the new image:

$ docker image inspect 099c0b759140 | grep Architecture
        "Architecture": "amd64",

No surprises. amd64 like my laptop!! So It will not run in the Pine64 cluster because nodes are ARMv8 architecture CPUs.

So I’m going to remove the image with sbt docker:clean or docker rmi 099c0b759140 -f and I’ll generate it again for amd64 and arm architectures.

Generate multi-arch Docker image, using buildx

So let’s create an image for amd64 and another one for armv8.

From Docker v19.03, there is a new experimental feature based in buildx tool. This tool allows cross platform image creation.

To use this tool, we need to avoid sbt docker:publishLocal and use docker cli directly with the Dockerfile generated by sbt. To do it:

  1. Generate the Dockerfile using SBT. Executing sbt docker:stage will generate the Dockerfile and all files used to generate the Docker image target/docker/stage, but it will not generate the image itself.

  2. Enable experimental features in Docker. So It’s necessary to add "experimental": true field into the config file /etc/docker/daemon.json and restart the Docker daemon. If the file doesn’t exist, create one.

    $ docker version -f '{{.Server.Experimental}}'
false
$ sudo vi /etc/docker/daemon.json
{
    "experimental": true
}
$ sudo systemctl restart docker
$ docker version -f '{{.Server.Experimental}}'
true

  3. Enable CLI experimental features (previously we enabled daemon experimental features) through the DOCKER_CLI_EXPERIMENTAL environment variable:

    $ docker buildx version
docker: 'buildx' is not a docker command.
See 'docker --help'

$ export DOCKER_CLI_EXPERIMENTAL=enabled

$ docker buildx version
github.com/docker/buildx v0.3.1-tp-docker 6db68d029599c6710a32aa7adcba8e5a344795a7

  4. Because I’m using Linux, I need to enable binfmt_misc. This gives the capability to the kernel to recognize and run certain types of applications. In our case, it will allow qemu images.

    To enable binfmt_misc I will run a docker image that is doing the dirty job for us. I’m using the last tag of the image, so before you run it, check for the last image.

    $ ls -l /proc/sys/fs/binfmt_misc/
total 0
--w------- 1 root root 0 Apr 14 12:59 register
-rw-r--r-- 1 root root 0 Apr 14 12:59 status
$ docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64
Unable to find image 'docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64' locally
a7996909642ee92942dcd6cff44b9b95f08dad64: Pulling from docker/binfmt
5d6ca6c8ba77: Pull complete 
b26a8e2c75fc: Pull complete 
3436361ddd98: Pull complete 
Digest: sha256:758ca0563f371b384cfd67b6590b5be2dc024fef45bc14a050ae104f0caad14e
Status: Downloaded newer image for docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64

$ ls -l /proc/sys/fs/binfmt_misc/
total 0
-rw-r--r-- 1 root root 0 Apr 14 13:06 qemu-aarch64
-rw-r--r-- 1 root root 0 Apr 14 13:06 qemu-arm
-rw-r--r-- 1 root root 0 Apr 14 13:06 qemu-ppc64le
-rw-r--r-- 1 root root 0 Apr 14 13:06 qemu-riscv64
-rw-r--r-- 1 root root 0 Apr 14 13:06 qemu-s390x
--w------- 1 root root 0 Apr 14 12:59 register
-rw-r--r-- 1 root root 0 Apr 14 12:59 status

$ grep enable /proc/sys/fs/binfmt_misc/qemu-*
/proc/sys/fs/binfmt_misc/qemu-aarch64:enabled
/proc/sys/fs/binfmt_misc/qemu-arm:enabled
/proc/sys/fs/binfmt_misc/qemu-ppc64le:enabled
/proc/sys/fs/binfmt_misc/qemu-riscv64:enabled
/proc/sys/fs/binfmt_misc/qemu-s390x:enabled

    Now, one of the platforms available and enabled is qemu-arm.

  5. By default, Buildx is using docker driver to provide the same experience as using the native docker build. It doesn’t support cross architecture builds, so it’s necessary to use docker-container driver. So I will create a new builder instance and enable it:

    $ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  PLATFORMS
default * docker                  
default default         running linux/amd64, linux/386
$ docker buildx create --use --name cross-platform-builder
cross-platform-builde

$ docker buildx ls
NAME/NODE                 DRIVER/ENDPOINT             STATUS   PLATFORMS
cross-platform-builder *  docker-container                     
cross-platform-builder0 unix:///var/run/docker.sock inactive 
default                   docker                               
default                 default                     running  linux/amd64, linux/386

  6. Now, it is possible to generate images for other platforms using buildx. So let’s do it:

    $ docker buildx build --platform=linux/arm64,linux/amd64 --push -t simplexspatial/simplexspatial .

  7. Done!!! Afterwards push into the repo, you will find them in Docker Hub repository. Docker Hub repository

Notes.

How to set a local registry:

Using the Docker registry container:

$ docker run -d -p 5000:5000 --name registry registry:2
Unable to find image 'registry:2' locally
2: Pulling from library/registry
486039affc0a: Pull complete 
ba51a3b098e6: Pull complete 
8bb4c43d6c8e: Pull complete 
6f5f453e5f2d: Pull complete 
42bc10b72f42: Pull complete 
Digest: sha256:7d081088e4bfd632a88e3f3bcd9e007ef44a796fddfe3261407a3f9f04abe1e7
Status: Downloaded newer image for registry:2
4aa12c82df6b1e52ded9560fb8935875a2bbec9920d0e2730b207220602e81a9
$ docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                    NAMES
4aa12c82df6b        registry:2                      "/entrypoint.sh /etc…"   6 seconds ago       Up 5 seconds        0.0.0.0:5000->5000/tcp   registry
90b9a5a4cea5        moby/buildkit:buildx-stable-1   "buildkitd"              2 hours ago         Up 2 hours                                   buildx_buildkit_cross-platform-builder0
$ docker login localhost:5000
Username: angelcc
Password: 
WARNING! Your password will be stored unencrypted in /home/angelcc/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
$ cat $HOME/.docker/config.json
{
	"auths": {
		"localhost:5000": {
			"auth": "XXXXXXX"
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.8 (linux)"
	}
}⏎

Local images.

BuildX requires the --output parameter, but there is a bug that is blocking to store it locally, so the easiest option is to use a repository. --push is an alias of --output=type=registry.

Alert!!!! I didn’t find the way to deploy into this local repository using buildx build --output. So please, if you know the way, let me know.

SCALA + java for ARM64

  • The default docker image used by SBT is openjdk:8, but it is not available for arm64 architectures.
  • Keep in mind the memory limitations of arm devices.
  • Not all openjdk versions are working the same way. I mean, it is easy to get errors related with the openjdk version included in the arm alpine base image. So you will need to play with it until to find a good combination.

References