Confusion over using Qemu in Docker

I'm currently learning to build ARM Docker images on x86 hosts machines. I found this guide, as well as a couple others which basically all say the same thing. They are all a few years old, so the answer to this question might as well be that either Qemu or Docker doesn't work that way anymore, but I wan't able to verify that.

Basically, you have to do two things:

1. Install the qemu-user-binfmt package. This will create a file named /proc/sys/fs/binfmt_misc/qemu-arm with the following content:

   enabled
   interpreter /usr/bin/qemu-arm-static
   flags: OCF
   offset 0
   magic 7f454c4601010100000000000000000002002800
   mask ffffffffffffff00fffffffffffffffffeffffff

2. Install the qemu-user-static package to get the /usr/bin/qemu-arm-static interpreter. This will remove the previous package, but the binfmt-entry will stay.

The guide then goes on to say that in order to build an ARM image, you have to copy the interpreter from step 2 to the same location it has on your system within the image, and in order to run an ARM image you have to mount the two copies using the option -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static.

To test this I created two images, qemu-test and qemu-test:noqemu, with the following Dockerfiles (I copied the interpreter into my working directory):

qemu-test:

FROM armv7/armhf-ubuntu
COPY qemu-arm-static /usr/bin
CMD ["sh", "-c", "echo 'Hello World'"]

qemu-test:noqemu:

FROM armv7/armhf-ubuntu
CMD ["sh", "-c", "echo 'Hello World'"]

I then tried the following four commands (still on my x86 machine):

docker run qemu-test
docker run -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static qemu-test
docker run qemu-test:noqemu
docker run -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static qemu-test:noqemu

And to my suprise, they all worked. As long as the binfmt-entry was present, that is. When I removed it, none of them worked. I also ran both images on an ARM machine, where they also both worked.

This leads me to my question(s): What's the actual point of putting the interpreter into the image? And what is the point of mounting it? Do I even need Qemu at all? Remember that those guides are a couple years old. If the answer is indeed "stuff doesn't work that way anymore", I would also be interested in how it does work now.

1 Answer

There is a single important field in the binfmt_misc kernel module config, the flag F. That flag indicates the path to the interpreter is fixed regardless of the mount namespace. Otherwise, when a file is encountered that appears to be for another platform, binfmt will look for the interpreter (/usr/bin/qemu-arm-static) in the same mount namespace as the command you are attempting to run, and you'll get a file not found error.

Part of the install for qemu-user-static is to run update-binfmts which needs the --fix-binary flag. This is typically done in newer versions of the package to handle the docker use case, but if you have an old version, you'll encounter the issue like this bug report describes.

Once you have this properly configured, my preferred way to build images for other platforms is to use buildkit, and docker makes this accessible with their buildx CLI. With buildkit, you get variables to select the platform for individual stages (it may be faster to download some binaries with your native platform and then copy them over to the target image). For more on buildx, see docker's documentation.