Running ARM Linux on your desktop PC: The foreign chroot way

Here is how you can run a non-native (ARM used here as an example) linux distro on your PC with the help of QEMU user emulation and binfmt_misc.

With QEMU user emulation you can run non-native executables. I.e. with QEMU configured for arm-linux-user you can run arm binaries. Unfortunately, without any further configuration, you can run only static executables. For dynamic executables you need to have all the libraries the executable depends on, built for the same architecture as the main executable (ARM here). This at first sight poses a problem -- you need pretty much the same standard libraries that you already have in your system (glibc, ...), but for a different architecture.

Such situation (wanting a different set of libraries) can be solved by a chroot. It would also nicely isolate your new ARM linux install. But ordinary chroot must keep the same architecture. Without one more ingredient to the mix you would only get:

chroot: failed to run command `/bin/bash': Exec format error

The last magic ingredient is binfmt_misc. Binfmt_misc generalizes the classical shebang, allowing you to associate custom interpreters to specific file types. Yes, you guessed it -- it allows us to associate ARM (or some other arch) ELF executables with our qemu. After setting up the association the executables can be run the same way as native executables, without specifying QEMU on the command line.

So, to recap: We use

1. qemu user emulation to run the ARM executables
2. chroot as a place to keep the installed distro including the required dynamic libraries
3. binfmt_misc to tell the kernel to run ARM ELF executables with the help of qemu

There's one last thing to resolve: We need a static build of QEMU, so that QEMU itself doesn't need any libraries. -- With dynamic QEMU, we'd find ourselves with the need of native libraries inside the chroot, which would defeat the purpouse.

Most distributions nowadays offer QEMU in their repositories, but static QEMU is not so common. But building it from the source is easy. Either grab a release (I tested 0.12) or, if you're feeling adventurous, clone QEMU git repo.

Configure QEMU with:

$ ./configure --disable-kvm --target-list=arm-linux-user --static

Invoking make should produce arm-linux-user/qemu-arm. If you want, grab a trivial ARM static executable (source) to test your qemu:

$ arm-linux-user/qemu-arm hello_world-arm-static
Hello world!

You can also check if the resulting QEMU binary is indeed static:

$ file arm-linux-user/qemu-arm
arm-linux-user/qemu-arm: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, not stripped

Let's set up binfmt_misc now. For ARM ELF it is (taken from QEMU's qemu-binfmt-conf.sh):

# echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/PATH/TO/QEMU:' > /proc/sys/fs/binfmt_misc/register

I find it convenient to put the qemu binary into /usr/local/bin/qemu-arm-static. You can put it anywhere you like, just keep in mind that later you want to use the same path inside your chroot. (Technically you could do with the qemu binary in the chroot only, but at least for this test we need it outside the chroot too.)

If everything worked well, you should get:

$ ./hello_world-arm-static
Hello world!

Now to the chroot: The easiest way to create one is to get a linux distribution for the target architecture. Here we use Debian, because it supports plenty or architectures, including ARM, and it super-easy to set up in a chroot. All you need is debootstrap -- many (even non-debian) distros have it in their repositories.

Because we're installing a foreign arch, we ask the installer to do only the first stage of the installation now (substitute $TARGET_PATH with your chosen target path):

# debootstrap --arch=armel --foreign --variant=minbase sid $TARGET_PATH http://ftp.cz.debian.org/debian

Now, we want to chroot to $TARGET_PATH and run the second stage of the install. For that, we need to put the QEMU binary into the chroot, so that the system can find it when starting /bin/bash inside the chroot. In my case I'd do

# mkdir -p debian-armel/usr/local/bin
# cp /usr/local/bin/qemu-arm-static debian-armel/usr/local/bin

Then we need to do some standard chroot preparations [1], [2]:

# mount --bind /dev $TARGET_PATH/dev
# mount --bind /dev/pts $TARGET_PATH/dev/pts
# mount --bind /dev/shm $TARGET_PATH/dev/shm
# mount --bind /proc $TARGET_PATH/proc
# mount --bind /sys $TARGET_PATH/sys

Now we can finally enter the chroot:

# chroot $TARGET_PATH

A word of warning: If the system can't find the interpreting binary (QEMU) inside the chroot, the error message you get is a bit unintuitive (took me a while to figure out what's wrong the first time I ran into it):

chroot: failed to run command `/bin/bash': No such file or directory

Once you enter the chroot, you should run the second stage of the Debian install:

# /debootstrap/debootstrap --second-stage

Congratulations! You've got your very own ARM Debian install on your PC! :-)

Now, there are some more things to do, but they are the same for ordinary chroot environment:

• copy /etc/recolv.conf from your real machine over $TARGET_PATH/etc/resolv.conf to get DNS working
• create /etc/apt/sources.list, e.g.: echo "deb http://ftp.cz.debian.org/debian/ unstable main" > /etc/apt/sources.list
  
• do an apt-get update to make apt happy
• install additional packages you want to have inside the chroot
• automate the binfmt_misc (un)registration and the chroot entering/leaving: Here are scripts I'm using. You could also find schroot helpful.