Saturday, December 12, 2009

Running ARM Linux on your desktop PC: The foreign chroot way

Here is how you can run a non-native (ARM used here as an example) linux distro on your PC with the help of QEMU user emulation and binfmt_misc.

With QEMU user emulation you can run non-native executables. I.e. with QEMU configured for arm-linux-user you can run arm binaries. Unfortunately, without any further configuration, you can run only static executables. For dynamic executables you need to have all the libraries the executable depends on, built for the same architecture as the main executable (ARM here). This at first sight poses a problem -- you need pretty much the same standard libraries that you already have in your system (glibc, ...), but for a different architecture.

Such situation (wanting a different set of libraries) can be solved by a chroot. It would also nicely isolate your new ARM linux install. But ordinary chroot must keep the same architecture. Without one more ingredient to the mix you would only get:
chroot: failed to run command `/bin/bash': Exec format error

The last magic ingredient is binfmt_misc. Binfmt_misc generalizes the classical shebang, allowing you to associate custom interpreters to specific file types. Yes, you guessed it -- it allows us to associate ARM (or some other arch) ELF executables with our qemu. After setting up the association the executables can be run the same way as native executables, without specifying QEMU on the command line.

So, to recap: We use
  1. qemu user emulation to run the ARM executables
  2. chroot as a place to keep the installed distro including the required dynamic libraries
  3. binfmt_misc to tell the kernel to run ARM ELF executables with the help of qemu

There's one last thing to resolve: We need a static build of QEMU, so that QEMU itself doesn't need any libraries. -- With dynamic QEMU, we'd find ourselves with the need of native libraries inside the chroot, which would defeat the purpouse.

Most distributions nowadays offer QEMU in their repositories, but static QEMU is not so common. But building it from the source is easy. Either grab a release (I tested 0.12) or, if you're feeling adventurous, clone QEMU git repo.

Configure QEMU with:
$ ./configure --disable-kvm --target-list=arm-linux-user --static

Invoking make should produce arm-linux-user/qemu-arm. If you want, grab a trivial ARM static executable (source) to test your qemu:
$ arm-linux-user/qemu-arm hello_world-arm-static
Hello world!

You can also check if the resulting QEMU binary is indeed static:
$ file arm-linux-user/qemu-arm
arm-linux-user/qemu-arm: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, not stripped

Let's set up binfmt_misc now. For ARM ELF it is (taken from QEMU's
# echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/PATH/TO/QEMU:' > /proc/sys/fs/binfmt_misc/register

I find it convenient to put the qemu binary into /usr/local/bin/qemu-arm-static. You can put it anywhere you like, just keep in mind that later you want to use the same path inside your chroot. (Technically you could do with the qemu binary in the chroot only, but at least for this test we need it outside the chroot too.)

If everything worked well, you should get:
$ ./hello_world-arm-static
Hello world!

Now to the chroot: The easiest way to create one is to get a linux distribution for the target architecture. Here we use Debian, because it supports plenty or architectures, including ARM, and it super-easy to set up in a chroot. All you need is debootstrap -- many (even non-debian) distros have it in their repositories.

Because we're installing a foreign arch, we ask the installer to do only the first stage of the installation now (substitute $TARGET_PATH with your chosen target path):
# debootstrap --arch=armel --foreign --variant=minbase sid $TARGET_PATH

Now, we want to chroot to $TARGET_PATH and run the second stage of the install. For that, we need to put the QEMU binary into the chroot, so that the system can find it when starting /bin/bash inside the chroot. In my case I'd do
# mkdir -p debian-armel/usr/local/bin
# cp /usr/local/bin/qemu-arm-static debian-armel/usr/local/bin

Then we need to do some standard chroot preparations [1], [2]:
# mount --bind /dev $TARGET_PATH/dev
# mount --bind /dev/pts $TARGET_PATH/dev/pts
# mount --bind /dev/shm $TARGET_PATH/dev/shm
# mount --bind /proc $TARGET_PATH/proc
# mount --bind /sys $TARGET_PATH/sys

Now we can finally enter the chroot:
# chroot $TARGET_PATH

A word of warning: If the system can't find the interpreting binary (QEMU) inside the chroot, the error message you get is a bit unintuitive (took me a while to figure out what's wrong the first time I ran into it):
chroot: failed to run command `/bin/bash': No such file or directory

Once you enter the chroot, you should run the second stage of the Debian install:
# /debootstrap/debootstrap --second-stage

Congratulations! You've got your very own ARM Debian install on your PC! :-)

Now, there are some more things to do, but they are the same for ordinary chroot environment:
  • copy /etc/recolv.conf from your real machine over $TARGET_PATH/etc/resolv.conf to get DNS working
  • create /etc/apt/sources.list, e.g.: echo "deb unstable main" > /etc/apt/sources.list
  • do an apt-get update to make apt happy
  • install additional packages you want to have inside the chroot
  • automate the binfmt_misc (un)registration and the chroot entering/leaving: Here are scripts I'm using. You could also find schroot helpful.


  1. The Problem I am facing is Fedora Don't have static libs. And I can't find a Static version of qemu-arm/qemu-armeb

  2. Yes, static qemu is no so common in distro repositories. That's why I addressed it in the post, just search for the paragraph beginning with "There's one last thing to resolve: We need a static build of QEMU" You gave up reading too early. :)

  3. Yours is by far the clearest post I've encountered on the subject of running a foreign chroot using QEMU. Sincere thanks!

    The remaining puzzle for me is how to use the native cross-compilers while inside the foreign chroot. Basically, to run ./configure and make via QEMU, but then have those invoke the cross gcc (e.g. arm-none-linux-gnuabi-gcc), while still having access to the dynamic libraries in the chroot. Why? Emulated compilers are slow, while static binaries are big.

    I had high hopes for scratchbox2 to do this, but after several hours of trying, got nowhere. If you get bored one day and figure this out, please write another article like this! I've spent hours searching the net, and there really isn't one which Just Works :/

  4. I was wondering if anyone had any issue with running the debootstrap within the chroot environment. I installed the qemu-arm-static package on my Ubuntu Lucid box and attempted to setup a Debian ARM squeeze-based chroot. I was able to chroot fine, but I get the following error:

    W: Failure trying to run: mount -t proc proc /proc

    I tried using schroot to see if it made a difference, but no luck.


    1. Nevermind! I figured out my mistake. I did the first stage debootstrap with fakeroot and it didn't like that. Doing the steps as root and then using schroot seems to be working fine. Thanks for the steps! That's really helpful.

  5. Does anybody knows how to build ARM static executable with codesourcery armgcc tool-chain involving many .c and .s files? Or is it possible to run .bix binary format using static qemu library?

    1. sorry for the typo, I meant .bin binary format.

  6. I just wanted to add some thanks for this. I have got an arm chroot working on Arch Linux. I had to build a static qemu like you said. To build that I had to build static glib2 libraries because there are only dynamic in the default configuration of the glib2 package. Regardless, I got it working and it's quite amusing to do "uname -a" on my x86 box and see it tell me its arm :)

    In testing it, however, I did find doing some things caused qemu to report it didn't support a system call. As I was just doing the eqivalent of debootstrap I was surprised by this.

    But very cool, nonetheless :)

  7. I'm not able to run any commands once setup the chroot. When I try to run any command(even the internal commands) it is giving a warning that cannot execute the binary.