Debugging non-native programs with QEMU + GDB

Here, I expand further on the topic of QEMU user-mode emulation.
You probably noticed that QEMU doc mentions the option "-g port", which mentions GDB, without much further info.

Also, if you've been curious enough, you might have tried running GDB within your (say) ARM Debian chroot. If you have not, let me demonstrate:

This GDB was configured as "arm-linux-gnueabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
...
Reading symbols from /root/test...done.
(gdb) break main
Breakpoint 1 at 0x83ac: file test.c, line 5.
(gdb) start
Temporary breakpoint 2 at 0x83ac: file test.c, line 5.
Starting program: /root/test
qemu: Unsupported syscall: 26
Hello World!
During startup program exited normally.
(gdb)

So - gdb doesn't work because of unsupported syscall sys_ptrace. But as QEMU supports the GDB remote protocol, you can use that for debugging.

Apart form QEMU you also need a cross GDB built for your target. You can either build your own, or look here for some tips for Gentoo or Debian. Or get it from CodeSourcery.

It might be a good idea to look into the GDB manual for some remote debugging background first.

Start QEMU like this (either on a static binary, or inside foreign chroot):

$ qemu-arm -g 1234 your-binary

You can use any port number you want (as long as it's not occupied already). Then you have to use the same when telling GDB to connect to this instance of QEMU.

Launch the cross GDB:

$ armel-unknown-linux-gnu-gdb
GNU gdb (Gentoo 7.0 p1) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=armel-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>.
(gdb)

Now you need to tell GDB a few things:

• what file you're debugging: use the file command for that
• where to look for dynamic libraries: use the command set sysroot (you can get around this if the binary you're debugging is static; for older gdb use set solib-absolute-prefix)
• where to look for sources: use the command dir (if it's not in the current directory)
• how to connect to QEMU: use target remote localhost:1234 (host:port is the general syntax)
• it seems GDB doesn't take sysroot into account when looking for debug symbols in case those were separated from the main libraries - so tell GDB explicitly where to look for them with set debug-file-directory

This implies you need to see (or have a current copy of) the binary and possible libraries the binary uses. In case of the foreign chroot that's easy -- you use the chroot path as the solib-absolute-prefix. Here's an example:

(gdb) file /mnt/data/debian-armel/root/test
Reading symbols from /mnt/data/debian-armel/root/test...done.
(gdb) set sysroot /mnt/data/debian-armel/
(gdb) set debug-file-directory /mnt/data/debian-armel/usr/lib/debug
(gdb) dir /mnt/data/debian-armel/root/
Source directories searched: /mnt/data/debian-armel/root:$cdir:$cwd
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
warning: Can not parse XML target description; XML support was disabled at compile time
[New Remote target]
[Switching to Remote target]
0x400817e0 in ?? () from /mnt/data/debian-armel/lib/ld-linux.so.3
(gdb) break main
Breakpoint 1 at 0x83ac: file test.c, line 5.
(gdb) c
Continuing.

Breakpoint 1, main () at test.c:5
5 puts("Hello World!");
(gdb) c
Continuing.

Program exited normally.
(gdb) quit

You can automate the set up via a .gdbinit file.

Running ARM Linux on your desktop PC: The foreign chroot way

Here is how you can run a non-native (ARM used here as an example) linux distro on your PC with the help of QEMU user emulation and binfmt_misc.

With QEMU user emulation you can run non-native executables. I.e. with QEMU configured for arm-linux-user you can run arm binaries. Unfortunately, without any further configuration, you can run only static executables. For dynamic executables you need to have all the libraries the executable depends on, built for the same architecture as the main executable (ARM here). This at first sight poses a problem -- you need pretty much the same standard libraries that you already have in your system (glibc, ...), but for a different architecture.

Such situation (wanting a different set of libraries) can be solved by a chroot. It would also nicely isolate your new ARM linux install. But ordinary chroot must keep the same architecture. Without one more ingredient to the mix you would only get:

chroot: failed to run command `/bin/bash': Exec format error

The last magic ingredient is binfmt_misc. Binfmt_misc generalizes the classical shebang, allowing you to associate custom interpreters to specific file types. Yes, you guessed it -- it allows us to associate ARM (or some other arch) ELF executables with our qemu. After setting up the association the executables can be run the same way as native executables, without specifying QEMU on the command line.

So, to recap: We use

1. qemu user emulation to run the ARM executables
2. chroot as a place to keep the installed distro including the required dynamic libraries
3. binfmt_misc to tell the kernel to run ARM ELF executables with the help of qemu

There's one last thing to resolve: We need a static build of QEMU, so that QEMU itself doesn't need any libraries. -- With dynamic QEMU, we'd find ourselves with the need of native libraries inside the chroot, which would defeat the purpouse.

Most distributions nowadays offer QEMU in their repositories, but static QEMU is not so common. But building it from the source is easy. Either grab a release (I tested 0.12) or, if you're feeling adventurous, clone QEMU git repo.

Configure QEMU with:

$ ./configure --disable-kvm --target-list=arm-linux-user --static

Invoking make should produce arm-linux-user/qemu-arm. If you want, grab a trivial ARM static executable (source) to test your qemu:

$ arm-linux-user/qemu-arm hello_world-arm-static
Hello world!

You can also check if the resulting QEMU binary is indeed static:

$ file arm-linux-user/qemu-arm
arm-linux-user/qemu-arm: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, not stripped

Let's set up binfmt_misc now. For ARM ELF it is (taken from QEMU's qemu-binfmt-conf.sh):

# echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/PATH/TO/QEMU:' > /proc/sys/fs/binfmt_misc/register

I find it convenient to put the qemu binary into /usr/local/bin/qemu-arm-static. You can put it anywhere you like, just keep in mind that later you want to use the same path inside your chroot. (Technically you could do with the qemu binary in the chroot only, but at least for this test we need it outside the chroot too.)

If everything worked well, you should get:

$ ./hello_world-arm-static
Hello world!

Now to the chroot: The easiest way to create one is to get a linux distribution for the target architecture. Here we use Debian, because it supports plenty or architectures, including ARM, and it super-easy to set up in a chroot. All you need is debootstrap -- many (even non-debian) distros have it in their repositories.

Because we're installing a foreign arch, we ask the installer to do only the first stage of the installation now (substitute $TARGET_PATH with your chosen target path):

# debootstrap --arch=armel --foreign --variant=minbase sid $TARGET_PATH http://ftp.cz.debian.org/debian

Now, we want to chroot to $TARGET_PATH and run the second stage of the install. For that, we need to put the QEMU binary into the chroot, so that the system can find it when starting /bin/bash inside the chroot. In my case I'd do

# mkdir -p debian-armel/usr/local/bin
# cp /usr/local/bin/qemu-arm-static debian-armel/usr/local/bin

Then we need to do some standard chroot preparations [1], [2]:

# mount --bind /dev $TARGET_PATH/dev
# mount --bind /dev/pts $TARGET_PATH/dev/pts
# mount --bind /dev/shm $TARGET_PATH/dev/shm
# mount --bind /proc $TARGET_PATH/proc
# mount --bind /sys $TARGET_PATH/sys

Now we can finally enter the chroot:

# chroot $TARGET_PATH

A word of warning: If the system can't find the interpreting binary (QEMU) inside the chroot, the error message you get is a bit unintuitive (took me a while to figure out what's wrong the first time I ran into it):

chroot: failed to run command `/bin/bash': No such file or directory

Once you enter the chroot, you should run the second stage of the Debian install:

# /debootstrap/debootstrap --second-stage

Congratulations! You've got your very own ARM Debian install on your PC! :-)

Now, there are some more things to do, but they are the same for ordinary chroot environment:

• copy /etc/recolv.conf from your real machine over $TARGET_PATH/etc/resolv.conf to get DNS working
• create /etc/apt/sources.list, e.g.: echo "deb http://ftp.cz.debian.org/debian/ unstable main" > /etc/apt/sources.list
  
• do an apt-get update to make apt happy
• install additional packages you want to have inside the chroot
• automate the binfmt_misc (un)registration and the chroot entering/leaving: Here are scripts I'm using. You could also find schroot helpful.

Running ARM Linux on your desktop PC

It comes as no surprise that you can run entire operating system inside another with the help of virtualization -- either by some sort of hypervisor, or for the more desktop-ish of us, through a virtual machine like QEMU, Bochs, VirtualBox or the proprietary VMware.

But in the typical setup the OS you're running in the virtual machine (guest) is targeted at the same architecture as the host. -- Most typical being host == guest == x86.

But some of these tools can do more:

• Bochs can be used to run x86 or x86_64 code on any reasonable host, because it emulates even the CPU.
• QEMU, like Bochs, can also emulate the CPU. QEMU seems to be able to achieve higher emulation speeds and supports more target architectures, albeit less host architectures.

So, the point is, with QEMU you can experiment with interesting architectures on your (most probably x86/x86_64) PC. Now, who could resist that? ;-) Ever wanted to know how does Debian for ARM feel like?

To make matters even more interesting -- with QEMU's "user emulation" and a little help from binfmt_misc you can chroot into a non-native distro install.

More on this interesting subject in a later post.

Configuring linux kernel for use on ALIX 2

I'm a happy owner of two ALIX boards (2c3 and newer 2d3).

Searching the net for info about how to configure the kernel for this sweet device doesn't bring much good, so I thought I'd add my take on it.

The CPU used is an AMD Geode LX800, so it's x86-compatible. And the board itself is almost PC-compatible. This means you can run stock x86 Debian on it.

To give you some idea on the insides, here's the output of lspci:

00:01.0 Host bridge: Advanced Micro Devices [AMD] CS5536 [Geode companion] Host Bridge (rev 33)
00:01.2 Entertainment encryption device: Advanced Micro Devices [AMD] Geode LX AES Security Block
00:09.0 Ethernet controller: VIA Technologies, Inc. VT6105M [Rhine-III] (rev 96)
00:0a.0 Ethernet controller: VIA Technologies, Inc. VT6105M [Rhine-III] (rev 96)
00:0b.0 Ethernet controller: VIA Technologies, Inc. VT6105M [Rhine-III] (rev 96)
00:0f.0 ISA bridge: Advanced Micro Devices [AMD] CS5536 [Geode companion] ISA (rev 03)
00:0f.2 IDE interface: Advanced Micro Devices [AMD] CS5536 [Geode companion] IDE (rev 01)
00:0f.4 USB Controller: Advanced Micro Devices [AMD] CS5536 [Geode companion] OHC (rev 02)
00:0f.5 USB Controller: Advanced Micro Devices [AMD] CS5536 [Geode companion] EHC (rev 02)

If you want to customize the kernel yourself (instead of sticking to whatever your distro provides) here are some tips based on my experience:

• Turn on General setup/Configure standard kernel features (for small systems). This lets you kill off some features that are useless on the ALIX anyway, like PC-speaker and Virtual Terminal support.
  
• You probably want to disable any sort of namespaces/virtualization support.
• In Processor type and features:
• Select Geode GX/LX as the Processor family.
• There is neither HPET nor MTRR, so turn that off.
• There's no ACPI/APM, so you can turn that off in Power management and ACPI options. But if you intend to use selective USB suspend, you need to keep Power Management support on. (All other options in this submenu can be off.)
• In Bus options: There is PCI, so that needs to be enabled. Also turn on NatSemi SCx200 support. But don't turn on NatSemi SCx200 27MHz High-Resolution Timer Support. Enable Geode Multi-Function General Purpose Timer (MFGPT) events. (As the help says this can be use as high-precision timer, turning on Processor type and features/High Resolution Timer Support might be a good idea(?))
• In Device drivers:
• Decide if you wish to use the legacy ATA way or the newer libata way. I've been using the libata approach for a long time without any problem, so I suggest you use that. But beware -- switching between those two results in the CF card being visible as either /dev/hda or /dev/sda and this can break you boot.
  
• For the old ATA way choose: ATA/ATAPI/MFM/RLL support/CS5536 chipset support (or AMD CS5535 chipset support in older kernels)
• For the libata way choose: Serial ATA (prod) and Parallel ATA (experimental) drivers/ATA SFF support/CS5536 PATA support
• Enable Network device support/Ethernet (10 or 100Mbit)/EISA, VLB, PCI and on board controllers/VIA Rhine support. You can also enable Use MMIO instead of PIO for VIA Rhine. (Works fine for me.)
• Input device support: Enable if you plan to connect any such thing to your ALIX. If it's just gonna sit on your shelf chewing some network traffic you don't need that.
  
• Character devices: Feel free to disable Virtual terminal as it's of no use on an ALIX 2 anyway. To use the HW random number generator that the Geode contains enable Hardware Random Number Generator Core support/AMD Geode HW Random Number Generator support. Don't forget to enable the serial console: Serial drivers/8250/16550 and compatible serial support and Console on 8250/16550 and compatible serial port.
• Enable Hardware Monitoring support/National Semiconductor LM90 and compatibles to be able to see what's going on inside your ALIX. (You also need to enable I2C Support for this.)
• There's hadware watchdog too. To use it, enable Watchdog Timer Support/AMD Geode CS5535/CS5536 Watchdog. In debian, look into the package "watchdog" for an app to use it.
  
• Graphics support: Feel free to uncheck everything there. :-)
• Sound card support: The same, unless you buy some USB sound thingy. :-)
• USB support: To get USB functionality enable EHCI HCD (USB 2.0) support and OHCI HCD support.
• To get access to the ALIX leds, enable LED Support/LED Support for ALIX.2 and ALIX.3 series and also select some LED Triggers.
• You can freely uncheck Real Time Clock. While there's an option to add a battery to the board so that it keeps the time, even if I enable RTC, I don't get /dev/rtc. Either way, without the battery you'll always boot into Jan 1st 2000, so I think RTC support is useless here.
• You can enable GPOI Support/AMD CS5535/CS5536 GPIO (Geode Companion Device) -- This can probably used to read the status of the front ALIX button (?) Also I2C Support/I2C Hardware Bus support/Geode ACCESS.bus support seems relevant. But I don't know more and so far was too lazy to look for it. Any info is welcome. :-)
• To use the hardware AES crypto engine enable Cryptographic API/Hardware crypto devices/Support for the Geode LX AES engine.
• I suggest you enable Magic SysRq key (Kernel hacking/Magic SysRq key), as it works also over serial (instead of SysRq, you send a break, see Documentation/sysrq.txt).

(The menu items are as per the 2.6.31.6 kernel, but they should be reasonably similar for any kernel version that is not too distant.)

The other options pretty much depend on your specific requirements and there's not anything board-specific that influences them.

Here is a working kernel configuration file for ALIX 2 that can be used as a base for further customization.

Compiling linux kernel for x86 on x86_64

Now, that's nothing overly exotic. Yet a search through the web suggests some wild solutions like setting up a real x86_64 -> x86 cross-compiler or even setting up a 32-bit linux distribution inside a chroot.

The reasoning is simple:

• If you have 32-bit chroot environment with a compiler, you can just compile the 32-bit kernel there in the usual fashion.
• If you have the right cross-compiler, you can use the standard kernel cross-compilation procedure.

The downside is that both options are gonna eat some space off your HDD and take some time to set up.

But if you're on x86_64, you can install the multilib version of gcc that adds support for building 32-bit binaries with a simple "-m32" option.

Then I found this hint:

$ (echo ‘#! /bin/sh’; echo ‘exec gcc -m32 “$@”‘) >~/bin/i486-linux-gnu-gcc
$ chmod +x ~/bin/i486-linux-gnu-gcc
$ for i in ar ld nm objcopy strip; do
$ ln -s `which $i` ~/bin/i486-linux-gnu-$i
$ done

This basically creates a "fake" i486-linux-gnu cross-toolchain out of the multilib x86_64/x86 toolchain. Then it can be used with kernel Makefile like this:

make ARCH=i386 CROSS_COMPILE=i486-linux-gnu- <target>

That's a way I've been using for some time. But as I was preparing this post, I fired up google once again and discovered this very brief answer:

make CROSS-COMPILE=i686-pc-linux-gnu- ARCH=i386 <target>

And that command works despite me having neither real nor "fake" i686-pc-linux-gnu- cross-toolchain installed. Experimenting further reveals that all that is needed is:

make ARCH=i386 <target>

Just don't forget you need to keep that ARCH=i386 parameter throughout the make invocations (modules_install doesn't seem to need them), e.g.:

make ARCH=i386 menuconfig
make ARCH=i386 bzImage modules

In this case of building the kernel on a different machine than the one it's supposed to run on, you certainly don't want to install the built modules into /lib/modules, so invoke the make like this:

make INSTALL_MOD_PATH=prefix modules_install

Also, don't do any of the steps as root. -- There's no need to. If you build the kernel as root, you risk accidentally overwriting your host modules or causing some other damage.

You knew it was coming

I was never a big fan of blogs. And I'd never think I'd actually start one.

Anyway, the reason I hopped on is simple. I'm a software engineer who likes to tinker with computers. This quite naturally means I like free software. Specifically, I enjoy fiddling with linux, doing all kinds of odd things with it. I sometimes manage to run into things that I'm not able to find good answers to and figure out stuff in the process. Up to now, the results of my tinkering were staying locked up in my installation of Zim. But I was thinking that maybe someone else might find them useful too.

So this is the premise. Let's see how this evolves.