Imaging Nokia’s N810

In order to scale our mobile testing infrastructure we need to be able to get devices up and running in production as soon as possible and be able to repair software issues as quickly as possible.  The N810 runs Maemo and due to its similarities to desktop Linux it is the only mobile platform we are able to run unit and performance tests yet.  Up until now, we have had to set up the devices by hand each time.  As well, when a device’s software stopped working right it needed to be erased, reflashed and reset manually.  This reseting process occurs multiple times a week and ends up taking a while to get the device back into production.  The work required, while uninteresting, is full of detail that may not be accurately expressed on the N810′s miniature keyboard.  The ability to image the N810 gives us a multi-faceted win in that we are able to get new devices up quicker, fix broken devices quicker and ensure all devices are set up consistently.

Anatomy of the N810

The Nokia N810 has internal raw flash memory (/dev/mtd*), internal controlled flash (/dev/mmcblk0) and a MiniSD card slot (/dev/mmcblk1).  The raw flash memory and controlled flash memory differ significantly in their operation and as a result use two totally different file-systems.  On a normal N810 there are 5 partitions:

• Bootloader – raw file
• Kernel image – raw file
• Initialization filesystem – JFFS2 filesystem
• Root filesystem – JFFS2 filesystem
• Data partition on internal controlled 2GB flash memory – FAT filesystem

My first approach was to generate a custom filesystem image of the root filesystem.  I accomplished this by mounting the root filesystem to an existing unused folder (mount -t jffs2 /dev/mtdblock4 /opt). I could have created a new folder, but this would mean that this new mountpoint folder would also be present in the image. Once mounted I used the JFFS2 filesytem creator to create a filesystem image(/home/user/initfs_flasher/mkfs.jffs2 -r /opt -o /media/mmc2/rootfs-moz-v1.jffs2 -e 128 -l -n). This command creates a filesystem image (i.e. not written out to a device file as you would with mkfs.ext3) with a root directory of /opt. The other parameters specify the erase block size, the byte-sex and not to have cleanmarkers.  This will write the file to the removable flash card which can then be inserted into a PC and flashed to the device using the Nokia’s flashing program (su; ./flasher-3.0 --rootfs rootfs-moz-v1.jffs2 --flash).

Shortcomings

This approached worked really well but it could not automate setting the device up to go straight into production.  Someone would need to manually change the hostname, change the buildbot.tac file and change the buildbot information page.  This got me to thinking that I could copy the files onto my PC, modify them, generate the filesystem image locally then flash the device.  I started by setting up rsync and openssh.  I mounted the root filesystem as before and ran rsync on my PC (su; mkdir rootfs; rsync -a root@10.0.0.1/. rootfs/.) and got a snack (note the critical -a flag on rsync). When I had gotten back I had all the files and created the filesystem (su; yum install mtd-utils; mkfs.jffs2 -r rootfs -o rootfs-moz-v2.jffs2 -e 128 -l -n) then flashed this onto the device (su -; ./flasher-3.0 --rootfs). This is where the wheels began to fall off. The device booted and it appeared to be working great.

I was testing the devices when I noticed that the shutdown prompt wasn’t themed properly.  When you press the power button on an N810, a menu comes up asking if you want to lock the device, suspend it or turn it off. The first issue with this approach was that this menu wasn’t themed using the proper hildon buttons and instead was using the default GTK button theme.  This was not in itself an issue but signalled that there was corruption happening somewhere.  We decided to run this image in our staging environment and unfortunately my concerns were realized when a lot of the tests were failing.

Improved Process

While investigating our options I came across a couple pages about booting from a flash card and advanced boot configurations on the maemo wiki.  My initial attempt followed the booting from a flash card page’s instructions.  This yeilded a working system but made the flashing process very complicated requiring work to be done on the PC and on the device.  This would not really solve any problem and as a result I did some further thinking.  It turns out that Nokia had the foresight to allow booting from an SD card in the stock firmware.  Now on my second approach, I used the flashing program to set the device that has the root filesystem (su; ./flasher-3.0 --set-root-device mmc) to one of the controlled flash cards. Important to note is that there is no place to choose between the internal memory and the memory card. At boot the N810 will scan the first partition on both devices looking for a specific file. When it finds that file it selects that device as the boot device.

To get the fully configured root filesystem onto the SD card, I remounted the raw flash root partition as in my first approach and rsync’d it to my PC as in my modified first approach. My first strategy for duplicating the SD card was to copy the files to the reference SD card then use dd to dump the contents of the card(dd if=/dev/sdc of=dump.sd bs=100M) to a file then write them to another card(dd if=dump.sd of=/dev/sdc bs=100M). This strategy has the disadvantage of needing to write out a lot of blank space which takes a long time. It also doesn’t shrink or grow the partition or filesystem which means that you need to use the exact same size or larger card.

Final Implementation

I decided to write a less naïve implementation. The result was a script that creates a blank filesystem on an SD card on my PC, copies the files over, modifies the files as needed then ejects the SD card. This card can then be put into a device that has had its root device set correctly.  I wrote a script to do all device setup correctly.  In this script I reflash the bootloader and kernel for safety and put the device into Research and Development mode to enable root access on device.  I also flash an empty root filesystem to the raw flash partition to ensure it isn’t used.  Using different types of flash could affect performance test results.

Something rather annoying is that the N810′s device file naming convention is for controlled flash devices is broken.  This means that while the MiniSD card should be /dev/mmcblk1 when booted from the raw flash /dev/mtdblock4 it is /dev/mmcblk0 when booted from MiniSD card itself.  Because I had copied the entire root filesystem to the internal card while testing things, the only way I was able to confirm that I was booted from the memory card was to remove the card while the system was operating.  Thankfully, this caused a major system crash and proved that I was running off the MiniSD card.

Hidden Benefits

As a side effect of running the device with all the files on the same MiniSD card the internal 2GB controlled flash card is now unused. This can now be used for lots of swap storage. To do this you need to create a file that is the size of the swap you need (dd if=/dev/zero of=${MOUNTPOINT}/.swap bs=1024 count=262144) then turn it into swap space (mkswap ${MOUNTPOINT}/.swap). This can either be done on device or on a PC with the device plugged into it.

Next Steps

I still haven’t tested the SD card images in staging yet.  I also need to work with Aki to move files from the old mount points of /media/mmc1 and /media/mmc2 which are no longer correct or used.  I also need to move the imaging machine somewhere other than my desk and check the code into Mozilla’s mercurial repository.

Useful Links

• http://wiki.maemo.org/Modifying_the_root_image
• http://wiki.maemo.org/Booting_from_a_flash_card
• http://wiki.maemo.org/Advanced_booting
• http://fanoush.wz.cz/maemo/index.html#initfs