My concern with this upgrade, however, was that files outside the /builds/scratchbox directory were going to be touched. I wanted to be thorough so I did an experiment. I ran find -mount -type f -exec openssl md5 '{}' \; | tee -a /file-list ; find -mount -type f -exec openssl md5 '{}' \; | tee -a /file-list before and after the scratchbox upgrade. The -mount and two different runs at our two mountpoints was to ensure that we didn’t hash things like the /dev, /proc, /sys filesystem. My original intent was to do diff file-list1 file-list2 but that resulted in showing me every single file that changed. I only wanted to know the files that changed outside of my scratchbox root directory of /builds/scratchbox. My diff was polluted by 77,000 files that resided in the scratcbox root. I figured that the best option at the time was to hack up a quick python script:

#!/usr/bin/python
#This file is a quick script to process the output of
# find / -mount -type f -exec openssl md5 '{}' \; | tee -a
import sys, os.path, re

if not len(sys.argv) == 3:
    print "purple monkey dishwasher"
    exit(1)
filename_a = sys.argv[1]
filename_b = sys.argv[2]
if not os.path.exists(filename_a) or not os.path.exists(filename_b):
    print "insert change into meter and press green button"
    exit(1)
data={}
pattern = re.compile("^MD5\((?P.*)\)= (?P.*)$")
#Get the data from A
f = open(filename_a, 'r')
for i in f.readlines():
    m = pattern.search(i)
    data[m.group('file')] = m.group('hash')
f.close()
f = open(filename_b, 'r')
sbfile = re.compile("^/builds/scratchbox") #pattern describing files to ignore
#Figure out diff to B
f = open(filename_b, 'r')
for i in f.readlines():
    m = pattern.search(i)
    if not data.has_key(m.group('file')):
        if not sbfile.search(m.group('file')):
            print 'new file - ', m.group('file')
    else:
         if not sbfile.search(m.group('file')):
             if not data[m.group('file')] == m.group('hash'):
                 print 'updated file - ', m.group('file')

This is code I have written to scratch my own itch. I am posting this as it might be useful to someone else. if you wanted to ignore a different directory you’d change sbfile = re.compile("^/builds/scratchbox") to be a pattern describing your path to ignore. If you wanted to find all things that changed over your whole partition you would remove sbfile and all sbfile checks to have a final bit of code like

#Figure out diff to B
f = open(filename_b, 'r')
for i in f.readlines():
    m = pattern.search(i)
    if not data.has_key(m.group('file')):
        print 'new file - ', m.group('file')
    else:
         if not data[m.group('file')] == m.group('hash'):
             print 'updated file - ', m.group('file')

In the end, I found that the scratchbox upgrade that I did only changed my bash_history and added some tarballs to /tmp. I am very glad that this is the case as it really simplifies our deployment of the new scratchbox!

Not everyone cares to find the root cause of a problem like this. It is also only sporadically reproducible, going to the same website might show different ads each time. I would bet that a lot of people would look at this and say “Firefox is slow”, especially because the ads are there on many different pages.  These ads are also not the primary reason someone goes to the page (I’d hope) which means that it is difficult to associate the flash ad with the purpose of their tab if they do try to figure out what the problem is. Having plug-ins in a separate process (Electrolysis) seems like a great idea. I hope that, like Safari on Mac, it shows up as a totally separate process which helps avoid people blaming Firefox for poor performance.

The most annoying part of this whole situation is that I’d love to be a Uverse subscriber.  It is bad enough that they aren’t offering it in my area, but to make my browser slower is a slap in the face!

These are the first two succesful runs

We are running some old machines saved from the Landings to Castro move. The P3s are around a decade old and the G4 Mac Mini and Dual G4 PowerMacs are about 4-5 years old. We also aren’t running matched hardware which is going to make detective work on failures difficult. I have found some machines to replace our mixture of P3s that do not even have SSE. They are based on the AMD Geode LX800 processor. These machines are what the OLPC XO have as a cpu. Because they are based on an ancient CPU core (Cyrix 5×86, technically a 486), they even lack SSE. That makes them the perfect for our testing. The model that I am looking at is the MSI Fuzzy. I don’t really understand their naming, but this machine would fit our needs perfectly. It has a fast (for the category) processor and slots for 1gb of ram. Most importantly, it allows us to have as many identical machines as we need. The Mac Mini is easier to standardize on because it was fairly popular at the time and there are only one or two minor variations of the PowerPC Mini.

Before we go rushing out and buying a bunch of new machines for this testing, we need to know how long there will be demand for this kind of testing. I would estimate that it would cost about $500 per Geode machine, and that we might be able to get nightly coverage on two to three branches on linux and windows with four machines.

Also of note is that because we are running on such old hardware, we are getting JavaScript timeouts. I don’t really know how to manually set the preference that ignores these timeout warnings, but it is causing a lot of the jobs to be killed off because they aren’t responsive. I’d love to know if you know how I can disable this in an easy to automate way. I was thinking that I could launch and kill Firefox to create a profile, modify the profile’s preference file then launch it again for the real tests. I don’t know which preference file I would need to do this on though.

This is where I’d like to ask people who are working on the JavaScript JIT to let me know how long we need to do this testing and whether this current coverage (mozilla192 and mozilla-central) is enough. A comment on this post, bug 463262 or ping me on #build (jhford).

As a side note, this configuration makes it possible for us to run unit tests on second tier support machines and operating systems. If there is any community interest, I can look making it possible for anyone to connect their machine to this buildbot master if they have an obscure machine with spare cycles. This would require you to be able to dedicate the box for this testing, us to already produce builds that run on that architecture.

If anyone wants, I can get some photos of the machines on Monday.

The third approach that we are going to move forward with uses the imaging process of the second approach to create a ‘master’ image.  This image has all the information already set up, including text files specifying which image revision the device is running on.  Once this is done, we use dd to dump an image of the entire sd card onto the PC’s hard disk (dd if=/dev/sdb of=moz-ref-v1.dump bs=100M). When this is complete, we have a 3.7GB file which contains the entire contents of the master card image. We can then write this file directly to another sd card to get an identical copy of the master (dd if=moz-ref-v1.dump bs=100M). The problem is that this doesn’t scale too well. We are aiming to be able to write to 14 cards at the same time. I have investigated using tee(dd if=moz-ref-v1.dump bs=100M | tee /dev/sdb /dev/sdc /dev/sdd > /dev/null) but found that it wouldn’t write to raw devices. Another option would be to use a for loop and start a bunch of dd processes in the background. While this would have worked, we would be using a really high amount of hard disk throughput, scaling linearly. Instead, I decided to write a limited subset implementation of dd in Python.  I used the optparse library to implement the command line interface and standard Python I/O for the cloning process.  After timing a few runs, I found that my python script was about 98% as fast as the canonical implementation.  I only measured wall time as that is the only thing of value for this situation.

A further optimization that I would like to do is the ability to store the image files in a compressed format and decompress them on the fly.  Because the dump files contain every single bit that the filesystem tracks it ends up being the same size as the SD card itself.  In our case, we have a 4GB filesystem even though only about a gig of that is used.  The most simple way to get around this is to compress the image files.  Rather than worrying about manually decompressing the file before feeding it into the duplication program, I am going to implement decompression in the duplicator program.  I found that Python has a really nice BZip2 module in the standard library.  This is module provides a full file interface for a BZip2 compressed file.  Before I decided to implement this, I wanted to check that the module is able to decompress files on the fly.  I started by generating a file with random data (dd if=/dev/random of=random bs=1024 count=1024) which I then computed a sha1 has for (openssl sha1 < random > random.sha1). At this I opened an interactive Python interpreter and ran the commands:

>>> import bz2
>>> in = bz2.BZ2File('random.bz2')
>>> out = open('random', 'w+')
>>> while True:
...  buffer = f.read(1024)
...  if buffer is '':
...   break
...  o.write(buffer)
...

Once this had completed, I exited the python interpreter and compared my sha1 hashes:

jhford$ cat random.sha1
dc34e2d6308786e5e5857f7b0b1126097060df6c
jhford$ openssl sha1 < random
dc34e2d6308786e5e5857f7b0b1126097060df6c

This tells me that I can safely use the BZ2File class for implementing compressed sd card images. My current implementation strategy is to have files that have a ‘.bz2′ extension automatically treated as either a file that is compressed (input) or should be compressed (output).

I am continually impressed by how comprehensive the Python standard library is.  It seems that every time I write something in Python, there is a built in module to do anything that isn’t specific to the problem at hand!

The code for my duplicator implementation is being developed at http://hg.johnford.info/multi-dd and the imaging scripts for the mobile work lives in the build repository at http://hg.mozilla.org/build/tools in the directory buildfarm/mobile

TinderboxPrint: jford@mozilla.com
TinderboxPrint: 1252624187
Comments: No description given

is not showing up in the waterfall column. I have seen this with other builders, so I am going to assume that it isn’t specific to my code.

This now means that when you submit your code to the Try Server a WinCE build will kick off. I hope that this helps the WinCE team! If you notice anything strange going on, please don’t hesitate to contact me. I am always on irc.mozilla.org in #build as jhford.

I have also moved them all to the conference table in FAIL to see if this improves our connection reliability.  This will really help us to keep posting reliable and timely numbers.  I still have 3 units running in staging which are using a MicroSD card for their entire os and data drive.  If this works out well, we will hopefully improve our end to end times on builds and make it even easier to reimage a malfunctioning device.

I also did an Equifax and Trans Union credit check tonight.  The Equifax one comes up clean as does the majority of the Trans Union one.  Unfortunately, the Trans Union report shows that Rogers has done a credit check on me.  This negatively impacts my credit rating and is the last thing I need right now.  I called the non-emergency number of the Police to report the fraud and was greeted by an apparent wall.  I wasn’t speaking to an officer though.  This person informed me that even though I live in Toronto normally, and that BOTH charges on my card show as Toronto, ON it isn’t a matter for Toronto Police unless I am physically in Toronto.  Nice to see my tax dollars hard at work!  This numpty suggested that I visit the local police here to give a report.  I don’t understand this.  Why on earth would a Mountain View police officer be the one to deal with a crime committed against a Canadian, in Canada with Canadian companies.  I doubt that it would even make its way up to Toronto.

This couldn’t have come at a worse time.  I wonder if the false credit check will be removed from my credit history as I absolutely didn’t request it, the $600 charge or the $1503 charge!  As I sit here, I wonder what I can do regarding this situation.

The new devices still in their boxes

The imaging process that I described in an earlier post requires some time to bake in a staging environment because it requires some fairly significant changes in how the device acts.  In the interim I have generated an image for flashing the N810s that still uses the internal raw flash memory.  To do this we set up a reference device exactly as needed.  After this I generated a new filesystem image (sudo gainroot; mount -t jffs2 /dev/mtdblock4 /opt; mount /dev/mmcblk1p1 /floppy; cd /floppy; mkfs.jffs2 -r /opt -o moz-ref-image-v1.jffs2 -e 128 -l -n). Once this was done I transfered the files to my desktop pc and wrote an automation script that flashes the devices. This puts the root filesytem onto the device. For the pagesets used by talos we need to use the internal controlled flash card. For this I have written a script which takes the device files for each plugged in maemo and rsync’s the contents of the drive. Using a helper script I have made this work on 6 devices at once.

Data being transfered to the new N810s

It took a less than one hour to get all the machines from unopened shipping box to running in our staging environment. If everything looks good, I will move these six new devices over to production tomorrow. This is a significant improvement over the multiple hour per device process of setting up a device we used before. This process scales very well. It takes 90 seconds to flash each device with a root file system and is done in serial. Setting up the page set file system took 6 minutes for 6 devices and is done in parallel.

In Staging

I am eagerly awaiting the arrival of 14 more N810s some time this or next week.

Anatomy of the N810

The Nokia N810 has internal raw flash memory (/dev/mtd*), internal controlled flash (/dev/mmcblk0) and a MiniSD card slot (/dev/mmcblk1).  The raw flash memory and controlled flash memory differ significantly in their operation and as a result use two totally different file-systems.  On a normal N810 there are 5 partitions:

• Bootloader – raw file
• Kernel image – raw file
• Initialization filesystem – JFFS2 filesystem
• Root filesystem – JFFS2 filesystem
• Data partition on internal controlled 2GB flash memory – FAT filesystem

My first approach was to generate a custom filesystem image of the root filesystem.  I accomplished this by mounting the root filesystem to an existing unused folder (mount -t jffs2 /dev/mtdblock4 /opt). I could have created a new folder, but this would mean that this new mountpoint folder would also be present in the image. Once mounted I used the JFFS2 filesytem creator to create a filesystem image(/home/user/initfs_flasher/mkfs.jffs2 -r /opt -o /media/mmc2/rootfs-moz-v1.jffs2 -e 128 -l -n). This command creates a filesystem image (i.e. not written out to a device file as you would with mkfs.ext3) with a root directory of /opt. The other parameters specify the erase block size, the byte-sex and not to have cleanmarkers.  This will write the file to the removable flash card which can then be inserted into a PC and flashed to the device using the Nokia’s flashing program (su; ./flasher-3.0 --rootfs rootfs-moz-v1.jffs2 --flash).

Shortcomings

This approached worked really well but it could not automate setting the device up to go straight into production.  Someone would need to manually change the hostname, change the buildbot.tac file and change the buildbot information page.  This got me to thinking that I could copy the files onto my PC, modify them, generate the filesystem image locally then flash the device.  I started by setting up rsync and openssh.  I mounted the root filesystem as before and ran rsync on my PC (su; mkdir rootfs; rsync -a root@10.0.0.1/. rootfs/.) and got a snack (note the critical -a flag on rsync). When I had gotten back I had all the files and created the filesystem (su; yum install mtd-utils; mkfs.jffs2 -r rootfs -o rootfs-moz-v2.jffs2 -e 128 -l -n) then flashed this onto the device (su -; ./flasher-3.0 --rootfs). This is where the wheels began to fall off. The device booted and it appeared to be working great.

I was testing the devices when I noticed that the shutdown prompt wasn’t themed properly.  When you press the power button on an N810, a menu comes up asking if you want to lock the device, suspend it or turn it off. The first issue with this approach was that this menu wasn’t themed using the proper hildon buttons and instead was using the default GTK button theme.  This was not in itself an issue but signalled that there was corruption happening somewhere.  We decided to run this image in our staging environment and unfortunately my concerns were realized when a lot of the tests were failing.

Improved Process

While investigating our options I came across a couple pages about booting from a flash card and advanced boot configurations on the maemo wiki.  My initial attempt followed the booting from a flash card page’s instructions.  This yeilded a working system but made the flashing process very complicated requiring work to be done on the PC and on the device.  This would not really solve any problem and as a result I did some further thinking.  It turns out that Nokia had the foresight to allow booting from an SD card in the stock firmware.  Now on my second approach, I used the flashing program to set the device that has the root filesystem (su; ./flasher-3.0 --set-root-device mmc) to one of the controlled flash cards. Important to note is that there is no place to choose between the internal memory and the memory card. At boot the N810 will scan the first partition on both devices looking for a specific file. When it finds that file it selects that device as the boot device.

To get the fully configured root filesystem onto the SD card, I remounted the raw flash root partition as in my first approach and rsync’d it to my PC as in my modified first approach. My first strategy for duplicating the SD card was to copy the files to the reference SD card then use dd to dump the contents of the card(dd if=/dev/sdc of=dump.sd bs=100M) to a file then write them to another card(dd if=dump.sd of=/dev/sdc bs=100M). This strategy has the disadvantage of needing to write out a lot of blank space which takes a long time. It also doesn’t shrink or grow the partition or filesystem which means that you need to use the exact same size or larger card.

Final Implementation

I decided to write a less naïve implementation. The result was a script that creates a blank filesystem on an SD card on my PC, copies the files over, modifies the files as needed then ejects the SD card. This card can then be put into a device that has had its root device set correctly.  I wrote a script to do all device setup correctly.  In this script I reflash the bootloader and kernel for safety and put the device into Research and Development mode to enable root access on device.  I also flash an empty root filesystem to the raw flash partition to ensure it isn’t used.  Using different types of flash could affect performance test results.

Something rather annoying is that the N810’s device file naming convention is for controlled flash devices is broken.  This means that while the MiniSD card should be /dev/mmcblk1 when booted from the raw flash /dev/mtdblock4 it is /dev/mmcblk0 when booted from MiniSD card itself.  Because I had copied the entire root filesystem to the internal card while testing things, the only way I was able to confirm that I was booted from the memory card was to remove the card while the system was operating.  Thankfully, this caused a major system crash and proved that I was running off the MiniSD card.

Hidden Benefits

As a side effect of running the device with all the files on the same MiniSD card the internal 2GB controlled flash card is now unused. This can now be used for lots of swap storage. To do this you need to create a file that is the size of the swap you need (dd if=/dev/zero of=${MOUNTPOINT}/.swap bs=1024 count=262144) then turn it into swap space (mkswap ${MOUNTPOINT}/.swap). This can either be done on device or on a PC with the device plugged into it.

Next Steps

I still haven’t tested the SD card images in staging yet.  I also need to work with Aki to move files from the old mount points of /media/mmc1 and /media/mmc2 which are no longer correct or used.  I also need to move the imaging machine somewhere other than my desk and check the code into Mozilla’s mercurial repository.

Useful Links

• http://wiki.maemo.org/Modifying_the_root_image
• http://wiki.maemo.org/Booting_from_a_flash_card
• http://wiki.maemo.org/Advanced_booting
• http://fanoush.wz.cz/maemo/index.html#initfs