An example of a push to try session to build against my user repository copy of mobile-browser might look like:

hg clone http://hg.mozilla.org/mozilla-central
cd mozilla-central
rm README.txt
echo 'users/jford_mozilla.com/mobile-browser' > mobile-repo
echo 'default' > mobile-rev
hg add mobile-repo mobile-rev
hg commit -m 'Testing to see if we build without a readme file'
hg push -f ssh://hg.mozilla.org/try

With this marks the end of the old try server. It also means that we have fully transitioned our try server build infrastructure to Buildbot 0.8.0.

My concern with this upgrade, however, was that files outside the /builds/scratchbox directory were going to be touched. I wanted to be thorough so I did an experiment. I ran find -mount -type f -exec openssl md5 '{}' \; | tee -a /file-list ; find -mount -type f -exec openssl md5 '{}' \; | tee -a /file-list before and after the scratchbox upgrade. The -mount and two different runs at our two mountpoints was to ensure that we didn’t hash things like the /dev, /proc, /sys filesystem. My original intent was to do diff file-list1 file-list2 but that resulted in showing me every single file that changed. I only wanted to know the files that changed outside of my scratchbox root directory of /builds/scratchbox. My diff was polluted by 77,000 files that resided in the scratcbox root. I figured that the best option at the time was to hack up a quick python script:

#!/usr/bin/python
#This file is a quick script to process the output of
# find / -mount -type f -exec openssl md5 '{}' \; | tee -a
import sys, os.path, re

if not len(sys.argv) == 3:
    print "purple monkey dishwasher"
    exit(1)
filename_a = sys.argv[1]
filename_b = sys.argv[2]
if not os.path.exists(filename_a) or not os.path.exists(filename_b):
    print "insert change into meter and press green button"
    exit(1)
data={}
pattern = re.compile("^MD5\((?P.*)\)= (?P.*)$")
#Get the data from A
f = open(filename_a, 'r')
for i in f.readlines():
    m = pattern.search(i)
    data[m.group('file')] = m.group('hash')
f.close()
f = open(filename_b, 'r')
sbfile = re.compile("^/builds/scratchbox") #pattern describing files to ignore
#Figure out diff to B
f = open(filename_b, 'r')
for i in f.readlines():
    m = pattern.search(i)
    if not data.has_key(m.group('file')):
        if not sbfile.search(m.group('file')):
            print 'new file - ', m.group('file')
    else:
         if not sbfile.search(m.group('file')):
             if not data[m.group('file')] == m.group('hash'):
                 print 'updated file - ', m.group('file')

This is code I have written to scratch my own itch. I am posting this as it might be useful to someone else. if you wanted to ignore a different directory you’d change sbfile = re.compile("^/builds/scratchbox") to be a pattern describing your path to ignore. If you wanted to find all things that changed over your whole partition you would remove sbfile and all sbfile checks to have a final bit of code like

#Figure out diff to B
f = open(filename_b, 'r')
for i in f.readlines():
    m = pattern.search(i)
    if not data.has_key(m.group('file')):
        print 'new file - ', m.group('file')
    else:
         if not data[m.group('file')] == m.group('hash'):
             print 'updated file - ', m.group('file')

In the end, I found that the scratchbox upgrade that I did only changed my bash_history and added some tarballs to /tmp. I am very glad that this is the case as it really simplifies our deployment of the new scratchbox!

Not everyone cares to find the root cause of a problem like this. It is also only sporadically reproducible, going to the same website might show different ads each time. I would bet that a lot of people would look at this and say “Firefox is slow”, especially because the ads are there on many different pages.  These ads are also not the primary reason someone goes to the page (I’d hope) which means that it is difficult to associate the flash ad with the purpose of their tab if they do try to figure out what the problem is. Having plug-ins in a separate process (Electrolysis) seems like a great idea. I hope that, like Safari on Mac, it shows up as a totally separate process which helps avoid people blaming Firefox for poor performance.

The most annoying part of this whole situation is that I’d love to be a Uverse subscriber.  It is bad enough that they aren’t offering it in my area, but to make my browser slower is a slap in the face!

These are the first two succesful runs

We are running some old machines saved from the Landings to Castro move. The P3s are around a decade old and the G4 Mac Mini and Dual G4 PowerMacs are about 4-5 years old. We also aren’t running matched hardware which is going to make detective work on failures difficult. I have found some machines to replace our mixture of P3s that do not even have SSE. They are based on the AMD Geode LX800 processor. These machines are what the OLPC XO have as a cpu. Because they are based on an ancient CPU core (Cyrix 5×86, technically a 486), they even lack SSE. That makes them the perfect for our testing. The model that I am looking at is the MSI Fuzzy. I don’t really understand their naming, but this machine would fit our needs perfectly. It has a fast (for the category) processor and slots for 1gb of ram. Most importantly, it allows us to have as many identical machines as we need. The Mac Mini is easier to standardize on because it was fairly popular at the time and there are only one or two minor variations of the PowerPC Mini.

Before we go rushing out and buying a bunch of new machines for this testing, we need to know how long there will be demand for this kind of testing. I would estimate that it would cost about $500 per Geode machine, and that we might be able to get nightly coverage on two to three branches on linux and windows with four machines.

Also of note is that because we are running on such old hardware, we are getting JavaScript timeouts. I don’t really know how to manually set the preference that ignores these timeout warnings, but it is causing a lot of the jobs to be killed off because they aren’t responsive. I’d love to know if you know how I can disable this in an easy to automate way. I was thinking that I could launch and kill Firefox to create a profile, modify the profile’s preference file then launch it again for the real tests. I don’t know which preference file I would need to do this on though.

This is where I’d like to ask people who are working on the JavaScript JIT to let me know how long we need to do this testing and whether this current coverage (mozilla192 and mozilla-central) is enough. A comment on this post, bug 463262 or ping me on #build (jhford).

As a side note, this configuration makes it possible for us to run unit tests on second tier support machines and operating systems. If there is any community interest, I can look making it possible for anyone to connect their machine to this buildbot master if they have an obscure machine with spare cycles. This would require you to be able to dedicate the box for this testing, us to already produce builds that run on that architecture.

If anyone wants, I can get some photos of the machines on Monday.

The third approach that we are going to move forward with uses the imaging process of the second approach to create a ‘master’ image.  This image has all the information already set up, including text files specifying which image revision the device is running on.  Once this is done, we use dd to dump an image of the entire sd card onto the PC’s hard disk (dd if=/dev/sdb of=moz-ref-v1.dump bs=100M). When this is complete, we have a 3.7GB file which contains the entire contents of the master card image. We can then write this file directly to another sd card to get an identical copy of the master (dd if=moz-ref-v1.dump bs=100M). The problem is that this doesn’t scale too well. We are aiming to be able to write to 14 cards at the same time. I have investigated using tee(dd if=moz-ref-v1.dump bs=100M | tee /dev/sdb /dev/sdc /dev/sdd > /dev/null) but found that it wouldn’t write to raw devices. Another option would be to use a for loop and start a bunch of dd processes in the background. While this would have worked, we would be using a really high amount of hard disk throughput, scaling linearly. Instead, I decided to write a limited subset implementation of dd in Python.  I used the optparse library to implement the command line interface and standard Python I/O for the cloning process.  After timing a few runs, I found that my python script was about 98% as fast as the canonical implementation.  I only measured wall time as that is the only thing of value for this situation.

A further optimization that I would like to do is the ability to store the image files in a compressed format and decompress them on the fly.  Because the dump files contain every single bit that the filesystem tracks it ends up being the same size as the SD card itself.  In our case, we have a 4GB filesystem even though only about a gig of that is used.  The most simple way to get around this is to compress the image files.  Rather than worrying about manually decompressing the file before feeding it into the duplication program, I am going to implement decompression in the duplicator program.  I found that Python has a really nice BZip2 module in the standard library.  This is module provides a full file interface for a BZip2 compressed file.  Before I decided to implement this, I wanted to check that the module is able to decompress files on the fly.  I started by generating a file with random data (dd if=/dev/random of=random bs=1024 count=1024) which I then computed a sha1 has for (openssl sha1 < random > random.sha1). At this I opened an interactive Python interpreter and ran the commands:

>>> import bz2
>>> in = bz2.BZ2File('random.bz2')
>>> out = open('random', 'w+')
>>> while True:
...  buffer = f.read(1024)
...  if buffer is '':
...   break
...  o.write(buffer)
...

Once this had completed, I exited the python interpreter and compared my sha1 hashes:

jhford$ cat random.sha1
dc34e2d6308786e5e5857f7b0b1126097060df6c
jhford$ openssl sha1 < random
dc34e2d6308786e5e5857f7b0b1126097060df6c

This tells me that I can safely use the BZ2File class for implementing compressed sd card images. My current implementation strategy is to have files that have a ‘.bz2′ extension automatically treated as either a file that is compressed (input) or should be compressed (output).

I am continually impressed by how comprehensive the Python standard library is.  It seems that every time I write something in Python, there is a built in module to do anything that isn’t specific to the problem at hand!

The code for my duplicator implementation is being developed at http://hg.johnford.info/multi-dd and the imaging scripts for the mobile work lives in the build repository at http://hg.mozilla.org/build/tools in the directory buildfarm/mobile

TinderboxPrint: jford@mozilla.com
TinderboxPrint: 1252624187
Comments: No description given

is not showing up in the waterfall column. I have seen this with other builders, so I am going to assume that it isn’t specific to my code.

This now means that when you submit your code to the Try Server a WinCE build will kick off. I hope that this helps the WinCE team! If you notice anything strange going on, please don’t hesitate to contact me. I am always on irc.mozilla.org in #build as jhford.

I have also moved them all to the conference table in FAIL to see if this improves our connection reliability.  This will really help us to keep posting reliable and timely numbers.  I still have 3 units running in staging which are using a MicroSD card for their entire os and data drive.  If this works out well, we will hopefully improve our end to end times on builds and make it even easier to reimage a malfunctioning device.

I also did an Equifax and Trans Union credit check tonight.  The Equifax one comes up clean as does the majority of the Trans Union one.  Unfortunately, the Trans Union report shows that Rogers has done a credit check on me.  This negatively impacts my credit rating and is the last thing I need right now.  I called the non-emergency number of the Police to report the fraud and was greeted by an apparent wall.  I wasn’t speaking to an officer though.  This person informed me that even though I live in Toronto normally, and that BOTH charges on my card show as Toronto, ON it isn’t a matter for Toronto Police unless I am physically in Toronto.  Nice to see my tax dollars hard at work!  This numpty suggested that I visit the local police here to give a report.  I don’t understand this.  Why on earth would a Mountain View police officer be the one to deal with a crime committed against a Canadian, in Canada with Canadian companies.  I doubt that it would even make its way up to Toronto.

This couldn’t have come at a worse time.  I wonder if the false credit check will be removed from my credit history as I absolutely didn’t request it, the $600 charge or the $1503 charge!  As I sit here, I wonder what I can do regarding this situation.

The new devices still in their boxes

The imaging process that I described in an earlier post requires some time to bake in a staging environment because it requires some fairly significant changes in how the device acts.  In the interim I have generated an image for flashing the N810s that still uses the internal raw flash memory.  To do this we set up a reference device exactly as needed.  After this I generated a new filesystem image (sudo gainroot; mount -t jffs2 /dev/mtdblock4 /opt; mount /dev/mmcblk1p1 /floppy; cd /floppy; mkfs.jffs2 -r /opt -o moz-ref-image-v1.jffs2 -e 128 -l -n). Once this was done I transfered the files to my desktop pc and wrote an automation script that flashes the devices. This puts the root filesytem onto the device. For the pagesets used by talos we need to use the internal controlled flash card. For this I have written a script which takes the device files for each plugged in maemo and rsync’s the contents of the drive. Using a helper script I have made this work on 6 devices at once.

Data being transfered to the new N810s

It took a less than one hour to get all the machines from unopened shipping box to running in our staging environment. If everything looks good, I will move these six new devices over to production tomorrow. This is a significant improvement over the multiple hour per device process of setting up a device we used before. This process scales very well. It takes 90 seconds to flash each device with a root file system and is done in serial. Setting up the page set file system took 6 minutes for 6 devices and is done in parallel.

In Staging

I am eagerly awaiting the arrival of 14 more N810s some time this or next week.