Through my work on imaging our Nokia test farm, I have developed 3 approaches to imaging the n810. The first is to set up an N810 then generate our own firmware image as a JFFS2 filesystem. This approach gives us an N810 that is essentially factory-stock. We found that we were still having devices fall over on a regular basis with this approach. The next approach I tried was to put the full Maemo operating system on an SD card and boot from it. This resulted in significantly improved reliability at a minor cost in test suite run time. The actual imaging process is far less human involved than the older approach. All that is required is that somebody is there to change blank sd cards and execute the command again. This process used rsync (sudo rsync -a moz-ref-v2/. /mount/point/.) to copy the files from a directory on the imaging machine onto the SD card. As a part of the imaging process, the hostname is set as are a couple other bits of information. It took me the better part of 3 days to image all 40 sd cards using this approach.
The third approach that we are going to move forward with uses the imaging process of the second approach to create a ‘master’ image. This image has all the information already set up, including text files specifying which image revision the device is running on. Once this is done, we use dd to dump an image of the entire sd card onto the PC’s hard disk (dd if=/dev/sdb of=moz-ref-v1.dump bs=100M). When this is complete, we have a 3.7GB file which contains the entire contents of the master card image. We can then write this file directly to another sd card to get an identical copy of the master (dd if=moz-ref-v1.dump bs=100M). The problem is that this doesn’t scale too well. We are aiming to be able to write to 14 cards at the same time. I have investigated using tee(dd if=moz-ref-v1.dump bs=100M | tee /dev/sdb /dev/sdc /dev/sdd > /dev/null) but found that it wouldn’t write to raw devices. Another option would be to use a for loop and start a bunch of dd processes in the background. While this would have worked, we would be using a really high amount of hard disk throughput, scaling linearly. Instead, I decided to write a limited subset implementation of dd in Python. I used the optparse library to implement the command line interface and standard Python I/O for the cloning process. After timing a few runs, I found that my python script was about 98% as fast as the canonical implementation. I only measured wall time as that is the only thing of value for this situation.
A further optimization that I would like to do is the ability to store the image files in a compressed format and decompress them on the fly. Because the dump files contain every single bit that the filesystem tracks it ends up being the same size as the SD card itself. In our case, we have a 4GB filesystem even though only about a gig of that is used. The most simple way to get around this is to compress the image files. Rather than worrying about manually decompressing the file before feeding it into the duplication program, I am going to implement decompression in the duplicator program. I found that Python has a really nice BZip2 module in the standard library. This is module provides a full file interface for a BZip2 compressed file. Before I decided to implement this, I wanted to check that the module is able to decompress files on the fly. I started by generating a file with random data (dd if=/dev/random of=random bs=1024 count=1024) which I then computed a sha1 has for (openssl sha1 < random > random.sha1). At this I opened an interactive Python interpreter and ran the commands:
>>> import bz2
>>> in = bz2.BZ2File('random.bz2')
>>> out = open('random', 'w+')
>>> while True:
... buffer = f.read(1024)
... if buffer is '':
... break
... o.write(buffer)
...
Once this had completed, I exited the python interpreter and compared my sha1 hashes:
jhford$ cat random.sha1 dc34e2d6308786e5e5857f7b0b1126097060df6c jhford$ openssl sha1 < random dc34e2d6308786e5e5857f7b0b1126097060df6c
This tells me that I can safely use the BZ2File class for implementing compressed sd card images. My current implementation strategy is to have files that have a ‘.bz2′ extension automatically treated as either a file that is compressed (input) or should be compressed (output).
I am continually impressed by how comprehensive the Python standard library is. It seems that every time I write something in Python, there is a built in module to do anything that isn’t specific to the problem at hand!
The code for my duplicator implementation is being developed at http://hg.johnford.info/multi-dd and the imaging scripts for the mobile work lives in the build repository at http://hg.mozilla.org/build/tools in the directory buildfarm/mobile
