Category Archives: Programming

General thoughts on programming

screenresolution tool for Mac OS X 10.6 release

I have been working on getting our new Mac Mini (macmini4,1) machines ready to become Mac OS X 10.6 testing machines.  One of the last things needed before I start testing on a larger scale is a program to set the resolution on the command line.  We need this because our tests depend on having a specific resolution set.  Even if there was no dependency on a specific resolution, the GPU could perform differently depending on the size of the display it is working on.  As well, we run these minis without a real display so its really important that we have active confirmation that the desired resolution is set.

Previously, we used cscreen.  This is a utility that has fallen off the internet and was only ever released as a PowerPC binary.  I needed to decide between finding a replacement for cscreen and installing Rosetta.  While we currently have Rosetta on our testing slaves, I didn’t want us to depend on this.  This is because partly because I just don’t think its good hygiene and partly because I want to prepare for OS X 10.7, where Rosetta has been removed.  I searched the internet for a tool to do this, and sadly, there was nothing that fit the bill.  I tried alternatives like modifying the com.apple.Boot.plist file to specify resolution, but it didn’t work.  I also found some binary plist files that I could have edited and might have worked, but it seemed like a really flaky solution that would break *all* *the* *time*.  I decided that I’d have to write something to do this.

I did find a program that looked like it would be a good starting point, however, this program lacked any error checking.  Furthermore, when I investigated the APIs used, it looked like many of the important APIs were deprecated in 10.6 and would not be available in 10.7.  One particularly annoying deprecation was CGDisplayBestModeForParameters, because it takes the exact data I have (height, width and bit depth) and picked the best mode available.  In the end, I think I have a better solution that iterates through all modes and picks one that matches my inputs exactly.  I also noticed that CGDisplayBestModeForParameters doesn’t always return a mode that matches exactly to what was requestion, though, it does have facilities to deal with this case.

Alas, I started writing an application called screenresolution.  This program can be used to query the current and available resolutions as well as setting the resolution on one or more screens attached to the system.  This is my first C project in quite some time and I was never a C rockstar to begin with.  If you have a desire for Mozilla testing infrastructure to not suck and know C and CoreGraphics please slide on over to the source to check for issues!  I know that I could organize the code better and write a better command line interface, but I don’t think the benefits of that outweigh the cost of waiting even longer for these new machines.

I have posted thecode on my github account and have licensed it with the GPL v2 license in hopes that no one ever go through the search engine hell that was Friday Aug 26, 2011.  Below is a build of verison 1.3.  Why not 1.0?  Even though I wrote a test for the program, I neglected to actually run it. Multiple times :(   Once I started to actually run the tests, I started finding issues before bumping version numbers, gasp!

screenresolution-1.3.dmg

Update: Thanks Josh Aas and Richard Newman for reviewing my code and the great suggestions! I have a new version posted
screenresolution-1.4.dmg

Because this is a command line application, I recorded a video of my machine running the test suite.

Figuring out which files are touched while installing software

We are working on getting our infrastructure up to speed for Maemo 5 and Maemo 5 QT builds. A critical part of Maemo5 building is the scratchbox. This is a toolkit that Nokia uses to make development on for their linux based phones easier. We have enough linux build slaves in production that it is impractical to deploy scratchbox by hand on each machine. Scratchbox also does internet package downloads which means that we could get different packages each time we try to install the scratchbox. We already have a fairly old version of scratchbox which is set up with the Chinook sdk that we have used for doing our Maemo builds thus far. Originally I was under the impression that we were going to need to have 2 totally seperate scratchbox installations but thanks to Doug T. for showing me how to upgrade our existing scratchbox 4 installation to scratchbox 5.

My concern with this upgrade, however, was that files outside the /builds/scratchbox directory were going to be touched. I wanted to be thorough so I did an experiment. I ran find -mount -type f -exec openssl md5 '{}' \; | tee -a /file-list ; find -mount -type f -exec openssl md5 '{}' \; | tee -a /file-list before and after the scratchbox upgrade. The -mount and two different runs at our two mountpoints was to ensure that we didn’t hash things like the /dev, /proc, /sys filesystem. My original intent was to do diff file-list1 file-list2 but that resulted in showing me every single file that changed. I only wanted to know the files that changed outside of my scratchbox root directory of /builds/scratchbox. My diff was polluted by 77,000 files that resided in the scratcbox root. I figured that the best option at the time was to hack up a quick python script:

#!/usr/bin/python
#This file is a quick script to process the output of
# find / -mount -type f -exec openssl md5 '{}' \; | tee -a
import sys, os.path, re

if not len(sys.argv) == 3:
    print "purple monkey dishwasher"
    exit(1)
filename_a = sys.argv[1]
filename_b = sys.argv[2]
if not os.path.exists(filename_a) or not os.path.exists(filename_b):
    print "insert change into meter and press green button"
    exit(1)
data={}
pattern = re.compile("^MD5\((?P.*)\)= (?P.*)$")
#Get the data from A
f = open(filename_a, 'r')
for i in f.readlines():
    m = pattern.search(i)
    data[m.group('file')] = m.group('hash')
f.close()
f = open(filename_b, 'r')
sbfile = re.compile("^/builds/scratchbox") #pattern describing files to ignore
#Figure out diff to B
f = open(filename_b, 'r')
for i in f.readlines():
    m = pattern.search(i)
    if not data.has_key(m.group('file')):
        if not sbfile.search(m.group('file')):
            print 'new file - ', m.group('file')
    else:
         if not sbfile.search(m.group('file')):
             if not data[m.group('file')] == m.group('hash'):
                 print 'updated file - ', m.group('file')

This is code I have written to scratch my own itch. I am posting this as it might be useful to someone else. if you wanted to ignore a different directory you’d change sbfile = re.compile("^/builds/scratchbox") to be a pattern describing your path to ignore. If you wanted to find all things that changed over your whole partition you would remove sbfile and all sbfile checks to have a final bit of code like

#Figure out diff to B
f = open(filename_b, 'r')
for i in f.readlines():
    m = pattern.search(i)
    if not data.has_key(m.group('file')):
        print 'new file - ', m.group('file')
    else:
         if not data[m.group('file')] == m.group('hash'):
             print 'updated file - ', m.group('file')

In the end, I found that the scratchbox upgrade that I did only changed my bash_history and added some tarballs to /tmp. I am very glad that this is the case as it really simplifies our deployment of the new scratchbox!

Python Frameworks

I came to a descision today that I was going to learn a Python ORM framework to use. My project is going to require a lot of database access and using an ORM framework simplifies this. The ORM also hides many of the complexities. If you are interested, I’d highly recomend following this tutorial.

In doing some research into my CGI + SQLite issue I have been constantly asked “…but why are you using CGI”. It turns out that CGI + Python is not really used much. Many people recomend using mod_wsgi, and I think that instead of using raw wsgi I would like to use a framework. So far, CherryPy is what is sticking out to me because it is very lightweight and includes its own development server. This makes it easier to test my code because I (hopefully) would be able to use the Eclipse/Pydev debugger. This is something I have wanted to for quite some time. The main reason I am looking into CherryPy is that it seems to not worry itself with forcing a templating system on you.

In a related note, I found a JavaScript jQuery plugin which looks really neat and could prove to be very valuable. It is called TableSorter. There is an example of a pagination system that really caught my eye here.

OSX Development

I am noticing that a lot of people are having trouble getting used to developing on OSX. I hope that this is useful to those people, I am targeting someone who has at least some experience with Linux or another unix-like OS. I haven’t been using OSX long myself, but I have adapted quickly :)

What is XCode?
XCode is more than an application, it is a complete development environment. When someone ‘installs xcode’ they are installing compilers, frameworks, headers and build tools. If you can compile a C program you have XCode. Just to be sure, you can check by using commands like this

vortex:~ jhford$ which gcc/usr/bin/gccvortex:~ jhford$ which g++/usr/bin/g++vortex:~ jhford$ which make/usr/bin/makevortex:~ jhford$ which xcodebuild/usr/bin/xcodebuild

What is MacPorts
MacPorts is a system utility to ease installing non-apple unix software. This program is used to install things like subversion, mercurial, libidl, autoconf213 and ccache. Basically, what this program does is fetch sources, patches and build instructions and compiles the program for your system. This is very similar to BSD ports. To use the utility, you use the port command. Some sample uses:

sudo port install ccache mercurial libidl autoconf213port search subversionport list installed

The man page is the best place to go for help with MacPorts (man port). Unlike Fink, which uses Debian/Ubuntu’s apt-get internally, all things you install through MacPorts are compiled on your machine during install. Word of warning: don’t need anything in a hurry! Our network is notoriously slow with MacPorts.

What is a .dmg?
A .dmg is a compressed disc image format. It contains one or more filessytems, usually HFS+ (read: mac filesystem). This is how most .app and .pkg files are distributed because it maintains the mac specific file meta data.

What is a .app? .pkg?
You might have noticed that some applications are just icons, like Firefox or OpenOffice. To install them you drag the icon into your /Applications folder and boom, intalled. The most important thing to remember here is .app is just a folder! I will prove it

This means that if you want to start an application that is in a .app from the command line to see console output, you can do something like $ /Applications/Firefox.app/Contents/MacOS/firefox or /Applications/OpenOffice.org.app/Contents/MacOS/soffice. Usually your application will live under Appname.app/Contents/MacOS/

A .pkg file is an installer file, kind of like a .run script on linux or a .msi file in windows. It is too just a folder. Lots of things that are folders have file extensions on OSX. If in doubt, run file on it in Terminal

Universal Binary?
In case you didn’t know, Macs used to have PowerPC cpus. To ease the transition from PowerPC to Intel, Apple created something called a ‘Universal Binary’. These are basically executable files which can be run on either processor. If you want to check if you have made one you can use the unix command ‘file’ on the application. An example of a Firefox release binary shows that this is a universal binary:

vortex:Firefox.app jhford$ file Contents/MacOS/firefox-bin Contents/MacOS/firefox-bin: Mach-O universal binary with 2 architecturesContents/MacOS/firefox-bin (for architecture i386): Mach-O executable i386Contents/MacOS/firefox-bin (for architecture ppc): Mach-O executable ppc

Sometimes the second architecture will mention ppc7440 or something similar, this means it is a G4+ binary (i.e. requires altivec, similar to MMX/SSE).

Command, Option, Control?
I guess this is only applicable if you are in front of the machine. Command, Option and Control are similar to but not identical to Start, Alt and Control. In OSX, things are a little more logical. If the key combo you want to use involves doing something, say, close a tab, it is Command + W. In Windows/Linux it is Control + W. Keep this in mind, nearly all ctrl or alt key combos for GUI programs on Windows or Linux are the same on Mac but with Command. Option, like the name implies, give you an option. Say you have save and save as. Command + S is like save and since save all is so similar, it may have Command + Option + S. Worst case, you can look around in the menu bar at the top of the screen. A clover-ish thing means Command, a downward sloping line is Option and an up arrow is Shift.

On the command line, Control works just like ctrl in Windows or Linux. Control + D will give you EOF, Control + C will quit the application.

Screenshots!
To take a screenshot simple press one of:
Command + Shift + 3 – Fullscreen
Command + Shift + 4 – An area of the screen, like Snipping Tool in Vista
Command + Shift + 4 then release and press space – A ‘view’, could be a window or a sheet. Like Alt+Printscreen
These commmands will place the pictures on your desktop in the format PictureX.png where X is automatically incrementing.

Hopefully this helps you get the hang of Mac. If you have any questions feel free to leave a comment or ping me in IRC at irc://irc.mozilla.org/#seneca

Pydoc Server ?!?!

For all of you working on anything python:pydoc -p 3500. This will create an http server at http://localhost:3500/ to browse all the python module documentation currently loaded.