Bug 583129 is about using native Windows tools for file removals.  In that bug, we’ve looked at a combination of the rmdir and attrib Windows tools.  The problem is that rmdir doesn’t like to delete files if they have the read-only or system attribute.  To fix that, we need to run attrib to remove those attributes then rmdir to delete them.  Running rmdir is fast but attrib is very slow.

Last year, I spent a bit of time writing a native windows version of rm.  I am by no means a Windows developer, so I spent a couple hours learning the basics of the Windows API.  The API is quite different to what I’m used to, but the documentation seems to have been written quite well for my purposes.  I was able to get the basics working pretty quickly.  Yesterday, I decided to finish up the program by adding directory deletion, recursive deletes and a command line parser.

I present to you winrm.  The code is available in my Mozilla user repo.  This tool works similarly to the standard posix rm utility.  For simplicity’s sake, only single character options are supported.  These options can be joined, like “-rf”, or specified individually, like “-r -f”.  The “–” option is also supported, signalling the program to treat all following arguments as files to be deleted.  Because of how the option parser works, files are deleted in reverse order to how they are on the command line.

jhford@JHFORD-VM ~/mozilla/jhford-native-rm
$ touch a b c d e

jhford@JHFORD-VM ~/mozilla/jhford-native-rm
$ winrm -v -- a b c d e
deleting "e"
deleted "e"
deleting "d"
deleted "d"
deleting "c"
deleted "c"
deleting "b"
deleted "b"
deleting "a"
deleted "a"

Because my program is written in standard Windows API and is a much simpler program, it is also much faster than the msys rm program.  To test this, I timed deletion of a mozilla-central clone using both mine and msys’ rm.  My program took 37s where the msys program took 113s.

If you know the Windows API and have the cycles, please let me know if you find any glaring errors with my program.   If you want to test the program without having to build it, I’ve uploaded a copy here winrm-0.1.

The next step was to figure out what I needed to do. I decided on using openvpn because I already use it for work and because it’s open source. I found the how-to document on the openvpn site to be really useful. I am using Fedora, so I skipped the section on installing openvpn from source and ran “sudo yum install openvpn“. My next step was to copy the pki support files into a directory by running “cp -r /usr/share/openvpn/easy-rsa/2.0/* .“. I then followed the directions for generating the pki infrastructure.

For this to work you need an open port on your server. I used the openvpn standard of 1194. I tested that the port was open with netcat by running “nc -l 1194” on my server and “nc server.name 1194“. Writing on either terminal will show the output on the other on EOL.

At this point, I needed to set up the server configuration. I copied the sample config file to my directory by running “cp cp /usr/share/doc/openvpn-2.1.4/sample-config-files/server.conf server.conf“. I found that the sample server config file seemed to work great for me with the following changes:

diff -U0 sample-config-files/server.conf config/server.conf
--- sample-config-files/server.conf	2011-12-12 21:43:31.000000000 -0800
+++ config/server.conf	2011-12-12 22:16:46.000000000 -0800
@@ -196,0 +197,2 @@
+push "dhcp-option DNS 0.0.0.0"
+push "dhcp-option DNS 0.0.0.0"
@@ -204 +206 @@
-;client-to-client
+client-to-client

The first change pushes DNS servers to my client (fake ips, obviously) and the second change is to allow different clients to talk to each other. I am not sure how useful the inter-client link will end up being.

I am using the Viscosity client because that’s the only sane way to do this on OS X and Windows. Sending all traffic over the vpn link is the default behaviour for Network Manager (Linux). I started with the sample by running “cp /usr/share/doc/openvpn-2.1.4/sample-config-files/client.conf .“. My changes where pretty basic:

diff -U0 sample-client.conf client.conf
--- sample-client.conf	2011-12-12 22:43:11.000000000 -0800
+++ client.conf	2011-12-12 21:49:17.000000000 -0800
@@ -42 +42 @@
-remote my-server-1 1194
+remote server.name 1194
@@ -89,2 +89,2 @@
-cert client.crt
-key client.key
+cert laptop.crt
+key laptop.key

At this point, the client side configuration was ready to transfer, so I tarred up the needed files with:

mkdir ovpn-configs
cp keys/ca.crt keys/laptop.crt keys/laptop.key client.conf ovpn-configs/
tar jcf laptop-openvpn-config.tar.bz2 ovpn-configs

and used scp to transfer the files over to my laptop.

Once on my laptop, I untarred the files and imported the configuration into Viscosity. I did this by:

• clicking on Viscosity menu icon then selecting preferences
• clicking on plus arrow with down, selecting “import connection” then selecting “from file”
• selected the client.conf file from the tarball

Next, I configured all my traffic to go over vpn. I selected the “client” configuration from the list of configurations and pressed the “edit” button. In the sheet, I navigated to the “networking” tab and checked the box for “send all traffic over VPN connection”. My client side configuration was complete.

At this stage, I tested that my machine was able to connect to my openvpn server. I gathered the various files needed for the openvpn server into a single directory:

mkdir ~/openvpn-server/
cp keys/* ~/openvpn-server #lazy
cp server.conf ~/openvpn-server

and started the server with “cd ~/openvpn-server && sudo openvpn server.conf“. I connected using viscosity to the server. The client connected properly, but I was unable to resolve anything on dns or reach anything other than my openvpn server. Reading the openvpn howto suggested setting up a NAT. I did some searching and found a page with information on setting up the NAT. I did:

echo 1 > /proc/sys/net/ipv4/ip_forward
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
/sbin/iptables -A FORWARD -i eth0 -o tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A FORWARD -i tun0 -o eth0 -j ACCEPT

At this point, everything worked! I ran traceroute, and the first hop was my vpn server’s vpn address (10.8.0.1). I also used some websites to check my public IP and it was showing as my server’s IP.

I hope this is useful to others. If I’ve done something really dumb, I’d appreciate any suggestions for how to do it better! I have left out information about how to start the openvpn service on boot. This isn’t really important to me right now but if I ever bother with it, I’ll update this blog post.

8217 ERROR TEST-UNEXPECTED-FAIL | /tests/toolkit/content/tests/widgets/test_videocontrols.html | Test timed out.
args: ['/usr/sbin/screencapture', '-C', '-x', '-t', 'png', '/var/folders/Hs/HsDn6a9SG8idoIya6p9mtE+++TI/-Tmp-/mozilla-test-fail_k9Dpdz']
SCREENSHOT: data:image/png;base64,iVBORw0.....

If you want to see this image, copy everything from ‘data:image’, inclusive, to the end of line and paste it into your browser’s awesome bar.

In case you want to see what this looks like in the wild, here is a sample log with a screenshot.

I am working on getting this enabled on windows as well. My automation.py.in changes should easily support the win32 screenshot utility written by Ted in bug 414049.

• Builds triggered as part of a push will not have PGO enabled on any platform
• We will be producing builds every four hours with PGO enabled on Windows and Linux for the following branches:
• Mozilla-Inbound
• Mozilla-Central
• Mozilla-Aurora
• Mozilla-Beta
• All nightlies produced for platforms we ship PGO enabled (linux, windows) will have PGO on for the nightly build
• Platforms that we ship with PGO enabled will have their PGO talos results report to the current graphserver branch.  This includes nightlies and the new 4-hourly builds
• Platforms that we ship with PGO enabled will have their non-PGO talos results to a ‘-Non-PGO’ suffixed branch, e.g. Firefox-Non-PGO
• Platforms that we do not ship with PGO enabled will report both per-push builds and nightlies  to the current graphserver branch
• TBPL was modified in bug 670037 to make PGO builds special.  The deployment of these changes is being tracked in bug 691550.  If this TBPL change isn’t deployed before we start generating PGO builds, you might see duplicate build and test entries
• Yes, we plan to teach try chooser how to optionally do PGO.  For now, please include ‘mk_add_options MOZ_PGO=1′  in your PGO platform’s mozconfig-extra-$platform file if you wish to have PGO enabled for try.  Your results will be on the Try branch  in graphserver.  This work is being tracked in bug 691673
• Yes, we plan to optimize scheduling so that we only do a build if there has been a push in the previous four hours.  This might allow us to add PGO builds on more branches and is tracked in bug 691675

The motivation for this project is to get results to developers quicker.  It is felt that this reduction in PGO coverage is a safe optimization because there have been very few PGO related bugs found so far.

If you have any concerns, please contact me.  I am jhford in #build on irc.mozilla.org

[1] Well, all active development branches.  We are leaving PGO on for Win32 on branches older than Firefox 5, like mozilla-1.9.2

We won’t have all 80 of these minis ready for production right away, so we’ll start with ten.  Once we have 80 rev4 minis in production and every test suite is running reliably green on them, we will turn off the rev3 10.6 minis so they can be moved to other pools.  If there are suites that are orange, we will hide those results while we look into the failures.

In order to differentiate the talos results from these two different spec machines, we are creating a new platform on the graphserver.  When looking for data on graphserver, the platforms drop down will have two Mac OS X 10.6 options:

• MacOSX 10.6.2 (rev3)
• MacOSX 10.6 (rev4)

Previously, we used cscreen.  This is a utility that has fallen off the internet and was only ever released as a PowerPC binary.  I needed to decide between finding a replacement for cscreen and installing Rosetta.  While we currently have Rosetta on our testing slaves, I didn’t want us to depend on this.  This is because partly because I just don’t think its good hygiene and partly because I want to prepare for OS X 10.7, where Rosetta has been removed.  I searched the internet for a tool to do this, and sadly, there was nothing that fit the bill.  I tried alternatives like modifying the com.apple.Boot.plist file to specify resolution, but it didn’t work.  I also found some binary plist files that I could have edited and might have worked, but it seemed like a really flaky solution that would break *all* *the* *time*.  I decided that I’d have to write something to do this.

I did find a program that looked like it would be a good starting point, however, this program lacked any error checking.  Furthermore, when I investigated the APIs used, it looked like many of the important APIs were deprecated in 10.6 and would not be available in 10.7.  One particularly annoying deprecation was CGDisplayBestModeForParameters, because it takes the exact data I have (height, width and bit depth) and picked the best mode available.  In the end, I think I have a better solution that iterates through all modes and picks one that matches my inputs exactly.  I also noticed that CGDisplayBestModeForParameters doesn’t always return a mode that matches exactly to what was requestion, though, it does have facilities to deal with this case.

Alas, I started writing an application called screenresolution.  This program can be used to query the current and available resolutions as well as setting the resolution on one or more screens attached to the system.  This is my first C project in quite some time and I was never a C rockstar to begin with.  If you have a desire for Mozilla testing infrastructure to not suck and know C and CoreGraphics please slide on over to the source to check for issues!  I know that I could organize the code better and write a better command line interface, but I don’t think the benefits of that outweigh the cost of waiting even longer for these new machines.

I have posted thecode on my github account and have licensed it with the GPL v2 license in hopes that no one ever go through the search engine hell that was Friday Aug 26, 2011.  Below is a build of verison 1.3.  Why not 1.0?  Even though I wrote a test for the program, I neglected to actually run it. Multiple times   Once I started to actually run the tests, I started finding issues before bumping version numbers, gasp!

screenresolution-1.3.dmg

Update: Thanks Josh Aas and Richard Newman for reviewing my code and the great suggestions! I have a new version posted
screenresolution-1.4.dmg

Because this is a command line application, I recorded a video of my machine running the test suite.

My concern with this upgrade, however, was that files outside the /builds/scratchbox directory were going to be touched. I wanted to be thorough so I did an experiment. I ran find -mount -type f -exec openssl md5 '{}' \; | tee -a /file-list ; find -mount -type f -exec openssl md5 '{}' \; | tee -a /file-list before and after the scratchbox upgrade. The -mount and two different runs at our two mountpoints was to ensure that we didn’t hash things like the /dev, /proc, /sys filesystem. My original intent was to do diff file-list1 file-list2 but that resulted in showing me every single file that changed. I only wanted to know the files that changed outside of my scratchbox root directory of /builds/scratchbox. My diff was polluted by 77,000 files that resided in the scratcbox root. I figured that the best option at the time was to hack up a quick python script:

#!/usr/bin/python
#This file is a quick script to process the output of
# find / -mount -type f -exec openssl md5 '{}' \; | tee -a
import sys, os.path, re

if not len(sys.argv) == 3:
    print "purple monkey dishwasher"
    exit(1)
filename_a = sys.argv[1]
filename_b = sys.argv[2]
if not os.path.exists(filename_a) or not os.path.exists(filename_b):
    print "insert change into meter and press green button"
    exit(1)
data={}
pattern = re.compile("^MD5\((?P.*)\)= (?P.*)$")
#Get the data from A
f = open(filename_a, 'r')
for i in f.readlines():
    m = pattern.search(i)
    data[m.group('file')] = m.group('hash')
f.close()
f = open(filename_b, 'r')
sbfile = re.compile("^/builds/scratchbox") #pattern describing files to ignore
#Figure out diff to B
f = open(filename_b, 'r')
for i in f.readlines():
    m = pattern.search(i)
    if not data.has_key(m.group('file')):
        if not sbfile.search(m.group('file')):
            print 'new file - ', m.group('file')
    else:
         if not sbfile.search(m.group('file')):
             if not data[m.group('file')] == m.group('hash'):
                 print 'updated file - ', m.group('file')

This is code I have written to scratch my own itch. I am posting this as it might be useful to someone else. if you wanted to ignore a different directory you’d change sbfile = re.compile("^/builds/scratchbox") to be a pattern describing your path to ignore. If you wanted to find all things that changed over your whole partition you would remove sbfile and all sbfile checks to have a final bit of code like

#Figure out diff to B
f = open(filename_b, 'r')
for i in f.readlines():
    m = pattern.search(i)
    if not data.has_key(m.group('file')):
        print 'new file - ', m.group('file')
    else:
         if not data[m.group('file')] == m.group('hash'):
             print 'updated file - ', m.group('file')

In the end, I found that the scratchbox upgrade that I did only changed my bash_history and added some tarballs to /tmp. I am very glad that this is the case as it really simplifies our deployment of the new scratchbox!

In doing some research into my CGI + SQLite issue I have been constantly asked “…but why are you using CGI”. It turns out that CGI + Python is not really used much. Many people recomend using mod_wsgi, and I think that instead of using raw wsgi I would like to use a framework. So far, CherryPy is what is sticking out to me because it is very lightweight and includes its own development server. This makes it easier to test my code because I (hopefully) would be able to use the Eclipse/Pydev debugger. This is something I have wanted to for quite some time. The main reason I am looking into CherryPy is that it seems to not worry itself with forcing a templating system on you.

In a related note, I found a JavaScript jQuery plugin which looks really neat and could prove to be very valuable. It is called TableSorter. There is an example of a pagination system that really caught my eye here.

What is XCode?
XCode is more than an application, it is a complete development environment. When someone ‘installs xcode’ they are installing compilers, frameworks, headers and build tools. If you can compile a C program you have XCode. Just to be sure, you can check by using commands like this

vortex:~ jhford$ which gcc/usr/bin/gccvortex:~ jhford$ which g++/usr/bin/g++vortex:~ jhford$ which make/usr/bin/makevortex:~ jhford$ which xcodebuild/usr/bin/xcodebuild

What is MacPorts
MacPorts is a system utility to ease installing non-apple unix software. This program is used to install things like subversion, mercurial, libidl, autoconf213 and ccache. Basically, what this program does is fetch sources, patches and build instructions and compiles the program for your system. This is very similar to BSD ports. To use the utility, you use the port command. Some sample uses:

sudo port install ccache mercurial libidl autoconf213port search subversionport list installed

The man page is the best place to go for help with MacPorts (man port). Unlike Fink, which uses Debian/Ubuntu’s apt-get internally, all things you install through MacPorts are compiled on your machine during install. Word of warning: don’t need anything in a hurry! Our network is notoriously slow with MacPorts.

What is a .dmg?
A .dmg is a compressed disc image format. It contains one or more filessytems, usually HFS+ (read: mac filesystem). This is how most .app and .pkg files are distributed because it maintains the mac specific file meta data.

What is a .app? .pkg?
You might have noticed that some applications are just icons, like Firefox or OpenOffice. To install them you drag the icon into your /Applications folder and boom, intalled. The most important thing to remember here is .app is just a folder! I will prove it

This means that if you want to start an application that is in a .app from the command line to see console output, you can do something like $ /Applications/Firefox.app/Contents/MacOS/firefox or /Applications/OpenOffice.org.app/Contents/MacOS/soffice. Usually your application will live under Appname.app/Contents/MacOS/

A .pkg file is an installer file, kind of like a .run script on linux or a .msi file in windows. It is too just a folder. Lots of things that are folders have file extensions on OSX. If in doubt, run file on it in Terminal

Universal Binary?
In case you didn’t know, Macs used to have PowerPC cpus. To ease the transition from PowerPC to Intel, Apple created something called a ‘Universal Binary’. These are basically executable files which can be run on either processor. If you want to check if you have made one you can use the unix command ‘file’ on the application. An example of a Firefox release binary shows that this is a universal binary:

vortex:Firefox.app jhford$ file Contents/MacOS/firefox-bin Contents/MacOS/firefox-bin: Mach-O universal binary with 2 architecturesContents/MacOS/firefox-bin (for architecture i386): Mach-O executable i386Contents/MacOS/firefox-bin (for architecture ppc): Mach-O executable ppc

Sometimes the second architecture will mention ppc7440 or something similar, this means it is a G4+ binary (i.e. requires altivec, similar to MMX/SSE).

Command, Option, Control?
I guess this is only applicable if you are in front of the machine. Command, Option and Control are similar to but not identical to Start, Alt and Control. In OSX, things are a little more logical. If the key combo you want to use involves doing something, say, close a tab, it is Command + W. In Windows/Linux it is Control + W. Keep this in mind, nearly all ctrl or alt key combos for GUI programs on Windows or Linux are the same on Mac but with Command. Option, like the name implies, give you an option. Say you have save and save as. Command + S is like save and since save all is so similar, it may have Command + Option + S. Worst case, you can look around in the menu bar at the top of the screen. A clover-ish thing means Command, a downward sloping line is Option and an up arrow is Shift.

On the command line, Control works just like ctrl in Windows or Linux. Control + D will give you EOF, Control + C will quit the application.

Screenshots!
To take a screenshot simple press one of:
Command + Shift + 3 – Fullscreen
Command + Shift + 4 – An area of the screen, like Snipping Tool in Vista
Command + Shift + 4 then release and press space – A ‘view’, could be a window or a sheet. Like Alt+Printscreen
These commmands will place the pictures on your desktop in the format PictureX.png where X is automatically incrementing.

Hopefully this helps you get the hang of Mac. If you have any questions feel free to leave a comment or ping me in IRC at irc://irc.mozilla.org/#seneca