Category Archives: Programming

Our Windows build machines use msys to emulate a posix environment.  Msys is great tool, providing a lot of common posix utilities, like cp, mv, rm.  Sadly there are bugs in the posix emulation.  For us, this manifests in the rm program being unable to delete certain files.

Bug 583129 is about using native Windows tools for file removals.  In that bug, we’ve looked at a combination of the rmdir and attrib Windows tools.  The problem is that rmdir doesn’t like to delete files if they have the read-only or system attribute.  To fix that, we need to run attrib to remove those attributes then rmdir to delete them.  Running rmdir is fast but attrib is very slow.

Last year, I spent a bit of time writing a native windows version of rm.  I am by no means a Windows developer, so I spent a couple hours learning the basics of the Windows API.  The API is quite different to what I’m used to, but the documentation seems to have been written quite well for my purposes.  I was able to get the basics working pretty quickly.  Yesterday, I decided to finish up the program by adding directory deletion, recursive deletes and a command line parser.

I present to you winrm.  The code is available in my Mozilla user repo.  This tool works similarly to the standard posix rm utility.  For simplicity’s sake, only single character options are supported.  These options can be joined, like “-rf”, or specified individually, like “-r -f”.  The “–” option is also supported, signalling the program to treat all following arguments as files to be deleted.  Because of how the option parser works, files are deleted in reverse order to how they are on the command line.

jhford@JHFORD-VM ~/mozilla/jhford-native-rm
$ touch a b c d e

jhford@JHFORD-VM ~/mozilla/jhford-native-rm
$ winrm -v -- a b c d e
deleting "e"
deleted "e"
deleting "d"
deleted "d"
deleting "c"
deleted "c"
deleting "b"
deleted "b"
deleting "a"
deleted "a"

Because my program is written in standard Windows API and is a much simpler program, it is also much faster than the msys rm program.  To test this, I timed deletion of a mozilla-central clone using both mine and msys’ rm.  My program took 37s where the msys program took 113s.

If you know the Windows API and have the cycles, please let me know if you find any glaring errors with my program.   If you want to test the program without having to build it, I’ve uploaded a copy here winrm-0.1.

I want to be able to send all my internet traffic to the Linux machine I have running in my apartment and I am not a networking expert. My motivation for this post is threefold; document my process for future reference, share my info and see if people have suggestions for how to do this better. I am not going to go through every option, just what I did and what worked for me.

The next step was to figure out what I needed to do. I decided on using openvpn because I already use it for work and because it’s open source. I found the how-to document on the openvpn site to be really useful. I am using Fedora, so I skipped the section on installing openvpn from source and ran “sudo yum install openvpn“. My next step was to copy the pki support files into a directory by running “cp -r /usr/share/openvpn/easy-rsa/2.0/* .“. I then followed the directions for generating the pki infrastructure.

For this to work you need an open port on your server. I used the openvpn standard of 1194. I tested that the port was open with netcat by running “nc -l 1194” on my server and “nc server.name 1194“. Writing on either terminal will show the output on the other on EOL.

At this point, I needed to set up the server configuration. I copied the sample config file to my directory by running “cp cp /usr/share/doc/openvpn-2.1.4/sample-config-files/server.conf server.conf“. I found that the sample server config file seemed to work great for me with the following changes:

diff -U0 sample-config-files/server.conf config/server.conf
--- sample-config-files/server.conf	2011-12-12 21:43:31.000000000 -0800
+++ config/server.conf	2011-12-12 22:16:46.000000000 -0800
@@ -196,0 +197,2 @@
+push "dhcp-option DNS 0.0.0.0"
+push "dhcp-option DNS 0.0.0.0"
@@ -204 +206 @@
-;client-to-client
+client-to-client

The first change pushes DNS servers to my client (fake ips, obviously) and the second change is to allow different clients to talk to each other. I am not sure how useful the inter-client link will end up being.

I am using the Viscosity client because that’s the only sane way to do this on OS X and Windows. Sending all traffic over the vpn link is the default behaviour for Network Manager (Linux). I started with the sample by running “cp /usr/share/doc/openvpn-2.1.4/sample-config-files/client.conf .“. My changes where pretty basic:

diff -U0 sample-client.conf client.conf
--- sample-client.conf	2011-12-12 22:43:11.000000000 -0800
+++ client.conf	2011-12-12 21:49:17.000000000 -0800
@@ -42 +42 @@
-remote my-server-1 1194
+remote server.name 1194
@@ -89,2 +89,2 @@
-cert client.crt
-key client.key
+cert laptop.crt
+key laptop.key

At this point, the client side configuration was ready to transfer, so I tarred up the needed files with:

mkdir ovpn-configs
cp keys/ca.crt keys/laptop.crt keys/laptop.key client.conf ovpn-configs/
tar jcf laptop-openvpn-config.tar.bz2 ovpn-configs

and used scp to transfer the files over to my laptop.

Once on my laptop, I untarred the files and imported the configuration into Viscosity. I did this by:

• clicking on Viscosity menu icon then selecting preferences
• clicking on plus arrow with down, selecting “import connection” then selecting “from file”
• selected the client.conf file from the tarball

Next, I configured all my traffic to go over vpn. I selected the “client” configuration from the list of configurations and pressed the “edit” button. In the sheet, I navigated to the “networking” tab and checked the box for “send all traffic over VPN connection”. My client side configuration was complete.

At this stage, I tested that my machine was able to connect to my openvpn server. I gathered the various files needed for the openvpn server into a single directory:

mkdir ~/openvpn-server/
cp keys/* ~/openvpn-server #lazy
cp server.conf ~/openvpn-server

and started the server with “cd ~/openvpn-server && sudo openvpn server.conf“. I connected using viscosity to the server. The client connected properly, but I was unable to resolve anything on dns or reach anything other than my openvpn server. Reading the openvpn howto suggested setting up a NAT. I did some searching and found a page with information on setting up the NAT. I did:

echo 1 > /proc/sys/net/ipv4/ip_forward
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
/sbin/iptables -A FORWARD -i eth0 -o tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A FORWARD -i tun0 -o eth0 -j ACCEPT

At this point, everything worked! I ran traceroute, and the first hop was my vpn server’s vpn address (10.8.0.1). I also used some websites to check my public IP and it was showing as my server’s IP.

I hope this is useful to others. If I’ve done something really dumb, I’d appreciate any suggestions for how to do it better! I have left out information about how to start the openvpn service on boot. This isn’t really important to me right now but if I ever bother with it, I’ll update this blog post.

As of the mozilla-inbound merge this morning, any time automation.py based tests timeout or crash, a screenshot will be base64 encoded and dumped into the test log on OS X. We’ve had this support for a while on Linux and I have matched the output format. In case you aren’t familiar with this, your logs will print out something that looks like:

8217 ERROR TEST-UNEXPECTED-FAIL | /tests/toolkit/content/tests/widgets/test_videocontrols.html | Test timed out.
args: ['/usr/sbin/screencapture', '-C', '-x', '-t', 'png', '/var/folders/Hs/HsDn6a9SG8idoIya6p9mtE+++TI/-Tmp-/mozilla-test-fail_k9Dpdz']
SCREENSHOT: data:image/png;base64,iVBORw0.....

If you want to see this image, copy everything from ‘data:image’, inclusive, to the end of line and paste it into your browser’s awesome bar.

In case you want to see what this looks like in the wild, here is a sample log with a screenshot.

I am working on getting this enabled on windows as well. My automation.py.in changes should easily support the win32 screenshot utility written by Ted in bug 414049.

This project has been discussed in a dev.planning thread, with the work being tracked in bug 658313.  We are going to be turning off PGO for incremental builds on all branches[1] on Wednesday, October 5, 2011.  In the exceedingly rare chance that you don’t read every single bug comment and dev.planning thread post, here is what will change:

• Builds triggered as part of a push will not have PGO enabled on any platform
• We will be producing builds every four hours with PGO enabled on Windows and Linux for the following branches:
• Mozilla-Inbound
• Mozilla-Central
• Mozilla-Aurora
• Mozilla-Beta
• All nightlies produced for platforms we ship PGO enabled (linux, windows) will have PGO on for the nightly build
• Platforms that we ship with PGO enabled will have their PGO talos results report to the current graphserver branch.  This includes nightlies and the new 4-hourly builds
• Platforms that we ship with PGO enabled will have their non-PGO talos results to a ‘-Non-PGO’ suffixed branch, e.g. Firefox-Non-PGO
• Platforms that we do not ship with PGO enabled will report both per-push builds and nightlies  to the current graphserver branch
• TBPL was modified in bug 670037 to make PGO builds special.  The deployment of these changes is being tracked in bug 691550.  If this TBPL change isn’t deployed before we start generating PGO builds, you might see duplicate build and test entries
• Yes, we plan to teach try chooser how to optionally do PGO.  For now, please include ‘mk_add_options MOZ_PGO=1′  in your PGO platform’s mozconfig-extra-$platform file if you wish to have PGO enabled for try.  Your results will be on the Try branch  in graphserver.  This work is being tracked in bug 691673
• Yes, we plan to optimize scheduling so that we only do a build if there has been a push in the previous four hours.  This might allow us to add PGO builds on more branches and is tracked in bug 691675

The motivation for this project is to get results to developers quicker.  It is felt that this reduction in PGO coverage is a safe optimization because there have been very few PGO related bugs found so far.

If you have any concerns, please contact me.  I am jhford in #build on irc.mozilla.org

[1] Well, all active development branches.  We are leaving PGO on for Win32 on branches older than Firefox 5, like mozilla-1.9.2

A while ago, we decided that we needed to expand our pool of machines to test Firefox on.  We are just about done getting the new 10.6 machines ready for production!  The machines we will be enabling are MacMini4,1 spec.  Internally, we are calling them ‘Rev4′ minis.

We won’t have all 80 of these minis ready for production right away, so we’ll start with ten.  Once we have 80 rev4 minis in production and every test suite is running reliably green on them, we will turn off the rev3 10.6 minis so they can be moved to other pools.  If there are suites that are orange, we will hide those results while we look into the failures.

In order to differentiate the talos results from these two different spec machines, we are creating a new platform on the graphserver.  When looking for data on graphserver, the platforms drop down will have two Mac OS X 10.6 options:

• MacOSX 10.6.2 (rev3)
• MacOSX 10.6 (rev4)