Category Archives: School

So this is my forth release and I finally feel like I have something of use to someone other than myself. Today I finished off the first release of Logfribulator. This odd sounding thing is a system which is going to take Mozilla testing logs and inject a serious amount of cool into them. To get to this stage I have written some patches to enable timestamps in the testing infrastructure. So far I have patched xpcshell, feed and reftest output to add “1234567890 | “ as a prefix to their test status messages. I have decided that I am going to try to get the patches through without needing an autoconf option to simplify my work. If this doesn’t work I am going to have to figure out how to pre-process javascript files. If these patches land as is (minus some tweaking in the xpcshell tests), life will be good.

I would like to take this time to introduce you to my system. As it is right now, this is the interface we have for looking at test logs . That same file put through Logfribulator looks something like . This is much easier to look at. Currently there is no sorting or filtering. This is going to be a tough one for me because I will have to do this browser side to avoid the rather long roundtrips. I am guessing that I am going to need to use the DOM to do this, but I am not too sure. I guess this gives me a reason to learn JavaScript more thoroughly

I would love to talk about what I have done to get here but that would take another 3 weeks to write out. Basically, I rewrote the parser that I had before. I added the ability to slurp an error log if it is in between “>>>>>>>” and “<<<<<<<<“ strings on their own line. An example is: TEST-OUTPUT
>>>>>>>
Test failed because…
<<<<<<<
This means that the line from “TEST-OUTPUT” has an error log of “Test failed because….” This is critical to the usefulness of this system because this system focuses on fails, not passes. I also changed the way I was picking out results. Instead of searching for “PASS” and “FAIL”, I am now making heavy use of Python’s regex class. This is a really well written library! I made heavy use of named groupings, which is a nice feature. Basically, when you use a regex like match = re.search(r“^(?P.*?”) (?P.*?)”, “John 416-555-0020”)you will get back a match object which you can search the groupings. Normally, you are limited to using integer keys, but by using the (?P) you can access the groups as a dictionary match.group(‘name’) will give you the name and match.groupdict()[‘number’] will give you the number. This is really neat and really helped keep my code readable.

Another thing I paid attention to in this release a lot was unit tests. I found them to be really useful during my development because I was able to find many errors. I also found that as soon as I discovered a weak spot in my code fixing it in the unit test first was most helpful as I knew when I had fixed the code. That being said, I still need to write a unit test for the ReftestLogLine class I wrote today. This class is particularly cool in my opinion because it has the ability to get the actual comparison images and show them on the webpage in a special way. This is critical for Reftest comparison because they don’t have any test error information other than the images and a count of pixels that are different.

I also learned the python CGI library today and have to say that like every python module I have looked at, it seemed really nice. It took about an hour to go from nothing to a cgi page with forms, and because i have been generating HTML markup since I wrote the parser, it only too a few hours to hook up my generation methods. I found that because of SELinux, on Fedora you can’t use urllib.urlopen in a CGI script because all network access is blocked from httpd. While this is a sensible default to stop people from making web-proxies, it stops my script from working. To enable this I set the SELinux boolean “httpd_can_network_connect” to true. After doing this, my script works

While writing the web ui for the parser today I figured out how I am going to hook this into the build infrastructure. I think I am going to have this script running on a computer and it is going have its database local. I will hook up the applicable buildbot steps to perform a post to an insert cgi page with the URL that the log to parse can be found at. my system will then grab this page, parse the log, insert into its local db then make it available for view using the cgi view.

For my next release I think I am going to focus on the web ui. My goal is to make the table sortable and filterable client-side. This is going to be a lot of work for me because I have never done anything with the DOM before. I am also going to tackle Mochitest output. If this turns out to be easier than I think, I am going to start looking into the database and creating an insertion page. Another thing Dave mentioned that might be a good idea is to link the name of the test to its source. While possible, this would require knowing the objdir of each build so I could lob the rest of the URI off and link it to MXR/OpenGrok. While not impossible, I am going to call this polish to look at when the rest of my system is working.

This week was really fun for me, I really learned a TON about python and it made me really excited that I am going to be working on Mozilla code full time this summer! I also hope that this system is useful to people. Oh, before I forget here is a sample system running Logfribulator here

In an effort to develop a standard place for test failures to go, I have written a new version of my xpcshell test timestamp patch. This patch has been modified as per Ted’s recommendations. After speaking with Ted on IRC I found out that just hard coding enabled timestamps would be ok with him. I have decided to leave in the timestamp enable flag only until I have contacted the owners of Mochitest and Reftest regarding this change. I have decided to attack Mochitest first because I was able to find it’s harness the easiest. It seems to be in runtests.py.in. Conveniently these tests use Python including the optparse library. My task for tomorrow morning is going to be to figure out this library and make the changes needed to allow the enabling/disabling of timestamps. I will try to contact the owners before I write some changes.

Here is a screen shot of my parser in action. I have run a sample log that was generated by some of my patches:

Isn’t that just the most interesting thing you have ever seen

Some lines spill over, but that isn’t highly important right now. There is also tons of bogus information.

Today I did my second demo for Dave. I showed him the starts of my parser and went over my plans. My parser currently has a couple issues with some test and is waiting for time stamps from Reftests and Mochitests. I have also posted this to my bug as a WIP link. Dave and I were discussing how my parser was going to get invoked. Right now my plan is to have my parser as a standalone and library. As a standalone it will be able to parse output from tests and as a library it will be integrated to the special Mozilla BuildBot steps to parse the output. I have noticed that there is a Log File Observer in buildbot which might prove fruitful. Eventually the parsed data will be going into a DB which will likely be SQLite. Dave suggested that I queue up all my transaction in memory then do one large commit so I don’t add too much overhead. I feel that I am on track as far as the parser goes for scanning passing unit tests. I wouldn’t mind writing some simple tests that I know are going to fail so that I can verify my parser’s handling of real failed steps.

On the patch front I am having more success with adding timestamps. My first patch got an r- for a number of totally valid reasons. Ted, who did the review, mentioned that there is little value to being able to specify the time stamp program and format. I think that is right, so in that respect I have decided that I will be using unix time as my format. This makes sense because every language I need to interface with seems to support this format (JavaScript new Date().getTime()/1000, Python time.Time() and Bash date +%s. This makes my life a TON easier! Ted recommended renaming –enable-timestamp to –enable-unit-test-timestamps as the former is very generic sounding. This will be rolled into one big hyper-mega-super-patch I hope to have done very soon. I asked if there was even any value in having timestamps being able to be disabled. Ted said he is ok with that, but that I should talk to the Mochitest and Reftest owners about the possibility.

Regarding the option of enabling/disabling unittest timestamps, I spoke to Dave and Nino about this and found out that I can make javascript files get preprocessed by adding them to the Makefile.in “EXTRA_PP_COMPONENTS” section. This would allow me to use standard preprocessor directives in JavaScript with which I could surround my special printing

This morning I finally got around to asking about the anomalous testing output from the feed unit tests and it seems that it is indeed a bug. I have filed bug 479976 along with a patch which corrects this and adds timestamps to unit test output.

I have had a lot of fun working on this over the past two weeks and feel that I have made some progress. I am also LOVING Python. It is a really neat language. I plan on looking at some sort of Python web framework for the web interface to my parser. Anyone have any recommendations? Ideally it would be something already in use by Mozilla to simplify deployment.

After reading Benjamin Smedberg’s blog post on pymake, I realised that I should be writing unit tests for my parser. I didn’t realise that python had a built in unit testing framework. I am glad I found this out while still in the early stages of my python learning!