For anyone interested, I have posted my code through a BitBucked repository here:http://bitbucket.org/jhford/logfribulator/

I plan to make a video to demonstrate my system tomorrow as it requires quite a bit of setup to be fully functional

I have finally finished my DPS911 course work. For this release I have written a log view page and hooked Logfribulator into the BuildBot. After fixing a bunch of bugs with data persistence I wrote a viewing page. Because of the data persistence framework I am using (SQLAlchemy), this was an enjoyable experience. This framework makes querying very logical, almost functioning like jQuery. I haven’t enabled the filtering that was on the old page because I am going to be re-implementing the HTML generation using Pylon’s Mako templating engine. Originally I had planned to use CherryPy for the controller of my page.

After a chat on irc.freenode.net/#python I have decided to look at Pylons instead. It is a popular framework and already has SQLAlchemy built in. The only reason I don’t like this framework is because I don’t understand how it works yet. Unlike CherryPy, I haven’t yet figured out how to run the development server from within Eclipse. This is something which would be very nice as I love the Eclipse debugger interface.

To help with debugging my system and aid in hacking on BuildBot, I wrote a simple ‘dump’ script in Python. I used SQLAlchemy to create a simple table which I could use anywhere. This was a great solution for my next bit of hacking. I finally hooked up BuildBot to submit logs to Logfribulator as they are completed. To do this I spent quite a few hours figuring out how each class in BuildBot is connected. In the end I was successful. I found that between the exception handler build into BuildBot and my database dump script this wasn’t as painful as I thought it would be.

The main thing I learned from this exercise is that code searching is invaluable. I used the built in Eclipse project search function. An example was where the string form of an object had the information I wanted. It would print in the format <Build ‘full-osx’>. When I did a search for “<Build “ I was directed to code which shows how this information was retreived. The hardest bit of information to find was the BuildBot’s running base URL. Once I found this I tried to use the Python urllib.urlopen method. This didn’t work the way I had hoped. This function blocks the main thread of BuildBot. If I had left this as is, I would have frozen BuildBot at the end of every test run. That is unacceptable.

BuildBot is coded on a framework called Twisted. This is a very network oriented framework and luckily has a built in http client. After some time on the Matrix irc channel I had a working replacement, sans freezing. After I put together the URL and tested the post I was greated by a nice little surprise. my code works I only have XPCShells fully tested but it shouldn’t be difficult to modify the Reftest and Mochitest build steps to work with my system.

On the patch front, I have reworked my XPCShell harness patch after Ted’s requests. There is question of the merit of individual test times. I am neutral on this issue and will let other people decide if this is useful information. I have designed my system to be able to store this information, so it wouldn’t be too difficult to add at a later time if need be.

I am really excited that I am going to California to do an internship with Mozilla. I am going to continue to work on this system there and I hope that it is used, once the kinks are worked out.

0.9 is done! For this release I have implemented Data Persistence. To do this I made use of the SQL Alchemy framework. This framework is similar to the Hibernate Java framework from the JBoss project. A couple major stumbling blocks were encountered. The main source of frustration this release occured with the Log -> LogLine relation. I thought I had coded everything correctly, but it turns out that I had forgotten to derive my LogLine class from the special SQL Alchmey base class. Once I did this things worked much better.

I have also done some investigation into my SQLite issue and it turns out that it is not possible for SQLite to work properly. The CGI environment is limited to one writable file at a time. SQLite requires that the db file be writable as well as four temporary files. Dave told me about a pragma which can instruct SQLite to use memory for temporary files but it turns out that only 1 of the 4 files obey this request. In order for the SQLite library to use purely memory for temporary files you would need to recompile the SQLite library. This means that you would likely need to recompile PySqlite (the Python -> SQLite3 binding).

I also found that I needed to write another class. The class structure I have was not using 1:1 mappings between domain objects and code objects. Each log has tests. Each failing test has its own log. Right now I am storing this ’sub-log’ as an array of strings. In order to simplify the logic in my application and the persistence I am going to seperate this into its own class. I can use the __init__ and __repr__ methods in place of special names that I am currently using.

I would have like to get more done in this release but between some really annoying bugs and my white paper I just didn’t have the time. This comming release cycle should have a lot in it because I have nothing other than an iPhone app to write

For my 1.0 release I would like to have the insertion and viewing logic completed and I would like to convert my code to CherryPy from CGI. CherryPy is an HTML controller which works on WSGI. I am hoping that this framework will allow SQLite to work as I am finding Postgres to be a little slow for what I am doing.

Because of the impending doom that is the BTR820 white paper I didn’t have any time to finish up my 0.9 release this weekend. I plan to get this done for Monday/early Tuesday in time for my Demo!

This release represents a large amount of work. I am having trouble remembering what I have done, so I will mention what I can remember. I started this release cycle by testing out a whole bunch of different ways to display the data in markup and different ways I could deal with in JavaScript. On the markup side I tried to write a div+css table to replace the default HTML tables. This has been done by others with success, but I couldn’t quite get it to size horizontally correctly. I have decided that for now, this is a secondary concern to me.

Another thing I worked on was doing table filtering. I found that my javascript code to do filtering on test status worked really well. The only problem is that it also worked quite slowly. It actually took all browsers I tried a non-trivial amount of time to render the changes. All of my approaches involved either changing a CSS class to alter its display:none property or to change the table rows to have a new css class show and hide. My first attack was raw DOM queries followed by performing regular expressions on the className property for each row. This was a very naive implementation which was incredibly slow on the 2800 test sample I was using. After some chat on IRC Dave suggested that I check out jQuery. I did. I was amazed. How have I not found this insanely cool library before? I was able to very quickly replace my horrid code with about 3-4 lines which worked much faster. Even with jQuery, the show/hide functionality took a long time to render.

The next suggestion I was given (by ted if i remember correctly) was to use CSSOM. This method involves modifying the actual CSS classes programatically. I found that this had the unwanted side effect of hiding my summary table’s counts. This was to be expected because they are using the same CSS class. This method was about as slow as the jQuery method so I decided to go back to jQuery. My hypothesis is that the delay I was experiencing was just the time it took Gecko to do the actual rendering of the table and either method will yield the same results. The more I work on my markup, the more I realise that I will make extensive use of the jQuery framework. One area I know that I am going to do this, specifically to cut down overhead is the popup image links I have in the log display. Currently, each image has an onclick=imgClick(this). This is at least 23byte overhead for each image. The sample log i was working on had 6 images. While this is not going to break anything, efficiency is efficiency. On a side note, i did a test with 48000 line file and the filters, regardless of type froze Firefox!

It was at this point that I decided that I needed to start thinking about how I was going to integrate this into the Mozilla infrastructure. When there is a check in, Mozilla has a buildbot cluster at work building the changed tree and running the unittest suite. Right now, there is no easy way to look at the output of the unittesting. What my project is doing is unifying the output logs across all machines to a single system which maintains history of each machine and each test run. In order to hook into the buildbot I needed to write a whole lot of code.

The first step was to figure out a simple way to send a log to my system. Since buildbot is written in python, I wrote a script which posts a request to a new page I created called insertlog.py. This page takes a whole bunch of log parameters including the URL that the log can be found. The reason these parameters are not included in the log itself is that it would require a modification to buildbot to put this information into the build logs. It is really easy to find out this information, and adding it to the post is one append() away. A snippet of code that will do a post is:

    data = urllib.urlencode(params)    file = urllib.urlopen('http://localhost/logs/insertlog.py', data)

Assuming that params is a dictionary, this will post to the URL with all the key value pairings.

Once the insertlog.py page has been posted to it will download the url, parse it to become a Log object and send it to the database. This is an area where my system needs major improvement. I need to talk about the design with someone. Currently I just have a function that knows how to take a Log object and insert it into the database. This solution works, but is very inflexible and I feel that I will be embarrassed by it. I hate writing code that I know will break, and I know 1000000% that this code will break on the first change I make to it. I am thinking that this might be enough to get by, but I’d really like to discuss it with someone else before going full steam ahead with it.

I have also doodled (image coming soon) a diagram of the information flow of Logfribulator. As it stands right now, there is an insert and view page. I am thinking that a basic administration page would be of use to deal with common situations like “purge all tests before xxx date” or “delete tests from machine xyz”. This is something to look at later. The view page is still up in the air. I am currently thinking that for DPS911 I am going to aim to have a list of the logs in the database as well as the ability to view all the records in the database when clicked in the log list.

I received a conditional r+ on my XPCShell timestamps patch from Ted. I have a couple issues that need to be addressed, but it was otherwise acceptable. Thanks for your reviewing time Ted My autoconf option patch got r-. I have to figure out whether this option should be kept. For this, I will need to speak to the Reftest and Mochitest folks to find out if it is ok to just have the timestamps always on. If it is decided that there should be an option, I will need to figure out whether the default should be on or off and rework the autoconf option patch. I am starting to wonder if there is any value in writing a little document specifying a standard way that Mozilla unit tests could present testing output.

This brings me to the final portion of my 0.8 release. I decided that I was getting sick of working with my code. I felt that I had learned so much in the last little while that I had made lots of mistakes. I rewrote 90% of my parser. The only things that wasn’t touched was the parser core which worked, and works well for my purposes. I also rewrote my unit tests from scratch. My older ones were a little shoddy and didn’t really have good test coverage. For the absolutely critical parts, like the functions which test if a line has passed, I have covered every permutation. Thankfully, they all work. I am still debating the value of the cgi script to upload a log file and show the parsed output. I don’t think this is as valuable as a python script that the individual developer can run against their log and get the output in a format of their choosing. I am going to do some optparse magic on my main parser, as it works as a standalone parser.

I am really enjoying working on this project. I have learned so much this semester!