Monthly Archives: April 2009

For anyone interested, I have posted my code through a BitBucked repository here:http://bitbucket.org/jhford/logfribulator/

I plan to make a video to demonstrate my system tomorrow as it requires quite a bit of setup to be fully functional

I have finally finished my DPS911 course work. For this release I have written a log view page and hooked Logfribulator into the BuildBot. After fixing a bunch of bugs with data persistence I wrote a viewing page. Because of the data persistence framework I am using (SQLAlchemy), this was an enjoyable experience. This framework makes querying very logical, almost functioning like jQuery. I haven’t enabled the filtering that was on the old page because I am going to be re-implementing the HTML generation using Pylon’s Mako templating engine. Originally I had planned to use CherryPy for the controller of my page.

After a chat on irc.freenode.net/#python I have decided to look at Pylons instead. It is a popular framework and already has SQLAlchemy built in. The only reason I don’t like this framework is because I don’t understand how it works yet. Unlike CherryPy, I haven’t yet figured out how to run the development server from within Eclipse. This is something which would be very nice as I love the Eclipse debugger interface.

To help with debugging my system and aid in hacking on BuildBot, I wrote a simple ‘dump’ script in Python. I used SQLAlchemy to create a simple table which I could use anywhere. This was a great solution for my next bit of hacking. I finally hooked up BuildBot to submit logs to Logfribulator as they are completed. To do this I spent quite a few hours figuring out how each class in BuildBot is connected. In the end I was successful. I found that between the exception handler build into BuildBot and my database dump script this wasn’t as painful as I thought it would be.

The main thing I learned from this exercise is that code searching is invaluable. I used the built in Eclipse project search function. An example was where the string form of an object had the information I wanted. It would print in the format <Build ‘full-osx’>. When I did a search for “<Build “ I was directed to code which shows how this information was retreived. The hardest bit of information to find was the BuildBot’s running base URL. Once I found this I tried to use the Python urllib.urlopen method. This didn’t work the way I had hoped. This function blocks the main thread of BuildBot. If I had left this as is, I would have frozen BuildBot at the end of every test run. That is unacceptable.

BuildBot is coded on a framework called Twisted. This is a very network oriented framework and luckily has a built in http client. After some time on the Matrix irc channel I had a working replacement, sans freezing. After I put together the URL and tested the post I was greated by a nice little surprise. my code works I only have XPCShells fully tested but it shouldn’t be difficult to modify the Reftest and Mochitest build steps to work with my system.

On the patch front, I have reworked my XPCShell harness patch after Ted’s requests. There is question of the merit of individual test times. I am neutral on this issue and will let other people decide if this is useful information. I have designed my system to be able to store this information, so it wouldn’t be too difficult to add at a later time if need be.

I am really excited that I am going to California to do an internship with Mozilla. I am going to continue to work on this system there and I hope that it is used, once the kinks are worked out.

0.9 is done! For this release I have implemented Data Persistence. To do this I made use of the SQL Alchemy framework. This framework is similar to the Hibernate Java framework from the JBoss project. A couple major stumbling blocks were encountered. The main source of frustration this release occured with the Log -> LogLine relation. I thought I had coded everything correctly, but it turns out that I had forgotten to derive my LogLine class from the special SQL Alchmey base class. Once I did this things worked much better.

I have also done some investigation into my SQLite issue and it turns out that it is not possible for SQLite to work properly. The CGI environment is limited to one writable file at a time. SQLite requires that the db file be writable as well as four temporary files. Dave told me about a pragma which can instruct SQLite to use memory for temporary files but it turns out that only 1 of the 4 files obey this request. In order for the SQLite library to use purely memory for temporary files you would need to recompile the SQLite library. This means that you would likely need to recompile PySqlite (the Python -> SQLite3 binding).

I also found that I needed to write another class. The class structure I have was not using 1:1 mappings between domain objects and code objects. Each log has tests. Each failing test has its own log. Right now I am storing this ‘sub-log’ as an array of strings. In order to simplify the logic in my application and the persistence I am going to seperate this into its own class. I can use the __init__ and __repr__ methods in place of special names that I am currently using.

I would have like to get more done in this release but between some really annoying bugs and my white paper I just didn’t have the time. This comming release cycle should have a lot in it because I have nothing other than an iPhone app to write

For my 1.0 release I would like to have the insertion and viewing logic completed and I would like to convert my code to CherryPy from CGI. CherryPy is an HTML controller which works on WSGI. I am hoping that this framework will allow SQLite to work as I am finding Postgres to be a little slow for what I am doing.

Because of the impending doom that is the BTR820 white paper I didn’t have any time to finish up my 0.9 release this weekend. I plan to get this done for Monday/early Tuesday in time for my Demo!