I plan to make a video to demonstrate my system tomorrow as it requires quite a bit of setup to be fully functional

After a chat on irc.freenode.net/#python I have decided to look at Pylons instead. It is a popular framework and already has SQLAlchemy built in. The only reason I don’t like this framework is because I don’t understand how it works yet. Unlike CherryPy, I haven’t yet figured out how to run the development server from within Eclipse. This is something which would be very nice as I love the Eclipse debugger interface.

To help with debugging my system and aid in hacking on BuildBot, I wrote a simple ‘dump’ script in Python. I used SQLAlchemy to create a simple table which I could use anywhere. This was a great solution for my next bit of hacking. I finally hooked up BuildBot to submit logs to Logfribulator as they are completed. To do this I spent quite a few hours figuring out how each class in BuildBot is connected. In the end I was successful. I found that between the exception handler build into BuildBot and my database dump script this wasn’t as painful as I thought it would be.

The main thing I learned from this exercise is that code searching is invaluable. I used the built in Eclipse project search function. An example was where the string form of an object had the information I wanted. It would print in the format <Build ‘full-osx’>. When I did a search for “<Build “ I was directed to code which shows how this information was retreived. The hardest bit of information to find was the BuildBot’s running base URL. Once I found this I tried to use the Python urllib.urlopen method. This didn’t work the way I had hoped. This function blocks the main thread of BuildBot. If I had left this as is, I would have frozen BuildBot at the end of every test run. That is unacceptable.

BuildBot is coded on a framework called Twisted. This is a very network oriented framework and luckily has a built in http client. After some time on the Matrix irc channel I had a working replacement, sans freezing. After I put together the URL and tested the post I was greated by a nice little surprise. my code works I only have XPCShells fully tested but it shouldn’t be difficult to modify the Reftest and Mochitest build steps to work with my system.

On the patch front, I have reworked my XPCShell harness patch after Ted’s requests. There is question of the merit of individual test times. I am neutral on this issue and will let other people decide if this is useful information. I have designed my system to be able to store this information, so it wouldn’t be too difficult to add at a later time if need be.

I am really excited that I am going to California to do an internship with Mozilla. I am going to continue to work on this system there and I hope that it is used, once the kinks are worked out.

I have also done some investigation into my SQLite issue and it turns out that it is not possible for SQLite to work properly. The CGI environment is limited to one writable file at a time. SQLite requires that the db file be writable as well as four temporary files. Dave told me about a pragma which can instruct SQLite to use memory for temporary files but it turns out that only 1 of the 4 files obey this request. In order for the SQLite library to use purely memory for temporary files you would need to recompile the SQLite library. This means that you would likely need to recompile PySqlite (the Python -> SQLite3 binding).

I also found that I needed to write another class. The class structure I have was not using 1:1 mappings between domain objects and code objects. Each log has tests. Each failing test has its own log. Right now I am storing this ’sub-log’ as an array of strings. In order to simplify the logic in my application and the persistence I am going to seperate this into its own class. I can use the __init__ and __repr__ methods in place of special names that I am currently using.

I would have like to get more done in this release but between some really annoying bugs and my white paper I just didn’t have the time. This comming release cycle should have a lot in it because I have nothing other than an iPhone app to write

For my 1.0 release I would like to have the insertion and viewing logic completed and I would like to convert my code to CherryPy from CGI. CherryPy is an HTML controller which works on WSGI. I am hoping that this framework will allow SQLite to work as I am finding Postgres to be a little slow for what I am doing.

Another thing I worked on was doing table filtering. I found that my javascript code to do filtering on test status worked really well. The only problem is that it also worked quite slowly. It actually took all browsers I tried a non-trivial amount of time to render the changes. All of my approaches involved either changing a CSS class to alter its display:none property or to change the table rows to have a new css class show and hide. My first attack was raw DOM queries followed by performing regular expressions on the className property for each row. This was a very naive implementation which was incredibly slow on the 2800 test sample I was using. After some chat on IRC Dave suggested that I check out jQuery. I did. I was amazed. How have I not found this insanely cool library before? I was able to very quickly replace my horrid code with about 3-4 lines which worked much faster. Even with jQuery, the show/hide functionality took a long time to render.

The next suggestion I was given (by ted if i remember correctly) was to use CSSOM. This method involves modifying the actual CSS classes programatically. I found that this had the unwanted side effect of hiding my summary table’s counts. This was to be expected because they are using the same CSS class. This method was about as slow as the jQuery method so I decided to go back to jQuery. My hypothesis is that the delay I was experiencing was just the time it took Gecko to do the actual rendering of the table and either method will yield the same results. The more I work on my markup, the more I realise that I will make extensive use of the jQuery framework. One area I know that I am going to do this, specifically to cut down overhead is the popup image links I have in the log display. Currently, each image has an onclick=imgClick(this). This is at least 23byte overhead for each image. The sample log i was working on had 6 images. While this is not going to break anything, efficiency is efficiency. On a side note, i did a test with 48000 line file and the filters, regardless of type froze Firefox!

It was at this point that I decided that I needed to start thinking about how I was going to integrate this into the Mozilla infrastructure. When there is a check in, Mozilla has a buildbot cluster at work building the changed tree and running the unittest suite. Right now, there is no easy way to look at the output of the unittesting. What my project is doing is unifying the output logs across all machines to a single system which maintains history of each machine and each test run. In order to hook into the buildbot I needed to write a whole lot of code.

The first step was to figure out a simple way to send a log to my system. Since buildbot is written in python, I wrote a script which posts a request to a new page I created called insertlog.py. This page takes a whole bunch of log parameters including the URL that the log can be found. The reason these parameters are not included in the log itself is that it would require a modification to buildbot to put this information into the build logs. It is really easy to find out this information, and adding it to the post is one append() away. A snippet of code that will do a post is:

    data = urllib.urlencode(params)    file = urllib.urlopen('http://localhost/logs/insertlog.py', data)

Assuming that params is a dictionary, this will post to the URL with all the key value pairings.

Once the insertlog.py page has been posted to it will download the url, parse it to become a Log object and send it to the database. This is an area where my system needs major improvement. I need to talk about the design with someone. Currently I just have a function that knows how to take a Log object and insert it into the database. This solution works, but is very inflexible and I feel that I will be embarrassed by it. I hate writing code that I know will break, and I know 1000000% that this code will break on the first change I make to it. I am thinking that this might be enough to get by, but I’d really like to discuss it with someone else before going full steam ahead with it.

I have also doodled (image coming soon) a diagram of the information flow of Logfribulator. As it stands right now, there is an insert and view page. I am thinking that a basic administration page would be of use to deal with common situations like “purge all tests before xxx date” or “delete tests from machine xyz”. This is something to look at later. The view page is still up in the air. I am currently thinking that for DPS911 I am going to aim to have a list of the logs in the database as well as the ability to view all the records in the database when clicked in the log list.

I received a conditional r+ on my XPCShell timestamps patch from Ted. I have a couple issues that need to be addressed, but it was otherwise acceptable. Thanks for your reviewing time Ted My autoconf option patch got r-. I have to figure out whether this option should be kept. For this, I will need to speak to the Reftest and Mochitest folks to find out if it is ok to just have the timestamps always on. If it is decided that there should be an option, I will need to figure out whether the default should be on or off and rework the autoconf option patch. I am starting to wonder if there is any value in writing a little document specifying a standard way that Mozilla unit tests could present testing output.

This brings me to the final portion of my 0.8 release. I decided that I was getting sick of working with my code. I felt that I had learned so much in the last little while that I had made lots of mistakes. I rewrote 90% of my parser. The only things that wasn’t touched was the parser core which worked, and works well for my purposes. I also rewrote my unit tests from scratch. My older ones were a little shoddy and didn’t really have good test coverage. For the absolutely critical parts, like the functions which test if a line has passed, I have covered every permutation. Thankfully, they all work. I am still debating the value of the cgi script to upload a log file and show the parsed output. I don’t think this is as valuable as a python script that the individual developer can run against their log and get the output in a format of their choosing. I am going to do some optparse magic on my main parser, as it works as a standalone parser.

I am really enjoying working on this project. I have learned so much this semester!

#!/usr/bin/env pythonimport sqlite3import cgiimport osprint "Content-type: text/html\n"try:  con = sqlite3.connect('/tmp/db.sqlite')  cur = con.cursor()  cur.execute("select * from test;")  for line in cur:      print line  cur.execute("""INSERT INTO test VALUES(1);""")  con.commit()  con.close()except:  cgi.print_exception()

Either way, I have decided to focus my efforts on working with Postgres. I have done this because I have spent hours and hours trying to get this to work with no progress. I just get an “OperationalError: unable to open database” message. I feel that in the long term Postgres will be the better solution. Each time unit tests are run for a build on the build bot cluster there will be approximately 40-60 thousand tests that will need to be inserted.

My next challenge was to get Postgres working on my Macbook. I have been working on my Macbook because it makes my workflow very simple. I just save my python scripts and they are live in my CGI environment. To install Postgres I had a choice of using macports or the semi-official EnterpriseDB installer. I ended up going with the EnterpriseDB version for no particular reason. The next step in converting to Postgres was to install a Postgres driver for python. I am using the system default python installation located at /Library/Python/2.5. This means that when I try to install python modules, I cannot use macports as it uses its own python installation in /opt/local. This has worked really well for me so far, but while trying to install the database driver I kept getting a file not found error. This was a very cryptic and highly annoying error. Turns out that the problem I was having was that the driver, called psycopg2, uses a C extension to interface with Postgres. This C extension requires that the ‘pg_config’ executable be in the PATH variable during build time. I solved this by changing the pg_config attribute in setup.cfg for psycopg2 to pg_config=/Library/PostgreSQL/8.3/bin/pg_config. I am assuming that I could also accomplish this by adding my Postgres installation’s bin directory to my path.

Once I had this, I checked that I had a working module by launching the interactive python interpreter and running import psycopg2. It didn’t blow up and that makes me happy!

Here is a sample of code for Postgres which works in the CGI environment:

#!/usr/bin/env pythonimport psycopg2import cgi#import osprint "Content-type: text/html\n"print "Hello"try:  con = psycopg2.connect("dbname='logfribulator' user='logfribulator' password='mozilla'")  cur = con.cursor()  try:      cur.execute("""INSERT INTO log (build_number, start_time, end_time, builder, factory, slave, machine_name, log_url)VALUES(1,timestamp '2001-09-28 01:00',timestamp '2001-09-28 01:00','builder', 'factory', 'slave', 'machine', 'url4');""")  except psycopg2.IntegrityError:      con.rollback()  else:      con.commit()  cur.execute('select * from log;')  for line in cur:      print line  con.close()except:  cgi.print_exception()

This script will only do the insert if there is no issue. If there is an issue only the select query will be done.

EDIT: Our planet doesn’t seem to like html being in the post’s body

Notice anything? Well, if you are like me you will notice that nothing shows up. No error, the tag isn’t ignored and there is no user notification as to why it isn’t showing up. The reason is that the parser sees this as an opening tag and doesn’t find a closing tag by the end of the document. Because nothing is visible from within <script%gt; blocks, nothing is displayed. I didn’t even know this issue existed!

Reference:
http://hixie.ch/advocacy/xhtml