This means you no longer have to download Gaia sources and build it yourself to test a site on B2G or check out our progress. Building Gaia is easy if you already have a development environment, but setting up a development environment can be pretty painful, especially on Windows. As of last week, our nightly builds now include a fully built Gaia profile and a small wrapper program to allow launching the Firefox OS simulator with one click. This is especially useful on Mac, where launching the B2G directly in a terminal instead of through Finder causes input to be redirected to the terminal.

These builds end up here: http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-central/

On Windows and Linux, downloading the appropriate package, extracting it then running “b2g/b2g” will launch the simulator. On Mac, you’ll need to mount the DMG then copy the B2G.app bundle to a writable location. The App bundle needs to be on a writable location so the profile can be modified by Gecko.

Because we use a wrapper program to launch B2G, we’ve moved the actual ‘b2g’ and ‘b2g.exe’ files to ‘b2g-bin’ and ‘b2g-bin.exe’, respectively. If you need to launch b2g directly, please use the new b2g-bin name. To see where the output of the B2G process goes on Mac, launch the ‘Console’ application

Note: there is a crash in libxul.dll on Windows that is affecting startup.

Integrating with Buildbot automation

Mock integrates with Buildbot through the MockInit, MockInstall, MockCommand and MockProperty classes. The first two commands are used to configure and create the environment while the third and fourth are used to execute commands inside of a build environment.

MockInit is a basic wrapper for calling mock --init -r target_name. It is provided to encapsulate the logic for this relatively simple operation. Internally, initializing an environment means making sure that the environment is completely clean and has been recreated based on the contents of the root cache and the base packages specified in the mock config file.

MockInstall is also another basic wrapper. MockInstall internally calls mock -r target_name --install package1 package2 packageN. This step will install into the specified target environment the packages needed and any the packages requested depend on. This requires all build tools and dependencies be packaged and added to a Yum repository. Doing this is highly desirable regardless of whether or not Mock is being used.

MockCommand is derived from the standard ShellCommand as intended to operate like the base class. When in non-Mock mode (i.e. use_mock is set to False), this command does nothing to the configuration options and passes all calls straight to the base ShellCommand. When in Mock mode, the command runs its magic method to modify the command line to properly pass the requested buildbot environment through to the Mock call as well as passing the correctly modified command line given the working directory requested (using build environment paths) and the target that should be used.

MockProperty is to SetProperty what MockCommand is to ShellCommand with the important distinction that MockProperty inherits singly from MockCommand and duplicates logic from SetProperty.

In the current implementation, Mock has /builds/ bind mounted into the build environment. This means that all paths under /builds/ have the exact same paths in the build environment and on the build host. If the build process is substantially reworked to use a scripting framework other than Buildbot, it might make sense to stop making the /builds/ directory available inside of the build root. All commands which are “build system” commands are run under a MockCommand with use_mock set to true. Currently, there are no uploads or X11 dependent jobs run under mock. I have a blog post that details X11 access and uploads could be fixed by bind-mounting the host’s ~/.ssh directory into the build environment.

RPM and Yum infrastructure

Because all dependencies are fetched and installed through Yum+RPM on each build, it is necessary to package all tools as RPMs. How RPMs are built and Yum repositories are created is outside the scope of this document. The current setup is for Mozilla to internally mirror the host OS and the build environment OS as well as a custom repo for each distribution. Ideally, the mirrored copy of the repositories would be completely unaltered. I haven’t experimented with overriding core packages yet, but we should be fine to put newer packages that override OS packages in the custom Mozilla repository. As for building RPMs, I’ve been using spec files in my repository to build the rpms.

This repository has a directory for each distributions for which custom packages are generated, currently only ‘centos6′. Inside the distribution directory are directories for each package. There are no directories for each architecture, since rpm has facilities for specifying which architectures are valid for a given package. Each package directory contains small patches and source files needed to build the package, a tooltool manifest for larger files and a symlink to a file called “actions.sh”. This is a symlink to a script in a hidden directory at the root of the repository. In this script, there are commands for the following actions:

• fetch — use tooltool to fetch the sources
• store — add the specified file to the tooltool manifest
• build — perform a simple rpmbuild on the given spec file
• mock_build — perform a mock build of the package using upstream’s mock package

It is the responsibility of the package author to add sources, specs and manifests to the repository and upload the files to the tooltool resource host. The mock_build command can be invoked with the syntax ./actions.sh mock_build fedora-16-x86_64 x86_64 my_package.spec. The SRPMs created should be included in the list of rpm files included in Yum repository creation, since they are the canonical source of what the binary packages were built from.

Creating a new target

In the case that a new build environment configuration is needed, I’d recommend starting with an existing mock_mozilla configuration. The values that you’re likely going to be working with are the root name, arch list, base package list, bind mount and repository list. Deploying the cfg file is required and should be done through something like puppet.

Development

A release of mock_mozilla can be generated by running

git clone git://github.com/jhford/mock_mozilla.git
cd mock_mozilla
./autogen.sh
make srpm # this is to get the tarball, don't care about the srpm created here
mock --buildsrpm --spec mock_mozilla.spec --sources . --resultdir srpm --target x86_64
mock -r epel-6-x86_64 mock_mozilla-1.0.1-1.el6.src.rpm --resultdir x86_64 --target x86_64

The code to mock_mozilla is on github.

Tooltool basics

Tooltool is a client side program written in Python that uses a file manifest in concert with HTTP servers to materialize large binary payloads for use in a job.  The manifests are JSON files which list details of individual files.  Each file is represented in the JSON by a dictionary with the keys “filename”, “digest”, “size” and “algorithm”. An example is located here.  The current JSONEncoder and JSONDecoder derived classes only understand how to work with these keys. Making the JSON encoder and decoder work with an extensible version of the manifests should not be difficult.

Tooltool fetch API

When Tooltool needs to download a file from the file server it does so by creating a URL.  In the current implementation, there is a single base url that is used to construct the URLs that are to be fetched.  A good change to make would be to convert this single URL to a list of possible base urls.  The address of the file to fetch is generated using a string concatenation of the base URL, a slash, the hashing algorightm’s name, another slash and the full hash value represented in hexadecimal. Given a base url of “http://files.r.us:8080/tooltool” and a file that has a SHA512 hash value of 0123456789abcdef, Tooltool will try to fetch  “http://files.r.us:8080/tooltool/sha512/0123456789abcdef”.

There is no server component for Tooltool yet.  As a result, the file server is currently implemented as a simple directory on an HTTP host that has the correct directory structure for responding to requests.  I’ve started working on a server side component but it isn’t finished. The server side component would allow for easy uploads by developers, easy listing of contents on the server and a way to store files in a nicer way on the server’s file system.

Using Tooltool

Tooltool has four commands presently: list, validate, add and fetch.  There are global options and command arguments.  All terminal interactions after the option parser finishes is done through the Python logging API.  The default is to print logging.INFO and higher messages.  Currently, the following global options exist:

• -q/--quiet tells Tooltool to print only logging.ERROR and higher messages
• -v/--verbose specifies to print logging.INFO and higher
• -m/--manifest <file> instructs Tooltool to reference a manifest file located at the specified path
• -d/--algorithm <algorithm> instructs Tooltool to use the specified algorithm
• -o/--overwrite tells Tooltool to overwrite a local file if the filename matches the manifest but the hash value is different to the manifest
• --url specifies the base url to be used for remote operations

Listing and Validating

The two most basic commands list a manifest and validate the local files against the manifest.  The list command lists out all of the files in the manifest as well as whether they are present and/or valid.  The return code from listing is zero unless there was an error in listing the files.  Absent or invalid files will still result in an exit code of zero if there was no error in the listing process.  The validate command is used to check if all the files in the manifest are present and valid.  The exit code for validating is zero if all files in the manifest are present and their hash matches the manifest.  It is non-zero if any file is missing locally or the file does not have the same hash as the manifest.

Listing and validating a Tooltool manifest. Note the exit codes

Fetching

The fetch command is used to materialize files locally.  Before fetching a file from a remote host, Tooltool will validate local files which match the filename specified in the manifest.  The default behaviour when a local file matches the filename but not the hash value is to exit with a non-zero exit code.  If –o or –overwrite is specified on the command line Tooltool will overwrite the local file without confirmation with the remote file.  The local file will be truncated as soon as Tooltool attempts to start writing the remote file locally.

Fetching a file that exists locally with contents differing from manifest. Note the difference in behaviour when using --overwrite

Adding

Files are added to manifests with the add command. This command looks for a manifest using either the default name of manifest.tt or by the value specified by the -m/–manifest command line option. If the manifest does not exist already on the file system, a new one will be created to store the first file given. An error message will be displayed if a file is added a second time, regardless of whether or not the local contents are the same as what is in the manifest. There is currently no logic to remove a file from a manfiest or overwrite a manifest entry.

Future direction and contributions

Tooltool is a generic lookaside cache. My intention is to keep it as general as possible by not including logic to deal with payloads. We currently include a bootstrap script in the b2g manifests that understands how to take the rest of the payload and set up a working toolchain. Using a bootstrap script means that Tooltool can be tool agnostic while still allowing complex operations on the fetched tools. A standardized bootstrap script name and interface that is called by Tooltool might make sense. I’d also like to finish the server-side component that has an interface to upload files with and a method for storing files in a less obtuse method than just mirroring the API paths. A command for removing files from a manifest would be helpful, as would the ability to update a manifest with the contents of the directory as they exist right now. Another really cool feature would be the ability to configure a system wide cache directory where files are downloaded to. Once the server component is working, I’d like to add a way for servers to automatically sync their file stores when they are asked for a file that exists on another server but not locally.

Tooltool is GPLv2 and the source is available on github. The best way to contribute code is to send a pull request on github. Things are more likely to be merged if they pass the whole test suite (make check), have tests, improve Tooltool and don’t make Tooltool overly specific.

If you find any issues with either of these toolchains, please let us know as soon as possible by filing a mozilla.org::Release Engineering bug and CCing me (:jhford)

To echo my message to a couple news groups, these are some things you might like to know:

1. you need http://hg.mozilla.org/mozilla-central/rev/b687cabf75a1 merged in to your branch to get the biggest build time improvement
2. pushing mozilla-aurora, mozilla-beta, mozilla-esr10 or mozilla-1.9.2 code to try will now attempt a build on 10.7 hardware.  Doing this is not suggested because your build may fail to build at all and it definitely will not match what we currently build on mozilla-aurora and older.
3. These builds are able to run on 10.5
4. we are using gcc-4.2 from xcode 4.1 running on OS X 10.7.2 right now and building against the 10.6sdk
5. the machines are macmini5,3 (quad-i7 2.0GHz, 8GB ram, 2x500GB 7200RPM in Raid0, exclusively intel graphics)
6. shark builds don’t work, bug 723301

I don’t know much about epub, but I do know that it is supported on the iphone and has better text layout than US letter sized PDF files on the device. Python docs are generated using a tool called Sphinx and it turns out that they have an epub builder. Important to note is that this epub builder is experimental.

To generate my epub file, I followed the directions in the Python-2.7.2/Doc/README.txt file, using a builder name of epub instead of one of the listed ones. The result was that I ran these commands:

wget http://www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz
tar zxf Python-2.7.2.tgz
cd Python-2.7.2/Doc
virtualenv doc-tools && source doc-tools/bin/activate
pip install sphinx jinja2 pygments
python tools/sphinx-build.py -bepub . build/epub

Doing this resulted in a large number of html files and an epub file, build/epub/Python.ebub. I opened the file with itunes open -a iTunes build/epub/Python.epub and synced my iphone.

So far my experience is that ibooks is very slow when working with this file. The file is twice as large as the complete works of Shakespeare and has a lot more markup! Most of the things I’d expect to be links are links and the indexes seem to be working just fine. Ibooks (the iphone epub reader) even shows the correct table of contents. Code samples are legible, but are a nested panning area, which is awkward when the same gesture flips to the next page.

If you are feeling lazy, here is a copy.

These builds also represent a huge step forward in making our infrastructure more responsive to developer requests.  We are decoupling our build host OS from our build environment using a fork of Fedora’s Mock tool.  We are checking in a tool manifest that empowers community contributed toolchains to be deployed with tooltool.  We are also using a new set of puppet manifests and a bunch of brand new HP build machines that are pretty fast.  These machines are set up using kickstart instead of a reference image, which means that when we want to rev our hardware, we aren’t tied to a particular model of hardware or even OS version.

These changes represent the direction we are moving in terms of our linux build environment.  A longer term goal is to move our Firefox desktop and mobile builds to mock build slaves for the same reasons we are doing b2g with mock.

Because of the shear number of brand new systems in play here there might be a few teething pains.  I’ve been running the full pool of 42 build machines on my staging master for 2 weeks with the code that landed today and for the last 3 weeks while in development and nothing has inexplicably failed.  Please don’t hesitate to file a bug if you see a problem with this new setup but make sure you CC me on it.

I’ll be writing up a more detailed blog post about what I did and the new tools written as time permits.  I am now focusing my attention towards getting our new quad-core Mac Mini machines into production, which will cut our nightly and release mac build times down from 4+ hours to 1:20.

I tried specifying the hostname in my DISPLAY environment variable but got the following error:

bash-4.2# DISPLAY=localhost:0 xeyes
Error: Can't open display: localhost:0

I did a bit of searching and found that a lot of people suggested that I run xhost +. I found that had no effect on what I was trying. Even more searching and I found out that Fedora and Centos have been invoking X11 with the --nolisten tcp option for some time. This is a great default, but not for what I want to do.

In my search, I found that there were many, many ways with which to specify how X11 is launched. All of these are part of the GDM configuration. GDM is the login system in Gnome. Over the years, there have been many substantial rewrites of GDM, each one seemingly completely changing where the options for the X server reside.

In the end, I found that the relevant file on CentOS 6.2 was /etc/gdm/gdm.schemas. The relevant passage is

<schema>
      <key>security/DisallowTCP</key>
      <signature>b</signature>
      <default>true</default>
</schema>

I switched the false to true and retried running my command, mock_mozilla -r test-f16-b --shell "/usr/bin/env DISPLAY=localhost:0 glxgears" which resulted in a working GLXGears!

The keen observer will notice that this screenshot has a fairly low frame rate. Even though this number is low, the build host gets approximately half this framerate outside of the chroot. It looks like the VMware 3D drivers are slower than pure software. Since glxgears doesn’t actually tell you whether or not you have acceleration, I ran glxinfo loking for the ‘direct rendering’ line.

libGL: OpenDriver: trying /usr/lib/dri/swrast_dri.so
libGL error: dlopen /usr/lib/dri/swrast_dri.so failed (/usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory)
libGL error: unable to load driver: swrast_dri.so
libGL: OpenDriver: trying /usr/lib/dri/swrastg_dri.so
libGL error: dlopen /usr/lib/dri/swrastg_dri.so failed (/usr/lib/dri/swrastg_dri.so: cannot open shared object file: No such file or directory)
libGL error: unable to load driver: swrastg_dri.so
libGL error: reverting to indirect rendering
direct rendering: No (If you want to find out why, try setting LIBGL_DEBUG=verbose)

Looks like the user land library component of the driver isn’t present in the build environment. I hit this problem with a massive hammer and installed all mesa related packages with mock_mozilla -r test-f16-b --install "*mesa*" and now I get direct rendering: Yes and framerates similar to what the build host is showing (~730). I compared the output of glxinfo with a fedora 16 vm on the same VMware host and noticed that I was getting very similar output. I’d want to double check this on real hardware, but it looks like I am getting the same level of 3D acceleration in the build environment as on the build host!

I have been working on a proof of concept for adapting Fedora’s Mock utility to build Firefox, Firefox for Mobile and other projects like boot2gecko in their own sandboxed build environment. Mock is what the Fedora project uses to build every single RPM included in their distribution. Mock separates our build host from our build environment, allows us to scale build hosts any way we desire, allows us to be more efficient and agile, improves security, improves the developer workflow and enables community engagement.

Mock’s Architecture Simplified

When Mock is used, yum and rpm populate the basic sandbox. Each time a command is run, Mock changes the root directory (chroot) to this sandboxed build environment. Mock then drops privileges as requested and runs the command in the sandbox. To speed up the creation of these sandboxes, Mock instructs yum to locally cache the rpm files it downloads. Mock also creates a tarball containing the sandbox after installing the base packages to further speed up sandbox creation. In all, it takes Mock approximately one minute on my local Fusion VM to initialize a build environment once the caches are warmed.

Build Environment Isolation

The host operating system is responsible for providing the kernel, the tools to drive Mock and Mock itself. Because each build environment is built on demand, the libraries that the host operating system provides aren’t important or used. This means that we can take security patches on the build host OS without risking our build environments. We are also able to bring up new fast hardware which may not support our build environment’s OS natively.  Mock uses the chroot system call. This call has been around for quite some time and its abilities and flaws are well known. Chroot works on virtual machines which means we can use Mock even if we decide to scale to virtualized hardware.

Efficiency and Agility

Having no build dependencies installed on the build hosts, other than Mock itself, allows us to use the same configuration on our test and build machines. The test machines could even benefit from having Mock by allowing us to run tests against Gnome 2 and Gnome 3 on the same pool of slaves.

A common scenario that I’ve dealt with having to update a library which we link to Firefox. Lets say this we are using libfoo version 1 in Firefox 10. To support a developer landing a change to Firefox 11 that requires libfoo version 2, we currently need to figure out if Firefox 10 can maintain binary compatibility when built with libfoo version 2 or figure out how to install version 2 in a separate location and have only Firefox 11 and newer use libfoo version 2.  Using Mock, we would create a new build environment that only has libfoo version 2.  Because libfoo version 1 is unavailable to Firefox 11 builds and libfoo version 2 is unavailable to Firefox 10, we don’t risk depending on a new version of libfoo in Firefox 10 and we don’t risk silently falling back to using libfoo version 1 on Firefox 11.

Mobile moves quickly, often needing a completely unique build environment. When we split each mobile project out to its own Mock build environment, we are able to set up new environments significantly quicker because we are able to limit the scope of changes and no longer risk breaking Firefox by changing anything related to mobile. We can also allow mobile projects to use much newer Fedora software in their build environment while still running the more stable CentOS on our build hosts.  This means we enable mobile developers to use modern software while avoiding rebuilding a slew of packages in an alternate location.

Security

I understand that chroots are not a perfect sandboxing tool. I think we need to make sure that we don’t make perfect the enemy of good. We currently have no real sandboxing on our build slaves. Mock will enable us to run all developer submitted commands in an unprivileged chroot that is rebuilt for every build. Mock allows us to keep our system software updated on the build host to lessen our exposure to security vulnerabilities.

A special case in our infrastructure is Try. Each Try build is currently able to modify the system in ways that we might not want it to.  As a result, we currently have a completely separate pool of machines which run our Try builds.  Mock allows us to sandbox all developer submitted code into a disposable sandbox. Because all of the developer submitted code is limited to the sandbox, we can start looking at merging the Try and production pools.  The failure mode here is that the developer is able to exploit a kernel bug to gain root permissions in their sandbox.  Once a user has root in the sandbox they would be able to chroot back out of the sandbox.  This will be important when we discuss merging our Try and production pools.

Correctness

Each build environment will install the exact set of packages required for the build.  When a new dependency is required, we will explicitly add it to the list of dependencies. This means we won’t be surprised when Firefox starts depending on another library or tool.  We are also able to easily move actively developed branches of Firefox forward without taking risky changes to the build environment of shipping versions of Firefox. This is  increasingly important with the ESR (Extended Support Release)/ release.  Using Mock, we can continue to build in the exact same environment we use today for the rest of the ESR release even though we are building a 10 month newer version of Firefox with different dependencies on the same build host.  Having a single pool of slaves that can do all of our Linux based builds without having to figure out alternate install locations for each dependency is a major scalability win in my books.

Developer workflow improvements

Upgrading a library on our build slaves is currently a pain for developers and releng alike. Setting up a local copy of our Linux build machines is a pain. Mock makes this easier by allowing developers to use their existing machines to exactly replicate our official build environments.  Setting up a copy of our build environment on their machine would becomes trivially easy.  Developers could test their code changes in an exact match of our production build environment. Furthermore, with a tool like mozharness, a developer would be able to test an entire build using the exact same bits we use in production with the exact same commands and flow control.

Mock allows us to engage the community more effectively. Community members are able to work on our build machine configuration without impediment and would have access to the exact tools and packages that we use to build Firefox and Mobile.  When a community member finds or fixes a bug in our configuration or wants to upgrade a dependency, they are able to do so and submit patches for review.  This is a lot easier than our current process of signing out a slave for them, adding them to our vpn, letting them do their work followed by removing them from our vpn, reimaging the slave and setting it up again to be in production.  Project like SeaMonkey will be able to match official Firefox build environments with a lot less work required.

Conclusion

Mock is a very useful tool. Using it enables us to handle developer requests quicker, engage the community, improve developer workflow, improve security and reduce risk of breaking shipping versions of Firefox when moving development versions forward.  There are some changes that I’d like to make to the proof of concept before calling mock_mozilla complete, mainly how the driver script (mock_mozilla.py) parses the command line arguments and how build environment locking works.

If you’d like to take a look at the code for yourself, please feel free to check it out on github! https://github.com/jhford/mock_mozilla.

Demonstrations

Before you start running code in a Mock sandbox, you need to intialize it.  Initializing cleans out the old contents and sets up the base packages specified in the sandbox config file.

~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --init
INFO: mock_mozilla.py version 1.1.17 starting...
INFO: State Changed: init plugins
INFO: selinux disabled
INFO: State Changed: start
INFO: State Changed: lock buildroot
INFO: State Changed: clean
INFO: State Changed: unlock buildroot
INFO: State Changed: init
INFO: State Changed: lock buildroot
INFO: Mock Version: 1.1.17
INFO: Mock Version: 1.1.17
INFO: Mock Version: 1.1.17
INFO: calling preinit hooks
INFO: enabled root cache
INFO: State Changed: unpacking root cache
INFO: enabled yum cache
INFO: State Changed: cleaning yum metadata
INFO: enabled ccache
INFO: State Changed: running yum
INFO: State Changed: unlock buildroot
INFO: State Changed: end

Mock uses yum to install packages into the sandbox, but that doesn’t mean that we are limited to using yum and rpm to populate tools.

~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --shell "zip --version"
INFO: mock_mozilla.py version 1.1.17 starting...
INFO: State Changed: init plugins
INFO: selinux disabled
INFO: State Changed: start
INFO: State Changed: lock buildroot
INFO: State Changed: shell
/bin/sh: zip: command not found
INFO: State Changed: unlock buildroot
~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --install zip
INFO: mock_mozilla.py version 1.1.17 starting...
INFO: State Changed: init plugins
INFO: selinux disabled
INFO: State Changed: start
INFO: Mock Version: 1.1.17
INFO: Mock Version: 1.1.17
INFO: Mock Version: 1.1.17
INFO: State Changed: lock buildroot
INFO: installing package(s): zip

================================================================================
 Package        Arch              Version               Repository         Size
================================================================================
Installing:
 zip            x86_64            3.0-3.fc15            fedora            251 k

Transaction Summary
================================================================================
Install       1 Package(s)

Total size: 251 k
Installed size: 770 k

Installed:
  zip.x86_64 0:3.0-3.fc15                                                      

INFO: State Changed: unlock buildroot
INFO: State Changed: end
~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --shell "zip --version"
INFO: mock_mozilla.py version 1.1.17 starting...
INFO: State Changed: init plugins
INFO: selinux disabled
INFO: State Changed: start
INFO: State Changed: lock buildroot
INFO: State Changed: shell
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip,
as of above date; see http://www.info-zip.org/ for other sites.

Compiled with gcc 4.6.0 20110205 (Red Hat 4.6.0-0.6) for Unix (Linux ELF) on Feb  8 2011.

Zip special compilation options:
     USE_EF_UT_TIME       (store Universal Time)
     SYMLINK_SUPPORT      (symbolic links supported)
     LARGE_FILE_SUPPORT   (can read and write large files on file system)
     ZIP64_SUPPORT        (use Zip64 to store large files in archives)
     UNICODE_SUPPORT      (store and read UTF-8 Unicode paths)
     STORE_UNIX_UIDs_GIDs (store UID/GID sizes/values using new extra field)
     UIDGID_NOT_16BIT     (old Unix 16-bit UID/GID extra field not used)
     [encryption, version 2.91 of 05 Jan 2007] (modified for Zip 3)

Encryption notice:
     The encryption code of this program is not copyrighted and is
     put in the public domain.  It was originally written in Europe
     and, to the best of our knowledge, can be freely distributed
     in both source and object forms from any country, including
     the USA under License Exception TSU of the U.S. Export
     Administration Regulations (section 740.13(e)) of 6 June 2002.

Zip environment options:
             ZIP:  [none]
          ZIPOPT:  [none]
INFO: State Changed: unlock buildroot

Mock allows us to run developer submitted code as an unprivileged user to isolate their code and files from the rest of the machine.

~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --shell "whoami"
INFO: mock_mozilla.py version 1.1.17 starting...
INFO: State Changed: init plugins
INFO: selinux disabled
INFO: State Changed: start
INFO: State Changed: lock buildroot
INFO: State Changed: shell
root
INFO: State Changed: unlock buildroot
~/software/mock_mozilla $ mock_mozilla -r mozilla-f15-x86_64 --unpriv --shell "whoami"
INFO: mock_mozilla.py version 1.1.17 starting...
INFO: State Changed: init plugins
INFO: selinux disabled
INFO: State Changed: start
INFO: State Changed: lock buildroot
INFO: State Changed: shell
mock_mozilla
INFO: State Changed: unlock buildroot

Putting all of the above together, I have written a proof of concept script  and recorded a screencast of it building Firefox in a Mock sandbox.  I’ve cut the middle bit out because this takes a while on my Macbook-based VM.

• Nightly build: http://johnford.org/files/2012-02-02-03-02-01-mozilla-central/
• Debug build: http://johnford.org/files/mozilla-central-macosx64lion-debug/

The keen eye will notice that there are no links to shark builds.  As it happens, shark support on 10.7 is something we still need to figure out.  Once I have 10 more machines in our data center to test with, I am going to start producing these builds in a production environment for general consumption.  Soon after, we will want to start switching over to this new builder.  I’ll set these machines to build mozilla-central and project branches first and as we gain confidence, I’ll request approval to move Aurora and possibly Beta to these new machines.