On repeatable builds

One of the key features of software engineering, as distinct from cowboy coding or hacking, is that it should be repeatable. That doesn’t mean that you should do the same project twice in identical ways from beginning to end: that would be a waste of time (and you can bet the requirements have moved the second time around). But it does mean that two people, investigating the same project at the same point in the project, should be able to reproduce the same results.

What does that mean for builds?

If you ask me to investigate a bug a customer reported with version 1.5.3 of your app, then I should be able to build version 1.5.3 of your app even if you’re now working on version 3.0.2. The product I end up with by building version 1.5.3 should be exactly the same as the one you’d end up with, and also exactly the thing the customer was using.

In practice, this means that your build should be as simple as possible. Given access to your source repository, any developer should be able to see what they need to check out, to correspond to a particular version of your app (a released version, a development branch, whatever). They should then hit the one button that gets that product built.

If there are any dependencies in the build process, these should be handled by the build process. If a particular version of your app uses a particular version of a framework or plug-in, that should be codified in the source for that version of your app. Leave nothing to chance, external configuration or intelligent interaction. Which are basically the same thing.

Why should I care?

Well, for a start, if you don’t know that the thing you’re debugging and the thing your customer’s using are the same, then you don’t know whether you can even find the customer’s bug, let alone fix it. There are ancillary benefits too.

Get new developers up to speed quickly

One of the things that I do to differentiate from other security consultants is offer a half-day “I just want you to fix this problem” service, which is a bit like Apple DTS or MSDN Technical Incidents for security (but with a security boffin on tap, and also available on Android ;->).

Now, if you hire me for half a day, you probably want me to spend that half-day solving your problem. I certainly want to spend my time that way. Neither of us wants me to spend a couple of hours trying to work out how your product’s built. Complicated build procedures are the top reason for failed/unreproducible builds. Every time a developer has to think about how to build the product, that’s a point where he can introduce a mistake.

Anyone can turn out a new release

You’ve just found out about a crashing bug in your million-dollar app, but you’re already on that Carribean cruise you’ve bought pre-emptively with the profits. No problem, you use the satellite phone to call a junior developer, and tell him to fix the bug and submit a new build to iTunes Connect.

…can he do it? Will he get it right? If not, can your company last the two weeks until you get back from burning the profits?

You can migrate to new hardware

Maybe you finally got that new MacBook Air you’ve been lusting after, or the Mac Pro to go in your evil lair. Or your dev computer just died, and you need to get a new one up and running. And your backups failed.

If you forget how to set up your dev environment, or you get it a bit wrong, you’ll waste time and get angry. Then you’ll get it more wrong.

You can automate your builds

You’ve seen those open source projects like WebKit that do nightly builds? I do that too. In fact, I have a build triggered whenever I commit source code to version control. If the build succeeds, it runs some tests. If any of that fails, I get an e-mail.

It’s called Continuous Integration, and provides very rapid feedback on the “health” of a project. But to get it done, you need for your builds to be unattended.

OK, I’m convinced. How do I do it?

Let’s take a look at the independent parts of your typical iOS app, and see what it would take to ensure that we always get the same version of them for every build.

Source code

You already use version control, right? Right. Do you tag your releases?

A tag in git or subversion (or any other version control system that supports them, but life’s too short) is just a way to name a particular commit. In git, it really is just a name (and optionally some other info) attached to a certain commit. In subversion, it’s actually a new branch, but one that you shouldn’t commit any changes to.

So let’s say you’ve just finished version 1.0, it doesn’t contain any known bugs, and it’s time to submit it to the store. You should tag the version of the source that corresponds to what you submit as the version 1.0 candidate.

Now, if you ever need to go back to version 1.0, to address an issue reported by the store reviewers, or to investigate a bug reported by a customer, you just check out that tagged version of your source. If you need another developer to look into something related to version 1.0, then she checks out that tagged version.

Of course, checking out a particular tag only gets you the same built product if the source under version control represents a complete definition of the product. Let’s see a few cases where that might not be true.

Subprojects

Apps sometimes contain classes that have been developed as a separate project, particularly collections of helper or utility classes.

Building version 1.0 of the app requires building whatever version of the helpers was used in version 1.0. If you change the two independently, and don’t have a way to keep both in sync, then you don’t have a way to build the same product again. That sucks.

Building a particular version of your app should automatically bring along the correct version of a subproject. The easiest way to do that is with git submodules or its equivalent in your preferred SCM. If that’s not possible for whatever reason, then you can use a shell script as part of the build process. Keep the shell script in version control so you get the correct version of the script that checks out the correct version of the subproject.

Building your app should automatically build the subproject. There is nothing worse than sitting and waiting for a build, only to find random link errors at the end. You send an email to the project maintainer, then after the weekend’s over she sends a reply saying “yeah, you need to build this project first”. Gah!

In Xcode, adding a subproject is as simple as dragging the subproject’s file into the list of files in your main project. You then edit your app target, and add the target for the subproject as a dependency for the app target. Now Xcode checks when you build the app whether it needs to build the subproject first. You’re guaranteed to link against an up-to-date version, not whatever old cruft Xcode found in your search paths.

Third-party frameworks/libraries

Sometimes, your project depends on someone else’s framework or library. You have that library, but you don’t have the source to it.

Check the headers and the binary for the library into version control. Yes, it’s icky. But now, when someone else checks out a particular revision of your source, they automatically get the correct revision of the library too, and compile with the correct headers.

Set up the header/library search paths for your Xcode target such that Xcode searches the folder where the revision-controlled library too, not whatever nasty old version you, I or someone else happens to have knocking around /usr/local/. This path should be relative to $(SRCROOT), i.e. it should say how to get from where your project is to where that library is.

It should not be an absolute path. Other developers don’t have the same path structure as you. They probably have a different username, for a start, so /Users/leeg/Library/Frameworks doesn’t exist for them. They may keep your source in a disk image. If you’re using continuous integration, the source probably gets checked out into a temporary workspace that moves every time. They key thing is don’t rely on the environment being the same for every build, rely on the checked-out source being sufficient to completely describe the build.

Interface Builder plug-ins

More common in the Mac world than iOS, IB plugins provide access to custom objects in Interface Builder and allow you to add them to your XIBs, inspect their properties and so on. If Xcode can’t find an ibplugin needed to compile a XIB, then builds start failing.

Again, check the IB plugin (or the source needed to build it) into version control. If you commit the source, don’t forget to make it a dependency of your app target so it definitely gets built when people try to build your app.

You don’t need an IB Plugins installed to build an Xcode project that relies on it! In your app’s target editor, you can set the paths to the required plug-ins or the search paths for ibtool to find plug-ins when it needs them. Remember to make these paths relative.

A developer trying to edit your XIBs will need to install the plug-in, but now he knows where to find it: he takes the version that was checked out when he checked out your app.

That all sounds hard to set up.

Not really. Create a new user account on your Mac, check out your app, and build it. Fix the problems until there are none. If you can, find a Mac you’ve never coded on before and repeat the same process.

Conclusion

You can avoid wasting a lot of time, and having failed or incorrect builds, by instituting a simple, repeatable build process. There should be exactly one thing to check out of version control, and important versions of that thing should be tagged. Opening the Xcode project that was in version control and hitting ‘build’ should be all that’s needed to get your product built, no matter how complicated the project and how many libraries it depends on. Hitting that ‘build’ button on one version of the source should always build the same product, on anybody’s computer.