Mac App Sandboxing: it may not be for you (but that’s probably OK)

The MAS section of devforums is, along with a healthy subsection of the rest of the interwebs, aflame with the news that the deadline for sandboxing store-delivered apps is further away than it used to be, but still too close for some people.

What’s the deal?

Bugs have been getting easier to detect over time, as the tools used to create software have got better. Anyone who remembers using a microcomputer BASIC interpreter will be familiar with the syntax error, where you’re allowed to enter nonsensical commands and the program runs anyway, stopping unceremoniously when it reaches the unparseable input. Today, syntax errors can be detected and even automatically corrected in the IDE’s text editor.

A program made of 100% acceptable syntax can still have logic errors, which most people would know as “bugs”. The app is supposed to do one thing, but instead does another. Software testing practices are designed to catch logic errors, and code analysis can detect some of these problems too.

With a little bit of hand-waving, we can describe the next level of complexity as the security error or vulnerability. At this level, an application that both compiles and does what the user expects can be made to do something unexpected under misuse. In other words, logic errors are “will it work”, security errors are “can it be made not to work”. Automatic detection and correction of security errors is, in many situations, less well-advanced than detection of other types of error. Techniques for designing and coding security errors out during software development are still very immature.

A little more hand-waving and we get to the final, and most insidious, level of error I will consider here: the blended threat. This can be defined as “can this application be used in conjunction with other applications to do something unexpected”. An example. Safari for Windows suffered a vulnerability called a “carpet bombing” attack, where the author of a web page could cause the browser to download a file without any user intervention. Not that big of a deal: the browser is supposed to allow downloads, this is a little unexpected but not outside the security model of an application.

However, Safari also has that “automatically open safe files after downloading” feature, and this is where the fun begins. Having downloaded, say, a document file, the browser automatically opens it. Now, what if that document is a PDF, and exploits a vulnerability in Acrobat Reader to execute code on the user’s behalf? A local code execution problem like this might be considered a medium-severity issue by the Reader developers: it requires a user to be coerced or tricked into opening a malicious file, doesn’t it?

Well, no. The combination of the low-severity automatic download bug, the automatic open feature and the medium-severity local code execution bug produce a high-severity remote code execution bug.

Detecting and reacting to this kind of “blended threat” is harder than dealing with any of the aforementioned classes of failure. It relies on knowing the interaction between your own app and the vast number of other apps that exist on the platform, understanding the capabilities and bugs of those other apps and how those interact with the capabilities and bugs in your own. It could, in principle, be done, but currently cannot.

Enter sandboxing.

Fundamentally, the problem is that the expression of different roles in an operating system like Mac OS X is incomplete. I’ll not go into this again: my post on the new lion security things covers this ground. The TL;DR is that Mac OS X, Windows NT and friends all assume that the actors in a computer system are the users, and ignore the fact that the code represents a collection of actors too.

OK, so a user is allowed to download a document, read it, and to delete all of her documents. But does that mean that a collection of apps—each one notionally a separate actor independent from the user—should conspire together to enable this? Probably not; or if so, only over a well-defined inter-process communication system that doesn’t allow this sort of thing to happen implicitly. Creating an Automator action to get some content from a PDF on a web server and delete some documents as a result might be something the OS needs to support, but one click in a browser probably shouldn’t make all of that stuff happen automatically.

App sandboxing introduces two important features: the identification of different apps as different actors with their own privileges on the system is one, and (partially voluntary, on the developers’ parts) restrictions on the ways apps communicate is the other. This restriction is implemented both as a limitation on the IPC mechanisms an app can make use of, and on the files the app may access: the filesystem is an app’s version of sneaker net.

With these restrictions in place, the opportunities for performing a blended-threat attack are severely reduced. If one app is compromised it either doesn’t have permission to contact the apps that the threat relies on, or it tries to contact apps that don’t have permission to talk to it.

My app doesn’t fit the sandbox limitations.

Complaints of that nature on iOS, where the limitations have been in place both for ages and from the beginning of the platform’s support for third-party code, have pretty much died out now. Developers have become used to the idea that if an app cannot fit the limitations of the sandboxed environment, then it cannot be distributed on the platform.

There are two issues that limit the applicability of the “suck it up” solution for Mac apps. The first, and weakest, is that there are existing apps which don’t fit the mould. This is weak because on the Mac there are other distribution mechanisms for getting the app into the customers’ hands; mechanisms that have existed and worked for years, and that don’t require the same controls enforced by the Mac App Store policy. Yes, it would be better if every app were sandboxed, but having any app sandboxed reduces the attack surface of the Mac so a less than 100% hit rate is still a win.

Of course, some apps are only incompatible due to minor technicalities, such as legacy decisions over where to locate user files. In these cases it’s usually somewhat harmless to migrate to a situation where the app is sandbox-compatible, for example using the mediated file read permission to load documents from the legacy storage area.

Other apps are basically never going to work in a sandbox: utility software is a common casualty, especially things like anti-virus apps, disk partition editors, and file managers. IDEs are also likely to fall down here (though one that makes good use of the workspace concept need not). We can conclude that Apple doesn’t want to sell ISV software in those categories on the Mac app store, so using a different store is the solution.

The other, stronger limitation is that there are applications that are already on the store that work currently, because the store doesn’t enforce sandbox restrictions, but won’t work in the future, because they’re incompatible with the sandbox restrictions. Developers of these apps could migrate to a different store but would be unable to bring their existing customers with them, to give those existing customers updated versions, or even to contact those customers to tell them about the change.

Yes, this is a serious problem for the minority of affected apps. Yes, it is a limitation of the App Store’s capability that results from Apple’s business decisions. Yes, it’s going to be a ball ache for affected developers to deal with. No, it’s not a reason to throw the baby out with the bathwater and give up on sandboxing completely. No, I haven’t provided a solution yet. As it’s a business problem your first port of call should be your business contacts at Apple: this is such a big issue that it deserves a separate post. The short version is that if you’re doing business with Apple (or any other company) you need a business relationship with that company.

OK, so technical solutions. It all depends on the design and architecture of your app. Perhaps there are features that can be removed, without destroying the essence of your app while gaining compatibility with the app store. Consider the business impact of removing that one feature versus removing your whole app. Also, remember that unless you’ve sold about 40 million copies, there are more potential customers out there than current customers.

Another possibility is that you separate the product into two components, a service and a controlling app. The service part is incompatible with the app store and distributed separately. The app is available as an additional feature to support easier use of the service. Take a look at the number of apps for controlling MySQL, or Postgres, and you’ll see that having an app store app as a front-end to an external service is something that’s both supported and viable (well, if it isn’t viable, a lot of companies enjoy not making money).