Packaging software

Posted on 2018-11-27 by Graham

I’ve been learning about Debian Packaging. I’ve built OS X packages, RPMs, Dockerfiles, JARs, and others, but never dpkgs, so I thought I’d give it a go.

My goal is to make a suite of GNUstep packages for Debian. There already are some in the base distribution, and while they’re pretty up to date they are based on a lot of “default” choices. So they use gcc and the GNU Objective-C runtime, which means no blocks and no modern objc features. The packages I’m making are, mostly thanks to overriding some choices in building gnustep-make, built using clang, the next-generation objc runtime, libdispatch etc.

The Debian packaging tools are very powerful, very well documented, and somewhat easy to use. But what really impressed me along this journey was CPack. I’ve used cmake before, on a team building a pretty big C++/Qt project. It’s great. It’s easier to understand than autoconf, easier to build correct build rules over a large tree than make, and can generate fast builds using ninja or IDE-compatible projects for Xcode, IntelliJ and (to some extent) Eclipse.

What cpack adds is the ability to generate packages, of various flavours (Darwin bundles, disk images, RPMs, DEBs, NullSoft installers, and more) from the information about the build targets. That’s really powerful.

Packaging software is a really important part of the customer experience: what is an “App Store” other than a package selection and distribution mechanism? It irks me that packaging systems are frequently either coupled to the target environment (Debian packages and OpenBSD ports are great, but only work in those systems), or baroque (indeed autoconf may have gone full-on rococo). Package builder-builders give distributors a useful respite, using a single tool to build packages that work for any of their customers.

It’s important that a CD pipeline produces the same artefacts that your customers use, and also that it consumes them: you can’t make a binary, test it, see that it works, then bundle it and ship it. You have to make a binary, bundle it, ship it, install it, then test it and see that it works. (Obviously the tests I’m talking about here are the “end-to-end”, or “customer environment” tests. You don’t wait until your thing is built to see whether your micro-tests pass, you wait until your micro-tests pass to decide whether it’s worth building the thing.)

I know that there are other build tools that also include packaging capabilities. The point is, using one makes things easier for you and for your customers. And, it turns out, CMake is quite a good choice for one.

Posted in tool-support | Tagged | Leave a comment

Concurrent objects and SCOOP

Posted on 2018-11-21 by Graham

Representing concurrency in an object-oriented system has been a long-standing problem. Encapsulating the concurrency primitives via objects and methods is easy enough, but doesn’t get us anywhere. We still end up composing our programs out of threads and mutexes and semaphores, which is still hard.

Prior Art

It’s worth skimming the things that I’ve written about here: I’ve put quite a lot of time into this concurrency problem and have written different models as my understanding changed.

Clearly, this is a broad problem. It’s one I’ve spent a lot of time on, and have a few satisfactory answers to if no definitive answer. I’m not the only one. Over at codeotaku they’ve concluded that Fibers are the right solution, though apparently on the basis of performance.

HPC programs are often based on concurrent execution through message passing, though common patterns keep it to a minimum: the batch processor starts all of the processes, each process finds its node number, node 0 divvies up the work to all of the nodes (a message send), then they each run through their part of the work on their own. Eventually they get their answer and send a message back to node 0, and when it has gathered all of the results everything is done. So really, the HPC people solve this problem by avoiding it.

You’re braining it wrong, Graham

Many of these designs try to solve for concurrency in quite a general way. Serial execution is a special case, where you only have one object or you don’t submit commands to the bus. The problem with this design approach, as described by Bertrand Meyer in his webinar on concurrent Object-Oriented Programming, is that serial execution is the only version we really understand. So designing for the general, and hard-to-understand, case means that generally we won’t understand what’s going on.

The reason he says this is so is that we’re better at understanding static relationships between things than the dynamic evolution of a system. As soon as you have mutexes and condition locks and so on, you are forced to understand the dynamic behaviour of the system (is this lock available? Is this condition met?). Worse: you have to understand it holistically (can anything that’s going on at the moment have changed this value?).

Enter SCOOP

Meyer’s proposal is that as serial programs are much easier to understand (solved, one might say, if one has read Dijkstra’s A Discipline of Programming) we should make our model as close to serial programming as possible. Anything that adds concurrency should be unsurprising, and not violate any expectations we had if we tried to understand our program as a sequential process.

He introduced SCOOP (Simple Concurrent Object-Oriented Programming) developed by the Concurrency Made Easy group at ETH Zürich and part of Eiffel. Some of the design decisions he presented:

  • a processor is an abstraction representing sequential execution
  • there is a many-to-one mapping of objects to processors (this means that an object’s execution is always serial, and that all objects are effectively mutexes)
  • where an object messages another on a different processor, commands will be asynchronous (but executed in order) and queries will be synchronous
  • processors are created dynamically and opportunistically (i.e. whenever you create an object in a “separate” and as-yet unpopulated domain)

An implementation of this concurrency model in Objective-C is really easy. A proxy object representing the domain separation intercepts messages, determines whether they are commands or queries and arranges for them to be run on the processor. It inspects the objects returned from methods, introducing proxies to tie them to the relevant processor. In this implementation a “processor” is a serial operation queue, but it could equivalently be a dedicated thread, a thread pulled from a pool, a dedicated CPU, or anything else that can run one thing at a time.

This implementation does not yield all of the stated benefits of SCOOP. Two in particular:

  1. The interaction of SCOOP with the Eiffel type system is such that while a local (to this processor) object can be referred to through a “separate” variable (i.e. one that potentially could be on a different processor), it is an error to try to use a “separate” object directly as if it were local. I do not see a way, in either Swift’s type system or ObjC’s, to maintain that property. It looks like this proposal, were it to cover generic or associated types, would address that deficiency.
  2. SCOOP turns Eiffel’s correctness preconditions into wait conditions. A serial program will fail if it tries to send a message without satisfying preconditions. When the message is sent to a “separate” object, this instead turns into a requirement to wait for the precondition to be true before execution.

Conclusions

Meyer is right: concurrent programming is difficult, because we are bad at considering all of the different combinations of states that a concurrent system can be in. A concurrent design can best be understood if it is constrained to be mostly like a serial one, and not require lots of scary non-local comprehension to understand the program’s behaviour. SCOOP is a really nice tool for realising such designs.

This is something I can help your team with! As you can see, I’ve spent actual years understanding and thinking about software concurrency, and while I’m not arrogant enough to claim I have solved it I can certainly provide a fresh perspective to your team’s architects and developers. Book an office hours appointment and let’s take a (free!) hour to look at your concurrency problems.

Posted in code-level, OOP, performance | Tagged | 1 Comment

A little challenge

Posted on 2018-11-16 by Graham

A little challenge today: create a JS function that turns its arguments into a list of pairs. Actually, the brief was “using Ramda” but I ended up not doing that:

function basePairwise(xs) {
  if (xs.length == 0) return [];
  if (xs.length == 1) return [[xs[0], undefined]];
  return [[xs[0], xs[1]]].concat(basePairwise(xs.slice(2)));
}

function pairwise(...xs) {
  return basePairwise(xs);
}

One of the nice things about JavaScript (I won’t say that often, so note the date) is the introspective nature: the fact that I get to just look at the way a function was called, rather than saying “you must use exactly these types and this arity always”. Here, I’ve done that with the JS spread operator: I could have used the arguments pseudo-list for only a little more work.

Posted in FP, javascript | Leave a comment

Ultimate Programmer Super Stack: Last day!

Posted on 2018-11-13 by Graham

I already wrote about the Ultimate Programmer Super Stack, a huge bundle of books and courses on a range of technologies: Python, JS, Ruby, Java, HTML, node, Aurelia… and APPropriate Behaviour, my book on everything that goes into being a programmer that isn’t programming.

Today is the last day of the bundle. Check it out here, it won’t be available for long.

Posted in advancement of the self, books, edjercashun | Leave a comment

Coming to terms with fewer terms

Posted on 2018-11-12 by Graham

I was on a “Leadership in Architecture” panel organised by RP International recently, and was asked about problems we face using new techniques like Microservices, serverless and machine learning in the financial technology sector. The biggest blocker I see is the RFP (Request for Proposals), RFI (Request for Information), the MSA (Master Service Agreement), any document with a three-letter acronym. We would do better if they disappeared.

I’m fully paid up in my tithe to the church of “customer collaboration over contract negotiation”, and I believe that this needs to extend beyond the company boundary. If we’re going to spend a few months going back and forth over waving our certifications about, deciding who gets to contact whom within what time, and whether the question they asked constitutes a “bug report” or a “feature request”, then I don’t think it matters whether the development team use two-week sprints or not. We’ve already lost.

We’ve lost because we know that the interactions between the people involved are going to be restricted to the terms agreed during that negotiation. No longer are people concerned about whether the thing we’re making is valuable; they’re concerned with making sure their professional indemnity insurance is up to date before sending an email to the DRI (Definitely Responsibility-free Inbox).

We’ve lost because we had a team sitting on its hands during the negotiation, and used that time “productively” by designing the product, putting epics and stories in a backlog, grooming that backlog, making wireframes, and all of those other things that aren’t working software.

We’ve lost because each incompatibility between the expectation and our intention is a chance to put even more contract negotiation in place, instead of getting on with making the working software. When your RFI asks which firewall ports you need to open into your DMZ, and our answer is none because the software runs outside of your network on a cloud platform, we’re not going to get into discussions of continuous delivery and whether we both read the Phoenix Project. We’re going to get into discussions of whether I personally will warrant against Amazon outages. But here’s the thing: we don’t need the software to be 100% up yet, we don’t even know whether it’s useful yet.

Here’s an alternative.

  1. We, collectively, notice that the software we make solves the problem you have.
  2. We, collectively, agree that you can use the software we have now for a couple of weeks.
  3. We, collectively, discuss the things that would make the software better at solving the problem.
  4. We, collectively, get those things done.
  5. We, collectively, GO TO 2.

Notice that you may have to pay for steps 2-4.

Posted in software-engineering | Leave a comment

Microservices for the Desktop

Posted on 2018-11-10 by Graham

In OOP the Easy Way, I make the argument that microservices are a rare instance of OOP done well:

Microservice adopters are able to implement different services in different technologies, to think about changes to a given service only in terms of how they satisfy the message contract, and to independently replace individual services without disrupting the whole system. This […] sounds a lot like OOP.

Microservices are an idea from service-oriented architecture (SOA) in which each application—each microservice—represents a distinct bounded context in the problem domain. If you’re a movie theatre complex, then selling tickets to people is a very different thing from showing movies in theatres, that are coupled loosely at the point that a ticket represents the right to a given seat in a given theatre at a given showing. So you might have a microservice that can tell people what showings there are at what times and where, and another microservice that can sell people tickets.

People who want to write scalable systems like microservices, because they can scale different parts of their application separately. Maybe each franchisee in the theatre chain needs one instance of one service, but another should scale as demand grows, sharing a central resource pool.

Never mind all of that. The real benefit of microservices is that they make boundary-crossing more obvious, maybe even more costly, and as a result developers think about where the boundaries should be. The “problem” with monolithic (single-process) applications was never, really, that the deployment cost too much: one corollary of scale is that you have more customers. It was that there was no real enforcement of separate parts of the problem domain. If you’ve got a thing over here that needs that data over there, it’s easy to just change its visibility modifier and grab it. Now this thing and that thing are coupled, whoops!

When this thing and that thing are in separate services, you’re going to have to expose a new endpoint to get that data out. That’s going to make it part of that thing’s public commitment: a slightly stronger signal that you’re going down a complex path.

It’s possible to take the microservices idea and use it in other contexts than “the backend”. In one Cocoa app I’m working on, I’ve taken the model (the representation in objects of the problem I’m solving) and put it into an XPC Plugin. XPC is a lot like old-style Distributed Objects or even CORBA or DCOM, with the exception that there are more safety checks, and everything is asynchronous. In my case, the model is in Objective-C in the plugin, and the application is in Swift in the host process.

“Everything is asynchronous” is a great reminder that the application and the model are communicating in an arm’s-reach fashion. My model is a program that represents the domain problem, as mentioned before. All it can do is react to events in the problem domain and represent the changes in that domain. My application is a reification of the Cocoa framework to expose a user interface. All it can do is draw stuff to the screen, and react to events in the user interface. The app and the model have to collaborate, because the stuff that gets drawn should be related to the problem, and the UI events should be related to desired changes in the domain. But they are restricted to collaborating over the published interface of the XPC service: a single protocol.

XPC was designed for factoring applications, separating the security contexts of different components and giving the host application the chance to stay alive when parts of the system fail. Those are valid and valuable benefits: the XPC service hosting the model only needs to do computation and allocate memory. Drawing (i.e. messaging the window server) is done elsewhere. So is saving and loading. And that helps enforce the contract, because if I ever find myself wanting to put drawing in the model I’m going to cross a service boundary, and I’m going to need to think long and hard about whether that is correct.

If you want to talk more about microservices, XPC services, and how they’re different or the same, and how I can help your team get the most out of them, you’re in luck! I’ve recently launched the Labrary—the intersection of the library and the laboratory—for exactly that purpose.

Posted in architecture of sorts, code-level, OOP | Tagged | 1 Comment

Ultimate Programmer Super Stack

Posted on 2018-11-06 by Graham

There’s a great bundle of polyglot learning taking place over at the Ultimate Programmer Super Stack. My book, APPropriate Behaviour – the things every programmer needs to know that aren’t programming – is featured alongside content on Python, Ruby, Java, JS, Aurelia, Node, startups, and more.

The bundle is just up for a week, but please do check it out: for not much more than you’d probably pay for APPropriate Behaviour you’ll get a whole heap of stuff that should keep you entertained for a while :).

Posted in advancement of the self, books | Leave a comment

Introducing: the Labrary

Posted on 2018-11-05 by Graham

Is it that a month in the laboratory will save an hour in the library, or the other way around? A little more conversation, a little less action?

There are things to learn from both the library and the laboratory, and that’s why I’m launching the Labrary, providing consulting detective and training service to software teams who need to solve problems, and to great engineers who want to be great lead engineers, principal engineers and architects.

The Labrary is also the home to my books and other projects to come. So if you want to find out what a consulting detective can do for your team, follow the @labrarian on Mastodon or book office hours to talk things over.

Posted in advancement of the self, architecture of sorts | Leave a comment

On Blue Agile

Posted on 2018-11-01 by Graham

Ron Jeffries has some interesting posts lately on Dark Scrum, the idea that poor programmers are being chained to the code face in the software mines, forced to unthinkingly crank out features under Agile-sequel banners like “velocity” and “embracing change”.

In one such post, he refutes the notion of a shadowy Agile Indu$trial Complex. The AIC is a hypothetical network of consultants, tool vendors and project managers who collectively profit from every story point, every Jira “story”, and every grooming session.

Here’s the thing, though. You don’t need fnord filters and Masonic handshakes to explain Dark Scrum. You need to understand that Scrum is doing exactly as intended, and that it’s orthogonal to—not opposite to—the intentions of Agile software development.

Alan Kay had this analogy of two perpendicular planes that he used to explain the difference between Object-Oriented Programming and the thing programmers do in Java. The pink plane contains existing ideas and processes. You can get better at your craft by advancing along the pink plane.

Every so often, an idea comes along (OOP, says Kay, is one such idea) that is not better, it is different. It takes you out of the pink plane altogether into the orthogonal blue plane. You can’t ask “is this idea better” because it doesn’t make sense. It’s different. It has different qualities, so your question about what makes things better no longer makes sense.

Back when we were all allegedly doing waterfall software development, we were delivering software that satisfied requirements defined for some project. We can ask “how good are we at delivering software that satisfies requirements”, and define “better” answers to that question.

Scrum is a process improvement framework for delivering products. As such it provides a “good” baseline for software delivery, and tools to help us get “better” at it. Scrum is the pink plane.

The Agile crowd, on the other hand, stopped asking how much software we are delivering, and started asking how valuable our interactions with our customers are. It’s in the blue plane. The questions we had in the pink plane are no longer relevant, so if we’re still asking them, we will get nonsensical answers.

It is possible, even likely, that the rise of Scrum is due to trying to apply pink plane thinking to the Agile idea, in the way that the rise of C++ or SOLID is due to pink plane thinking about OOP. You could imagine someone who manages a software team seeing a lot of software coming out of an XP team. They read about the XP practices and conclude “I need to make my team adopt these practices, and we’ll make more software”.

But perhaps the XP team weren’t worried about making more software, and don’t even understand why making more software would be a goal.

That doesn’t make Scrum “bad”, indeed looked at along the pink plane it’s better than its predecessors. It just makes it unexpected, and disappointing as a result.

Posted in process | Tagged | Leave a comment

I’m probably holding it wrong

Posted on 2018-10-31 by Graham

If I wanted to do a table view data source in ObjC, it would look like this:

- tableView:aTableView objectValueForTableColumn:aColumn row:(NSInteger)row {
  return [representedObject.collection[row] valueForKey:[aColumn identifier]];
}

When I do it in Swift, it ends up looking like this:

func tableView(_ tableView: NSTableView, objectValueFor tableColumn: NSTableColumn?, row: Int) -> Any? {
    guard let identifier = tableColumn?.identifier else {
        assertionFailure("No table column")
        return nil
    }
    guard let obj = (self.representedObject as? ModelType)?.collection(at:row) else {
        assertionFailure("Can't find model object at \(row)")
        return nil
    }
    switch identifier {
    case NSUserInterfaceItemIdentifier(rawValue:"column1"):
        return obj.field1
    case NSUserInterfaceItemIdentifier(rawValue:"column2"):
        return objc.field2
    //...
    default:
        assertionFailure("Unknown table column \(tableColumn?.identifier ?? NSUserInterfaceItemIdentifier(rawValue: "unknown"))")
        return nil
    }
}

I can’t help feeling I’m doing it wrong.

Posted in AAPL, Swift | Leave a comment