Technical debt and jury service

Posted on 2017-07-11 by Graham

We have the idea that in addition to the product development backlogs for our teams, there’s an engineering backlog where technical debt paydown, process/tooling improvements, and other sitewide engineering concerns get recorded. Working on them is done in time that is, by definition, taken away from the product backlogs (because of Sustainable Pace).

A colleague recently described the time spent on the engineering backlog as a “tax”, which is an interesting analogy. A pejorative interpretation is that, like a tax, centralised engineering work is a cost imposed that takes away from realising more value on my direct projects.

A positive spin is that taxes go toward funding the commons: no one of us can afford to build a road between our house and the office, but having roads connecting all the houses to all the offices has strategic benefit to society as a whole (higher productivity, lower unemployment, more opportunities) so if we all pay in a fraction of the cost of a road we can all have a road. Similarly, one product team might grind to a halt if they spend all of their time on the new CD pipeline infrastructure, but all teams will benefit if they all chip in a bit.

This version of the analogy implies that there might be, like the treasury, a central agency deciding how to spend the common wealth. Somebody needs to decide how much tax everyone should pay, what to do with dissenters (is it OK if your product team focuses on its sprint for a fortnight and doesn’t do any of the engineering backlog?), whether to accept overpayments, and what those tax dollars should go on.

Only it’s not tax dollars, it’s tax hours. In this sense, a better analogy is conscription (I originally thought of the Anglo-Saxon fyrd, maybe jury service or non-military national service is a less aggressive way to consider this). Taxation means that I give all of my work time to Wealth Wizards but give a chunk of my money to the government. Conscription means that I don’t get to give all of my time to my employer: some of it has to go to the commons. Maybe Jonathan and Rebecca can’t give any time to their product teams this week because they’ve been “called up” to the engineering backlog effort.

That seems like a useful analogy for these tasks. I can think about what resources are available for products or “the commons”, because I can think about whether someone is working on “the day job” or has been conscripted. Maybe it doesn’t make sense for everybody to have equal likelihood of being “called up”, in the same way that it’s easier for students to get out of jury service than for full-time employees.

Posted in software-engineering | 1 Comment

The worst phrase in software marketing

Posted on 2017-07-06 by Graham

“Rewritten from the ground up”.

Please. Your old version mostly worked, except for those few corner cases that I’d learned how to work around. Now I don’t know whether the stuff that did work does work now, and I don’t know that I’ll find that stuff in the same place any more.

There’s a reason that old, crufty code in the core of your application was old and crufty. It is old because it works, well enough to pay the programmers who work on maintaining it and building the new things. It is crufty because the problem you’re trying to solve was not perfectly understood, is still not perfectly understood, and is evolving.

Your old, crufty code contains everything you’ve learned about your problem and your customers. And now you’re telling me that you’ve thrown it away.

Posted in Business | 2 Comments

Culture Smell

Posted on 2017-06-09 by Graham

A phrase I used in a discussion today. Developers are familiar with “code smells”, aspects of a codebase that aren’t necessarily wrong but do make you take a deeper look. By analogy, a culture smell surprising, but not necessarily wrong, behaviour on a team that should make you wonder what motivations lead to that behaviour, and what to change to remove or redirect those motivations.

It’s easy to get the wrong idea about culture though. Familiar with the developer concept of pure functions, some organisations seem to operate on the belief that by defining some principles and printing motivational posters, they can ensure conformant behaviour. Or that “the culture” is a concrete device that employees buy into.

Culture is the more slowly-varying norms established by the interactions between group members. You can guide, and lead, and of course admonish or eject, but you can’t control. As a result, more culture signals are “smells” than problems: what you see may not look right, but you have to explore more.

Posted in Uncategorized | Leave a comment

Literate Programming with LibreOffice

Posted on 2017-05-21 by Graham

This post comes in the form of an OpenDocumentFormat document containing a program that can extract programs from ODF documents, including the program contained in this document.

Posted in code-level, software-engineering, tool-support | 1 Comment

Prototypical object-oriented programming

Posted on 2017-05-12 by Graham

Some people think that the notion of classes is intrinsic to object-oriented programming. Bertrand Meyer even wrote a textbook about OOP called A Touch of Class. But back in the 1980s, Alan Borning and others were trying to teach object-oriented programming using the Smalltalk system, ostensibly designed to make simulation in computer programmers accessible to children. What they found was that classes are hard.

You’re not allowed to think about how your thing works before you’ve gone a level of abstraction up and told the computer all about the essence of thing-ness, what it is that’s common to all things and sets them apart from other ideas. And while you’re at it, you could well need to think about the metaclass, the essence of essence-of-thing-ness.

So Borning asked the reasonable question: why not just get rid of classes?. Rather than say what all things are like, let me describe the thing I want to think about.

But what happens when I need a different thing? Two options present themselves: both represent the idea that this thing is like that thing, except for some specific properties. One option is that I just create a clone of the first object. I now have two identical things, I make the changes that distinguish the second from the first, and now I can use my two, distinct things.

The disadvantage of that is that there’s no link between those two objects, so I have nowhere to put any shared behaviour. Imagine that I’m writing the HR software for a Silicon Valley startup. Initially there’s just one employee, the founder, and rather than think about the concept of Employee-ness and create the class of all employees, I just represent the founder as an object and get on with writing the application. Now the company hires a second employee, and being a Silicon Valley startup they hire someone who’s almost identical to the founder with just a couple of differences. Rather than duplicating the founder and changing the relevant properties, I create a new object that just contains the specific attributes that make this employee different, and link it to the founder object by saying that the founder is the prototype of the other employee.

Any message received by employee #2, if not understood, is delegated to the original employee, the founder. Later, I add a new feature to the Silicon Valley HR application: an employee can issue a statement apologising if anybody got offended. By putting this feature on the first employee, the other employee(s) also get that behaviour.

This simplified approach to beahvioural inheritance in object-oriented programming has been implemented a few times. It’s worth exploring, if you haven’t already.

Posted in javascript, OOP | 1 Comment

In defence of assertions

Posted on 2017-05-10 by Graham

The year is 2017 and people are still recommending processing out assertions from release builds.

  1. many assertions are short tests (whether or not that’s a good thing): this variable now has a value, this number is now greater than zero), which won’t cost a lot in production. Or at least, let me phrase this another way: many assertions are too cheap to affect performance metrics in many apps. Or, let me phrase that another way: most production software probably doesn’t have good enough performance monitoring to see a result, or constrained enough performance goals to care about the result.

  2. The program counter has absolutely no business executing the instruction that follows a failed assertion, because the programmer wrote the subsequent instructions with the assumption that this would never happen. Yes, your program will terminate, leading to a 500 error/unfortunate stop dialog/guru meditation screen/other thing, but the alternative is to run…something that apparently shouldn’t ever occur. Far better to stop at the point of problem detection, than to try to re-detect it based on a surprising and unsupportive problem report later.

  3. assertions are things that programmers believe to always hold, and it’s sensible to take issue with the words always and believe. There’s an argument that goes:

    1. I have never seen this situation happen in development or staging.
    2. I got this job by reversing a linked list on a whiteboard.
    3. Therefore, this situation cannot happen in production.

    but unfortunately, there’s a flaw between the axioms and the conclusion. For example, I have seen the argument “items are added to this list as they are received, therefore these items are in chronological order” multiple times, and have seen items in another order just as often. Assertions that never fire on programmer input give false assurance.

Posted in code-level | 1 Comment

In defence of large teams

Posted on 2017-05-07 by Graham

Seen on the twitters:

1) Bad reasons why tech startups have incredibly large mobile teams even though from an engineering perspective they don’t need it.
This is the No True Scotsman fallacy, as no true software department needs more than, say, 20 people.

I’m not going to get into the details of what you do with hundreds of mobile engineers. Suffice it to say that the larger-team apps I’ve worked on have been very feature rich, for better or worse. And that’s not just in terms of things you can do, but in terms of how well you can do them. When you in-source the small details that are important to your experience, they become as much work to solve as the overall picture.

Make a list of the companies that you think have “too big” a mobile software development team. Now review that list: all of those companies are pretty big and successful, aren’t they? Maybe big enough to hire a few hundred developers to work on how their customers access their products or services? No true software department needs to be that successful.

And that’s what I think of as the underlying problem with the “your team’s too big, you’re doing it wrong” fallacy: it’s part of the ongoing narrative to devalue all software. It says that your application can’t possibly be worth enough to spend all that developer time on. After all, mine isn’t, and I’m a true software developer.

Posted in Uncategorized | 1 Comment

Your build needs to be better

Posted on 2017-05-05 by Graham

I’ve said it before, build systems are a huge annoyance. If your build is anything other than seemingly instantaneous, it’s costing you severe money.

Your developers are probably off reading HN, or writing blog posts about how slow builds cost them, while the build is going. When they finish doing that, which may be some time after the build completes, they’ll have forgotten some of what they were doing and need to spend some time getting back up to speed.

Your developers are probably suspicious of any build failure, thinking that “the build is flaky” rather than “I made a mistake”. They’ll press the button again and go back to HN. When the same error occurs twice, they might look into it.

Your developers probably know that the build is slow, but not which bit of the build is slow. And they don’t have time to investigate that, where it takes so long to get any work done anyway. So everyone will agree that “there is a problem”, but nothing will get done. Or maybe cargo-cult things will get done, things that speed up “builds” but are not the problem with your build.

The Joel test asks whether you can make a build in one test. Insufficient. If you notice when you’re making a build, you’re slowing your developers down.

…which is not always the worst thing, of course. Sometimes a lengthy translation step from some source language to some optimised form of a machine language program yields better results for your customers, because they get to use a faster program and don’t need to care about the time taken to prepare that program. But let’s be clear: that’s part of the release, and your developers don’t always need to be working from the released product (in fact, they’re usually not). Releases should be asynchronous, and the latency between having something ready to be released and having released it can be fairly high, compared with the latency between having created some source and being able to investigate its utility.

Nonetheless, that should all go off in the background. So really, builds and releases should both be non-events to the developers.

Posted in architecture of sorts, code-level, tool-support | Leave a comment

I just want to point out that even the best of us aren’t doing what we expect the makers of acne creams to do.

What we actually know about software development, and why we believe it’s true by Greg Wilson.

Posted on 2017-04-25 by Graham | Leave a comment

Why your app is not massively parallel software

Posted on 2017-04-20 by Graham

That trash can Mac Pro that hasn’t been updated in years? It’s too hard to write software for.

Now, let’s be clear, there are any number of abstractions that have been created to help programmers parallelise their thing, from the process onward. If you’ve got a loop and can add the words #pragma omp parallel for to your code, then your loop can be run in parallel over as many threads as you like. It’s not hard.

Making sure that the loop body can run concurrently with itself is hard, but there are some rules to follow that either make it easy or tell you when to avoid trying. But you’re still only using the CPU, and there’s that whole dedicated GPU to look after as well.

Even with interfaces like OpenCL, it’s difficult to get this business right. If you’ve been thinking about your problem as objects, then each object has its own little part of the data – but now you need to get that information into a layout that’ll be efficient for doing the GPU work, then actually do the copy, then copy the results back from the GPU memory…is doing all of that worth it?

For almost all applications, the answer is no. For almost no applications, the answer is occasionally. For a tiny number of applications, the answer is most of the time, but if you’re writing one of those then you’re a scientist or a data “scientist” and probably not going to get much value out of a deskside workstation anyway.

What’s needed for that middle tier of applications is the tools – by which I mostly mean the libraries – to deal with this problem when it makes sense. You don’t need visualisations that say “hey, if you learned a different programming language and technique and then applied it to this little inner loop you could get a little speed boost for the couple of seconds that one percent of users will use this feature every week” – you need implementations that notice that and get on with it anyway.

The Mac Pro is, in that sense, the exact opposite of the Macintosh. Back in the 1980s, the Smalltalk software was ready well before there was any hardware that could run it well, and the Macintosh was a thing that took this environment that could be seen to have value, and made it kindof work on real hardware. Conversely, the Mac Pro was ready well before there was any software that could make use of it, and that’s a harder sell. The fact that, four years later, this is still true, makes it evident that it’s either difficult or not worth the effort to try to push the kind of tools and techniques necessary to efficiently use Mac Pro-style hardware into “the developer ecosystem”. Yes, there are niches that make very good use of them, but everybody else doesn’t and probably can’t.

Posted in whatevs | Leave a comment