Structure and Interpretation of Computer Programmers

My talk at AppDevCon discussed the Requirements Trifecta but turned it into a Quadrinella: you need leadership vision, market feedback, and technical reality to all line up as listed in the trifecta, but I’ve since added a fourth component. You also need to be able to tell the people who might be interested in paying for this thing that you have it and it might be worth paying for. If you don’t have that then, if anybody has heard of you at all, it will be as a company that went out of business with a product “five years ahead of its time”: you were able to build it, it did something people could benefit from, in an innovative way, but nobody realised that they needed it.

A 45 minute presentation was enough to introduce that framework and describe it, but not to go into any detail. For example we say that we need “market feedback”, i.e. to know that the thing we are going to build is something that some customer somewhere will actually want. But how do we find out what they want? The simple answer, “we ask them”, turns out to uncover some surprising complexity and nuance.

At one end, you have the problem of mass-market scale: how do you ask a billion people what they want? It’s very expensive to do, and even more expensive to collate and analyse those billion answers. We can take some simplifying steps that reduce the cost and complexity, in return for finding less out. We can sample the population: instead of asking a billion people what they think, we can ask ten thousand people what they think and apply what we learn to all billion people.

We have to know that the way in which we select those 10,000 people is unbiased, otherwise we’re building for an exclusive portion of the target billion. Send a survey to people’s work email addresses on a Friday, and some will not pick it up until Sunday as their weekend is Fri-Sat. Others will be on holiday, or not checking their email that day, or feeling grumpy and inclined to answer with the opposite of their real thoughts, or getting everything done quickly before the weekend and disinclined to think about your questions at all.

Another technique we use is to simplify the questions—or at least the answers we’ll accept to those questions, to make it easier to combine and aggregate those answers. Now we have not asked “what do you think about this” at all; we have asked “which of these ways in which you might think about this do you agree with?” Because people are inclined to avoid conflict, they tend to agree with us. Ask “to what extent do you agree that spline reticulation is the biggest automation opportunity in widget frobnication?” and you’ll learn something different from the person who asked “to what extent do you agree that spline reticulation is the least important automation opportunity in widget frobnication?”

We’ll get richer information from deeper, qualitative interactions with people, and that tends to mean working with fewer people. At the extreme small end we have one person: an agent talks to their client about what that client would like to see. This is quite an easy case to deal with, because you have exactly one viewpoint to interpret.

Of course, that viewpoint could well be inconsistent. Someone can tell you that they get a lot of independence in how they work, then in describing their tasks list all the approvals and sign-offs they have to get. It can also be incomplete. A manager might not fully know all of the details of the work their reports do; someone may know their own work very well but not the full context of the process in which that work occurs. Additionally, someone may not think to tell you everything about their situation: many activities rely on tacit knowledge that’s hard to teach and hard to explain. So maybe we watch them work, rather than asking them how they work. Now, are they doing what they’re doing because that’s how they work, or because that’s how they behave when they’re being watched?

Their viewpoint could also be less than completely relevant: maybe the client is the person paying for the software, but are they the people who are going to use it? Or going to be impacted by the software’s outputs and decisions? I used the example in the talk of expenses software: very few people when asked “what’s the best software you’ve ever used” come up with the tool they use to submit receipts for expense claims. That’s because it’s written for the accounting department, not for the workers spending their own money.

So, we think to involve more people. Maybe we add people’s managers, or reports, or colleagues, from their own and from other departments. Or their customers, or suppliers. Now, how do we deal with all of these people? If we interview them each individually, then how do we resolve contradiction in the information they tell us? If we bring them together in a workshop or focus group, we potentially allow those contradictions to be explored and resolved by the group. But potentially they cause conflict. Or don’t get brought up at all, because the politics of the situation lead to one person becoming the “spokesperson” for their whole team, or the whole group.

People often think of the productiveness of a software team as the flow from a story being identified as “to do” to working software being released to production. I contend that many of the interesting and important decisions relating to the value and success of the software were made before that limited part of the process.

There is a difference between a generalist software engineer, and a polyglot programmer. What is that difference, and why did I smoosh the two together in yesterday’s post?

A polyglot programmer is a programmer who can use, or maybe has used, multiple programming languages. That doesn’t mean that they have relevant experience or skills in any other part of software engineering. They might, and so might a monoglot programmer. But (in a work context) they’re working as a programmer and paid for doing the programming.

A generalist software engineer has knowledge and experience relevant to the rest of the software engineering endeavour. Not just the things that programmers need to know beyond the programming, but the whole practice.

In my experience, a generalist software engineer is usually also a polyglot programmer, for a couple of reasons.

• to get enough experience at all the bits of software engineering, the generalist’s career has probably survived multiple programming language hype cycles.
• in getting that experience, they have probably had to work with different languages in different contexts: whatever their software products are made out of; scripts for deployment and operations; plugins for various tools; changes to existing components.

More importantly, the programming part of the process is a small (crucial, admittedly, small nonetheless) intersection with the generalist’s work. Knowing all of the gotchas of any one language is something they can ask the specialists about. Fred Brooks would have had a Chief Programmer who knew what the team was trying to do, and a language lawyer who specialised in the tricks of the programming language.

Probably the closest thing that many organisations know how to pay for to the generalist software engineer in modern times is the “agile consultant”. I don’t think it’s strictly a fair comparison. Strictly speaking, an agile consultant is something like a process expert, systems analyst, or cybernetician. They understand how the various parts of the software engineering endeavour affect each other, and can help a team to improve its overall effectiveness by knowing which part needs to change and how.

And, because they use the word agile in their title, they do this with an air of team empowerment and focus on delivery.

Knowing how the threat model impacts on automated testing enables process improvement, but does not necessarily make a software engineering generalist. An actually-general generalist could perhaps also do the threat modelling and create the automated tests. To avoid drawing distinctions between genuine and inauthentic inhabitants of Scotland let’s say that a generalist can work in ‘several’ parts of the software engineering field.

We come back to the problem described in yesterday’s post, but without yesterday’s programming-centric perspective. A manager knows that they need a security person and a test automation person, but does not necessarily know how to deal with a security-and-test-automation-and-maybe-other-things person.

This topic of moving from a narrow focus on programming to a broader understanding of software engineering is the subject of my newsletter: please consider signing up if this topic interests you.

After publishing podcast Episode 53: Specialism versus generality, Alan Francis raised a good point:

This could be very timely as I ponder my life as a generalist who has struggled when asked to fit in a neat box career wise.

https://twitter.com/PossiblyAlan/status/1523755064879632384

I had a note about hiring in my outline for the episode, and for some reason didn’t record a segment on it. As it came up, I’ll revisit that point.

It’s much easier to get a job as a specialist software engineer than as a generalist. I don’t think that’s because more people need specialists than need generalists, though I do think that people need more specialists than they need generalists.

For a start, it’s a lot easier to construct an interview for a specialist. Have you got any experience with this specialism? What have you done with it? Do you know the answers to these in-group trick questions?

Generalists won’t do well at that kind of question. Why bother remembering the answer to a trick question about some specific technology, when you know how to research trick answers about many technologies? But the interviewer hears “doesn’t even know the first trick answer” and wonders how do I know you can deliver a single pint of software on our stack if you can’t answer a question set by a junior our-stack-ist?

If you want to hire a generalist software engineer…ah. Yes. I think that maybe some people don’t want to, whether or not they know what the generalist would do. They seem to think it’s a “plural specialist”, and that a generalist would know all the trick questions from multiple specialisms.

This is the same thinking that yields “a senior developer is like a junior developer but faster”; it is born of trying to apply Taylorian management science to knowledge work: a junior Typescript programmer can sling a bushel of Typescript in a day. Therefore a senior Typescript programmer can sling ten gallons, and a generalist programmer can sling one peck of Typescript, two of Swift, and one of Ruby in the same time.

I think that the hiring managers who count contributions by the bushel are the ones who see software engineering as a solitary activity. “The frontend folks aren’t keeping pace with the backend folks, so let’s add another frontend dev and we’ll have four issues in progress instead of three.” I have written about the flawed logic behind one person per task before.

A generalist may be of the “I have solved problems using computers before, and can use computers to solve your problem” kind, in which case I might set aside an hour to pair with them on solving a problem. It would be interesting to learn both how they solve it and how they communicate and collaborate while doing so.

Or they may be of the “I will take everybody on the team out to lunch, then the team will become better” kind. In which case I would invite them to guest-facilitate a team ceremony, be it an architecture discussion, a retrospective, or something else, and see how they uncover problems and enable solutions.

In each case the key is the collaboration. A software engineering generalist might understand and get involved with the whole process using multiple technologies, but that does not mean that they do all of the work in isolation themselves. You don’t replace your whole team, but you do (hopefully) improve the cohesiveness of the whole team’s activity.

Of course, a generalist shouldn’t worry about trying to get hired despite having to sneak past the flawed reasoning of the bushel of trick questions interview. They should recognise that the flawed reasoning means that they won’t work well with that particular manager, and look elsewhere.