On software engineering hermeneutics

When I use a word it means just what I choose it to mean — neither more nor less.

Humpty-Dumpty in Alice through the Looking Glass

In my recent round of TDD clarifications, one surprising experience is that folks out there don’t agree on the definition of TDD. I made it as clear as possible in my book. I thought it was clear. Nope. My bad.

Kent Beck in Canon TDD

I invented the term Object-Oriented, and I can tell you I did not have C++ in mind.

Alan Kay in The Computer Revolution Hasn’t Happened Yet

I could provide many other examples, where a term was introduced to the software engineering state of the art meaning one thing, and ended up meaning “programming as it’s currently done, but with this small change that’s a readily-observable property of what the person who introduced the term described”. Off the cuff: “continuous integration” to mean “running automated checks on VCS branches”; “Devops” to mean “hiring Devops people”; “refactoring” to mean “editing”; “software engineering” to mean “programming”.

I could also provide examples where the dilution of the idea was accompanied by a dilution of the phrasing. Again, just typing the first ideas that come into my head: Free Software -> Open Source -> Source Available; various 1990s lightweight methodologies -> Agile Software Development -> Agile.

Researchers of institutions and their structures give us tools that help understand what’s happening here. It isn’t that software engineers are particularly bad at understanding new ideas. It’s that software engineering organisations are set up to reproduce the ceremonies of software engineering, not to be efficient at producing software.

For an institution to thrive, it needs to be legitimate: that is, following the logic that the institution proposes needs to be a good choice out of the available choices. Being the rationally most effective or most efficient choice is one legitimising factor. So is being the thing that everybody else does; after all, it works for them, so why not for us? So is being the thing that we already do; after all, it got us this far, so why not further?

With these factors of legitimacy in mind, it’s easy to see how the above shifts in meaning can occur. Let’s take the TDD example. Canon TDD says to write a list of test scenarios; turn one item into a runnable test; change the code to make that test and all previous tests pass; optionally refactor to improve the design; then iterate from the second step.

First person comes along, and has heard that maybe TDD is more effective (rational legitimacy). They decide to try it, but their team has heard “working software over comprehensive documentation” so they don’t want to embarrass themselves by writing a list of test scenarios (cognitive legitimacy). So they skip that step. They create a runnable test; change the code to make that test pass; optionally refactor. That works well! They share this workflow under the name TDD (Red-Green-Refactor).

Second person comes along, and has heard that the cool kids (Kent Beck and first person) are doing TDD, so they should probably do it too (normative legitimacy). They decide to try it, but they notice that if they write the code they want, then write the tests they want, they end up in the same place (they have code, and they have tests, and the tests pass) that Canon TDD and TDD (Red-Green-Refactor) end up in. So where’s the harm? Now they’re doing TDD too! They show their colleagues how easy it is.

Now everybody is doing a slightly different TDD, but it’s all TDD. Their descriptions of what they do construct the reality in which they’re doing TDD, which is an example of what the researchers call performative discourse. TDD itself has become ceremonial; the first and subsequent people are doing whatever they want to do and declaring it TDD because the legitimate thing to do is called TDD.

This does give those people who want to change software engineering some pointers on how to do it. Firstly, overshoot, because everybody’s going to meet you a short way along the path. Secondly, don’t only talk up the benefits of your proposed change, but the similarities with what people already do, to reduce the size of the gap. Thirdly, make sure that the likely partial adoptions of the change are improvements over the status quo ante. Fourthly, don’t get too attached to the words you use and your choice of their meanings: they mean just what anybody chooses them to mean—no more and no less.

Posted in philosophy after a fashion, social-science, software-engineering | Leave a comment

On rational myths

In my research field, one characteristic of institutions is their “rational myths”; ideas that people tell each other are true, and believe are true, but which are under-explored, unverified, and under-challenged. Belief in these myths leads to supposedly rational actions that don’t necessarily improve efficiency or performance, but are done because everyone else does them, and everyone collectively believes they’re what one does.

We know, from Derek Jones’s Evidence-based software engineering, that what we know about software engineering is not very much. So what are the rational myths where you work? Do you recognise them? Could you change them? What would it take to support or undermine your community’s rational myths, and would you want to take that risk?

Posted in academia, social-science, software-engineering | 4 Comments

We shall return one day

On this day 80 years ago, 16th November 1943, the villagers of Tyneham near Lulworth was evacuated to allow Allied military forces to prepare for D-Day. Despite promises that the evacuation was temporary, the UK lurched directly from the second world war into the Cold War and decided to keep the land to practice against the new “enemy” and former ally, the Soviet Union. Tyneham remains uninhabited, and remains within a live firing range. People may only visit when the Ministry of Defence are ready for them.

In a time when people are still being displaced by war across the world, we remember the villagers of Tyneham, and an occasion when the country displaced its own citizens. The ten tracks on this album contain music, song, and storytelling from around Dorset. With no voices left in Tyneham, all parts are performed by the same person, but throughout we hear the message from the locals: “We shall return one day”.

Listen here: https://soundcloud.com/user-343604096/sets/we-shall-return-one-day

Posted in music | 1 Comment

In which things are given names

I recently joined in a very interesting discussion with some of my peers on the thorny subject of naming variables in programs. The core question was whether it’s OK to give a variable a temporary name while you’re still working out what it’s for, or whether you should pause and think out what it’s called before you move on.

The question raises other questions, and those are much more interesting to consider. For example, there’s an aphorism in computing that naming things is one of the hardest problems we have. That isn’t true. We’ve been naming things for 60,000-100,000 years, and writing down names for things for 5,000 years.

If you know about the thing you’re naming, and you know what the name should convey, then naming things is easy. For example, this blog post is on the topic of naming things, and communicating the topic is an important part of the title, so calling the post “On naming things” was very easy. Then, because I’m a comedian, I decided to go back and use a different name.

If naming something is hard, either we don’t know something about it, or we don’t know something about communicating that knowledge. The second of those is usually much simpler than the first, when it comes to variable names. The variable name is only used by other programmers: either inspecting the program text, or understanding dynamic behaviour in a debugger. The variable represents a snapshot of a part of the program state, and its name should communicate how that snapshot contributes to the valuable computation the program encapsulates.

It’s therefore likely that when we struggle to name a variable, it’s not because we haven’t identified the audience. It’s because we haven’t identified what the variable is, or what it’s for.

In most software design methodologies, we derive the existence of variables from aspects of the design. In the incremental refinement approaches described by people like Tony Hoare and Niklaus Wirth, as we refine a specification we identify the invariants that hold at each level, and the variables we need to preserve those invariants.

In DeMarco’s structured analysis, we design our systems by mapping the data flow: our variables hold those data and enable their transformation.

In Object-Oriented analysis and design, we design objects that take on particular roles in an interaction that models the domain problem. Our variables represent the responsibilities and collaborators known to each object.

In Test-Driven Development, we identify a desirable change in the system’s behaviour, and then enact that change. Our variables represent contributions to that behaviour.

It’s likely that when we can’t name a variable, it’s because we haven’t designed enough to justify introducing the variable yet.

As a specific example, if we’re thinking about an algorithm in a process-centric manner, we might have a detailed view of the first few steps, and decide to write a variable that stores the outcome of those steps and is used as input in the subsequent steps. In such a case, the variable doesn’t represent anything in the solution model, and is going to be hard to name. It represents “where I got to in the design before I started typing”, which isn’t a useful variable name. The solution in this case is neither to come up with a good name, nor to drop in a temporary name and move on. The solution is to remove the variable, and go back to designing the rest of the algorithm.

Posted in code-level | 1 Comment

I’ve vastly misunderstood the Single Responsibility Principle

So have a lot of other people; my understanding of it accords with much writing I’ve seen about the principle on the web, and elsewhere. For example, in Michael Feathers’ “Working Effectively with Legacy Code”:

Every class should have a single responsibility: It should have a single purpose in the system, and there should be only one reason to change it.

Michael Feathers, Working Effectively with Legacy Code (p.246)

I came to question that understanding today when I read Parnas’s article On the Criteria to be Used in Decomposing Systems into Modules. His top criterion is decision hiding: everywhere you make a choice about the system implementation, put that choice into a module with an interface that doesn’t expose the choice made.

If you combine that principle with my incorrect SRP (“one reason to change per module”), you get the absurd situation that each module may contain only one change point. In other words, each bit of software architecture information must exist in a separate module.

So, I went back to my understanding of the SRP. I found that it was flawed, and that the person who coined the phrase Single Responsibility Principle (Robert Martin) even said as much. He said it a long time ago, but years after the incorrect interpretation had got its shoes on and run twice around the world.

When you write a software module, you want to make sure that when changes are requested, those changes can only originate from a single person, or rather, a single tightly coupled group of people representing a single narrowly defined business function.

Robert Martin, The Single Responsibility Principle

So the principle is that the module’s behaviour is the responsibility of a single actor. It’s not that the module has a single reason to change, but that a single entity will request the changes. This is much easier to resolve alongside Parnas’ version of modularity.

This isn’t some new deep revelation or hidden insight, by the way. That post by Martin is referenced on the wikipedia entry for SRP, which states the true (well, the as-given) definition of the principle. The fact that I and so many others can hold a completely different view, for so long, in the face of such obvious contradictory evidence, tells us something about knowledge transfer in software engineering that we probably ought to attend to.

Posted in architecture of sorts | Tagged | 7 Comments

Programming, language

Programming languages represent two things: programming, and language.

Programming languages were previously designed very much with the former in mind. For Algol-style, imperative languages, design followed one of a few, mathematically-led approaches:

  • Denotational semantics: encourages a designer to identify a mathematical structure that correctly expresses transformations that programs in the desired language should represent, and design the language such that each operation realises a transformation in this structure.
  • Axiomatic semantics: encourages a designer to design operations that transform program state, and again to design the language so that it represents combinations of those operations.

Other semantics are available: for example if you operationalise the lambda calculus you end up with LISP, and if you operationalise the pi calculus you find occam-π. Indeed, when Backus complained about the imperative programming style in Can programming be liberated from the von Neumann style?: a functional style and its algebra of programs he wasn’t asking us to give up on a mathematical basis for programming languages.

Quite the reverse: he thought the denotational or axiomatic bases were too complex for programmers who weren’t also expert mathematicians to grasp, and that languages designed that way, with their word-at-a-time transformations of memory, led to “flabby” programs that don’t compose well. He called for finding a different mathematical basis for programming, using the composition of functions as a (relatively, much simpler) starting point but incorporating the history sensitivity required to permit stateful operations.

So much for programming. On the other hand, programming languages are also languages: constructions for communicating information between people, and between people and computers. Programming languages must be able to carry the information that people want to convey: and if they want to convey it to the computer, that information can’t reside in a comment.

Thus we get approaches to programming language design that ask people what they want to say, and how they want to say it. At one extreme, almost nihilist in its basis, is the Perl-style postmodernist philosophy: it’s not up to the language designer to constrain expression so the language gives you all the tools to say anything, however you want.

More common are varying degrees of participatory process, in which people who use the language collaborate on designing new features for the language. We could identify multiple forms of organisation, of which these are a few examples:

  • Jurocracy: rule of law. People submit requests to a central committee, who then decide what to accept and produce new versions of the language.
  • Tyrrany: rule of the one or the few. Whatever happens within the community, an individual or controlling group direct the language the way they want.
  • Megalofonocracy: rule of the loud voices. People submit requests to a notice board, and whichever ones get noticed, get implemented.

There are other structures within this region.

Both approaches have their merits, and address different needs that should both be reflected in the resultant programming languages. A language with no mathematical basis offers no confirmation that constructs are valid or correct, so may not represent programming. Programming with no agreed-upon vocabulary offers no confirmation that constructs are understood correctly by machine or human audiences, so may not represent language.

Unfortunately it may be the case that we previously went through a fashion for mostly-semantic programming language design, and are currently experiencing a fashion for mostly-linguistic programming language design.

Posted in tool-support | Leave a comment

I was wrong, 80 characters is fine

Do you remember when I said, in my 2019 post Why 80?, that “80 characters per line is a particular convention out of many that we know literally nothing about the benefit or cost of, even today”?

Today I saw a “rule of 66 non-white space characters per line” mentioned in connection with TeX. I couldn’t find that reference in the TeXbook, though it’s also in Bringhurst’s “The Elements of Typographic Style” so let’s go with it for the moment.

If there should be 66 non-white space characters per line, then a line should be 66 * (average word length + 1) / (average word length) characters long to hold 66 non-white space characters, on average, if it’s displaying a run of words. In English, the average word length is about five. That gives us 79.2 characters per line.

If you’re reading English, an 80 column terminal makes sense, if Bringhurst’s justification (pardon the pun) is valid. Though I still don’t know why people suggest 72 characters for commit message lines.

Posted in design | 5 Comments

Re-evaluating second brains

Because my undergraduate Physics teaching drilled into me the importance of keeping a lab book, I’ve always kept notebooks throughout my professional career too. If I want to know (and I’m not sure why I would) what process I created to upgrade the Solaris 7 kernel on a Sun Enterprise 450, I still have the notes I took at the time and could reconstruct that process. Same when I moved into testing, and software security, and programming; same now, as a documentation author.

Being in computering I’ve repeatedly looked for ways to transfer that note-taking to a digital context. I’ve got about 2,300 links saved, some of which have followed my from my undergraduate-era Netscape bookmarks file. How many of them still point to the resource I intended to record? No idea. How many of them can I find when I need to? Not as many as I might like.

So I switched to clipping web pages into the same note-taking application I use to write text notes. That’s better as I now have an archive of the content I was trying to record, but now I meet the limitations of the note-taking application. The particular application I used to use lost some of the features I relied on when the creators rewrote the UI using a cross-platform framework, and then they fired all of their developers anyway so I lost faith. I exported those notes to a different tool, which has a different UI with different issues.

Those “second brain” notes tools are somewhat good for recall (though not when I’ve used my readable-to-me-but-not-to-the-computer handwriting to take notes, and not so good at voice notes), but the whole “application” thing means that I have to want to enter the note-taking context to use them. So I don’t use them: the ultimate failure of a notes tool. I have notes I’ve jotted on journal articles in two different software systems that don’t integrate with the notes application, and also on paper.

Paper. That thing that I started using decades ago, still use now, and that I know I can find when I need to. I’m leaning into paper for a second brain. There’s a lot of suggestion that physically writing things aids recall, meaning that I can learn from the notes I took without having to actually go back and rediscover them. And paper notes take away the anxiety that comes from not having curated my second brain software just right: not because I can get it right with a paper system, but because I don’t expect to. I know that everything’s chronological, I know that I usually wrote tables of contents, and I know that I mostly don’t have to go back to those old pages anyway.

Posted in memory, writing | 1 Comment

Type safety, undefined behaviour, and us

There appears to be a shift towards programming languages that improve safety by providing an expressive type system, automatic memory management, and no gaps in the specification that lead to “undefined behaviour”. If your program is consistent with the logic of the programming language specification, then it compiles and executes the behaviour you would understand from the source code and the language documentation. If any of that isn’t true, then your program doesn’t compile: there are no gaps for not-quite-consistent programs to fall through, that get detected later when the program is running.

When I express this set of language design constraints, you may think of Swift, of Rust, of F#. You may try to point to Java, and say that it is very precisely defined and specified (it may be, but it still has undefined behavior: consider when fail-fast iterators throw ConcurrentModificationException, for example). But when was this trend started? When did people start to think about how these programming language properties might affect their programs?

Here’s a 1974 description of the SIMULA 67 programming language. This is SIMULA as a tool for extensible program products, by Jacob Palme, published in ACM SIGPLAN notices. The quotes are presented out of order, for expressivity.

Many programming language definitions contain the word “undefined” in many places. This is bad for security, since no one knows for sure what will happen if a programmer by mistake makes something “undefined” . In the definition of SIMULA, there is almost no occurence [sic] of the word “undefined” . This means that you always can know what is going to happen if you just have a program and the language definition . All errors can be discovered as soon as a rule of the language is broken. In other languages, errors are some times not discovered until strange consequences appear, and in these languages you then have to look at mysterious dumps to try to find the error. This can never happen in a fully working SIMULA system. All errors against the language are discovered as soon as they logically can be discovered, and the error message is always phrased in SIMULA terms, telling you what SIMULA error you have done. (Never any mysterious message like “Illegal memory reference”). The type checking also means that those programming errors which are not language errors in other languages will very often be language errors in SIMULA, so that they can be discovered by the compiler. Compared to other languages I know of, no language has such a carefully prepared, fully covering security system as SIMULA, and this is very important for the production of commercial program products.

Important for security is also that there is no kind of explicit statement for freeing the memory of a record no longer needed. Such statements are very dangerous, since the program can by mistake later on try to process a record whose memory has been freed. In SIMULA, records no longer needed are removed automatically by a so called garbage collector.

Important for security is also that the data fields of new records are initialized at allocation time. Otherwise, some garbage left from previous data could give very difficult errors.

When Palme says “security”, he doesn’t mean the kind of thing we might call “software security” or “information security” in 2023: the protection of assets despite possibly-intentional misuse of the program. He really means “safety”: the idea that the programmer can know the behaviour of the program, even when the program has been deployed to the customer’s computer and is out of the programmer’s direct control.

Now, what do we learn from this? Firstly that the problems evident in C were already known and solved when C was being developed. C’s first wave was between roughly 1972 (its inception) and 1978 (first edition K&R); this 1974 review identifies important qualities evinced by a 1967 programming language; qualities lacking in C.

Also, that they maybe aren’t that showstoppers, given how much software is successfully in use today and is written in C, or a related language, or on a runtime that’s written in C. Probably that software engineering is a fashion discipline, and that safe programming languages are fashionable now in a way that they weren’t in the 1960s, 1970s, 1980s, 1990s, 2000s, and 2010s: we had access to them, and we didn’t care to use them.

But also we see support for André Gide’s position: Toutes choses sont dites déjà; mais comme personne n’écoute, il faut toujours recommencer. That is: everything has already been said, but because nobody listens, it’s necessary to say it again.

That nobody listens isn’tt a criticism of everyday software engineers. There were a tiny number of people reading ACM SIGPLAN notices in 1974, and most people who’ve entered computing in the intervening 39 years don’t know any of them. There’s no way they could be reasonably expected to have encountered this particular review of SIMULA, let alone to have taken the time to reflect on its position with respect to everything else they know about programming languages so that it can influence their choices.

Software engineering is big enough, and up until recently growing fast enough, that to a first approximation anything that’s worth reading in the field hasn’t been read by any practitioner. If an idea is important, it bears repeating. The people who didn’t hear it the first time around aren’t ignorant; they’re busy.

Posted in software-engineering | Tagged | 1 Comment

On legitimacy and software engineering

More than 400,000 software engineers have lost their jobs in the last couple of years, I wouldn’t be surprised if it’s really significantly more than half a million as some won’t have been documented anywhere that the tracker project saw. In the years leading up to Layoffapalooza, software engineers commanded high salaries (though maybe not universally high) , often with significant perks. Do these shifts in employability privileges reflect a change in the legitimacy software engineering enjoys among its peers, clients, and other stakeholders?

Let’s first identify what legitimacy is. A dictionary definition would have it that legitimacy is something like “the ability to be defended”, so as our working definition for software engineering’s legitimacy let’s use the idea that software engineering legitimacy is the right, or ability, that software engineers have to define their work and their contribution on their own terms. That is, taking as an assumption that somebody somewhere in our society wants someone to write software for them, software engineering is more legitimate if software engineers get to decide what to write, how, and how they’re evaluated, and less legitimate if somebody else (clients, managers, governments, whoever) gets to decide that. This kindof ties legitimacy with autonomy, but it also connects it with status or privilege.

Following Mark Suchman’s Managing legitimacy: strategic and institutional approaches, let’s break this down into three categories. He’s talking about institutions, so I’m pretending to make an assumption here that “software engineering” is an institution. I suspect that some people (both inside and outside the field) see it as such, and others don’t. But it also might be useful to explicitly call out organisations, interest groups, user communities, or other subcultures within the field and investigate whether they are more or less institutional (so that we can

Cognitive legitimacy

The third of Suchman’s categories, cognitive legitimacy is the idea that an institution is legitimate if it doesn’t take much effort to see it as such: in other words, that it’s consistent with the worldview and values that people already have. It’s easy to maintain cognitive legitimacy, though perhaps hard to acquire. But it also doesn’t get you much, as it’s really about existing in the background. An institution that didn’t have cognitive legitimacy might look something like the Communist Party of the USA: every time you’re reminded that it’s there, it’s a surprise and you’re not sure what to make of it.

People pushed for the cognitive legitimacy of software engineering basically from the start of the field. The 1967 NATO working group chose the name because it was illegitimate:

The phrase ‘software engineering’ was deliberately chosen as being provocative, in implying the need for software manufacture to be based on the types of theoretical foundations and practical disciplines, that are traditional in the established branches of engineering.

Indeed the preface to the second of the NATO conferences on software engineering reports of the first conference:

The vast majority of these participants found commonality in a widespread belief as to the extent and seriousness of the problems facing the area of human endeavour which has, perhaps somewhat prematurely, been called “software engineering”.

Brian Randell, who edited the two conference reports, recalls that the second conference attempted to “fast-track” legitimacy; an attempt that failed.

Unlike the first conference, at which it was fully accepted that the term software engineering expressed a need rather than a reality, in Rome there was already a slight tendency to talk as if the subject already existed. And it became clear during the conference that the organizers had a hidden agenda, namely that of persuading NATO to fund the setting up of an International Software Engineering Institute. However things did not go according to their plan. The discussion sessions which were meant to provide evidence of strong and extensive support for this proposal were instead marked by considerable scepticism, and led one of the participants, Tom Simpson of IBM, to write a splendid short satire on “Masterpiece Engineering”.

Fast-forward 54 years from that second conference, and the phrase “software engineering” is indeed part of the cognitive background of programming computers. But the institutions that underpin it are not. Like other phrases, including Agile, DevOps, and Open Source, software engineering has been co-opted by the managerial class to mean “the people we hire to do the work we want them to do, the way we want them to do it”. Research institutes like the SEI, or special interest groups like ACM SigSoft, don’t have a seat at the software engineering table in the large. Even in academia, while software engineering was meant to base practice on “theoretical foundations and practical disciplines”, it’s common that if software engineering is researched at all it’s a field in the Computer Science department. All theoretical foundation, no practical discipline.

Pragmatic legitimacy

Pragmatic legitimacy is that which supports an organisation because doing so is in the rational self-interests of the audience. A lot of support for open source software is in the form of pragmatic legitimacy: we open source our database because that will encourage grass-roots adoption, which is cheaper than a sales channel. But notice that, as said with cognitive legitimacy, when we talk about open source we talk about a managerial decision to “open source”; we don’t talk about joining the Open Source Initiative, or bringing in a representative from the Software Freedom Law Center to train our attorneys. The idea holds legitimacy if not the institution.

Come down from the conceptual to the project level, and more pragmatic legitimacy holds. An organisation uses Linux, but it doesn’t want to maintain its own Linux source tree, so it’s in that organisation’s interest to accept the Linux Foundation as a legitimate collaborator. In general “upstream” is a legitimate institution: you upstream your changes because it’s in your interest to accept the governance of the external maintainer team.

Moral legitimacy

Moral legitimacy perpetuates an institution because it represents positive values, or “the right thing to do”. A lot of people in the Free Software movement see GNU, and the FSF, as moral imperatives, so occasional missteps in governance can be forgiven or overlooked as they don’t represent the overall trajectory or challenge the belief structure.

In academia, a lot of arguments for research software engineering come from the idea that it’s the right thing to do: making reproducible software, or high-quality software, is good for research, therefore we should do it and rely on the collective expertise of organisations like the society for RSE, or the software sustainability institute, or Better Scientific Software, to help us interpret the needs. But does that align with pragmatic legitimacy, and when it doesn’t, how is the conflict resolved? Is “high-quality software” a strong enough moral imperative among all stakeholders to influence the autonomy of a whole occupation?

Posted in academia, software-engineering | Leave a comment