While it’s far from finished, my PhD thesis is now complete: there are no to-do items left, no empty sections, no placeholders. Now the proof-reading, editing and corrections continue in earnest.
I look forward to poking my head out of that rabbit hole too, and finding out what else has been happening in the world since October 2020.
There’s this paper from August 1970, called Managing the Development of Large Software Systems, that’s considered something of a classic (either for good or for bad, depending on your worldview). The discussion often goes something like this:
Let’s say I have some software to build, and I think it’s going to take about a year to build it. Few people are going to happily say “go away for a year and tell me when its done”. Instead, most people will want to break down that year into smaller chunks, so they can monitor progress and have confidence that things are on track. The question then is how do we perform this break down?
The waterfall style, as suggested by the Royce sketch, does it by the activity we are doing. So our 1 year project might be broken down into 2 months of analysis, followed by 4 months design, 3 months of coding, and 3 months of testing.
Martin Fowler, Waterfall Process
Within the phase space of Royce paper discussions, there are those who say that he argues for the strict sequential process as Fowler does, based on Figure 2 in the paper. There are those who say that he predicts some iterative, evolutionary, (dare we say it?) agile contribution to the process, based on Figure 4 in the paper. But many agree with Fowler when he says “this paper seems to be universally acknowledged as the source of the notion of waterfall”.
It isn’t. The overall process had already been described in 1956 as a “structured” (ever wonder where the phrase “structured programming” comes from, when it clearly doesn’t refer to having data structures?), “top-down” approach, by Herb Benington in Production of Large Computer Programs. Methodologists contemporary with and later than Royce, including luminaries like Barry W. Boehm and Agilistas like Alan Moran, knew about this paper, so even if we don’t use it in our software mythologies any more, it isn’t obscure and wasn’t obscure in Royce’s time.
Both Benington and Royce write within a context of large-scale government-funded projects: Benington from his experience with SAGE (the Semi-Automatic Ground Environment) and Royce at the aerospace contractor TRW. Both talk about phased approaches with dependencies between tasks (so you can’t do coding until you’ve done design, for example). Both knew about the value of prototyping, though in a mistake that makes Hoare’s introduction of NULL look like loose change, Benington didn’t mention it until 1983:
I do not mention it in the attached paper, but we undertook the programming only after we had assembled an experimental prototype of 35,000 instructions of code that performed all of the bare-bone functions of air defense. Twenty people understood in detail the performance of those 35,000 instructions; they knew what each module would do, they understood the interfaces, and they understood the performance requirements. People should be very cautious about writing top-down specs without having this detailed knowledge, so that the decision-maker who has the “requirement” can make the proper trade-offs between performance, cost, and risk.
To underscore this point, the biggest mistake we made in producing the SAGE computer program was that we attempted to make too large a jump from the 35,000 instructions we had operating on the much simpler Whirlwind I computer to the more than 100,000 instructions on the much more powerful IBM SAGE computer. If I had it to do over again, I would have built a framework that would have enabled us to handle 250,000 instructions, but I would have transliterated almost directly only the 35,000 instructions we had in hand on this framework. Then I would have worked to test and evolve a system. I estimate that this evolving approach would have reduced our overall software development costs by 50 percent.
Herb Benington, Production of Large Computer Programs
Royce, on the other hand, describes the “write one to throw away” approach, in which a prototype informs the design of the final system but doesn’t become part of it:
A preliminary program design phase has been inserted between the software requirements generation phase and the analysis phase. This procedure can be criticized on the basis that the program designer is forced to design in the relative vacuum of initial software requirements without any existing analysis..As a result, his preliminary design may be substantially in error as compared to his design if he were to wait until the analysis was complete. This criticism is correct but it misses the point. By this technique the program designer assures that the software will not fail because of storage, timing, and data flux reasons. As the analysis proceeds in the succeeding phase the program designer must impose on the analyst the storage, timing, and operational constraints in such a way that he senses the consequences. When he justifiably requires more of this kind of resource in order to implement his equations it must be simultaneously snatched from his analyst compatriots. In this way all the analysts and all the program designers will contribute to a meaningful design process which will culminate in the proper allocation of execution time and storage resources. If the total resources to be applied are insufficient or if the embryo operational design is wrong it will be recognized at this earlier stage and the iteration with requirements and preliminary design can be redone before final design, coding and test commences.
Winston Royce, Managing the Development of Large Software Systems
Royce’s goal with this phase was to buttress the phased development approach, which he believed to be “fundamentally sound”, by adding data in earlier phases that informed the later phases. Indeed, if Royce finds any gap in the documentation on a project “my first recommendation is simple. Replace project management. Stop all activities not related to documentation. Bring the documentation up to acceptable standards. Management of software is simply impossible without a very high degree of documentation.”
So we have a phased, top-down, sequential development process in 1956, that survived until 1970 when more phases were added to reduce the risk accepted in lower phases. Good job those lightweight methodologists came along in the 1990s and saved us with their iterative, incremental development, right?
Not quite. Before Boehm knew of Benington’s paper, he’d already read Hosier’s Pitfalls and Safeguards in Real-Time Digital Systems with Emphasis on Programming from June 1961. Hosier presents (in figure 2) a flow chart of program development, including feedback from much of the process and explicit “revision” steps in the control routine, unit programming, and assembly.
It’s not so much that nobody knew about iterative development, or that nobody did it, or that nobody shared their knowledge. Perhaps it just wasn’t in vogue.
Another day, another exercise from Software: A Technical History…
A software engineering project might include both standard and nonstandard engineering components. Give an example of a software engineering project where this would be appropriate.
Kim W. Tracy, Software: A Technical History (p. 43)
Buy vs. build (or, in the age of free software, acquire vs. build) is perhaps the most important question in any software engineering endeavor. I would go so far as to say that the solution to the software crisis wasn’t object-oriented programming, or agile software development, or any other change in the related methods and tools of software—those have largely been fad-driven. It was the confluence of these two seminal events:
This, the creation of de facto standards in software commons, then drove adoption of the LAMP stack on the technology side, and fast-feedback processes including the lightweight methodologies that became known as agile, lean startup, lean software, and so on.
Staffing costs aside, software development can be very inexpensive at the outset, provided that the developers control the scope of their initiative to avoid “boiling the ocean”. Therefore it can be easy and, to some extent, low-impact, to get the buy-vs-build calculus wrong and build things it’d be better to buy. But, as code is a liability, making the wrong choice is cheap today and expensive tomorrow.
One technique that helps to identify whether to use a standard component is a Wardley map, which answers the question “how closely-related is this part of our solution to our value proposition?” If it’s something you need, but not something that’s core to your unique provision, there’s little need for a unique component. If it’s an important part of your differentiation, it probably ought to be different.
Another is Cynefin, which answers the question “what does this problem domain look like”? If it’s an obvious problem, or a complicated problem, the solution is deterministic and you can look to existing examples. If it’s complex or chaotic, you need to be more adaptive, so don’t want to be as constrained by what other people saw.
Bringing this all together into an example: the Global.health project has a goal to provide timely access to epidemiological data to researchers, the press, and the public. “Providing timely access to…” is a well-solved problem, so the project uses standard components there: Linux, HTTPS, hosted databases, event-driven processing. “Epidemiological data” is a complex problem that became chaotic during COVID-19 (and does again with other outbreaks), so the project uses nonstandard components there: its own schemata, custom code, and APIs for researchers to write their own integrations.
Continuing the theme of exploring the exercises in Software: A Technical History:
Give an example of a specific physical phenomenon that software depends
Kim W. Tracy, Software: A Technical History (p. 43)
on in order to run. Can a different physical phenomenon be used? If so, give
another example phenomenon. If not, explain why that’s the only physical
phenomenon that can be used.
My short, but accurate, answer is “none”. Referring back to the definition of software I quoted in Related methods and tools, nothing in that definition implies or requires any particular physical device, technology, or other phenomenon.
Exploring the history of computing, it’s clear that the inventors and theoreticians saw computers as automation (or perhaps more accurately flawless repetition) of thought:
We may compare a man in the process of computing a real number to a machine which is only capable of a finite number of conditions…
Alan M. Turing, On Computable Numbers, with an Application to the Entscheidungsproblem (§1)
Or earlier:
Whenever engines of this kind exist in the capitals and universities of the world, it is obvious that all those enquirers who wish to put their theories to the test of number, will apply their efforts so to shape the analytical results at which they have arrived, that they shall be susceptible of calculation by machinery in the shortest possible time, and the whole course of their analysis will be directed towards this object. Those who neglect the indication will find few who will avail themselves of formulae whose computation requires the expense and the error attendant on human aid.
Charles Babbage, On the Mathematical Powers of the Calculating Engine
For any particular physical tool you see applied to computing—mercury delay line memory, “silicon” chips (nowadays the silicon wafer is mostly a substrate for other semiconductors and metals), relays, thermionic valves, brass cogs, hydraulic tubes—you can replace it with other tools or even with a person using no tools at all.
So it was then, when the mechanical or digital computers automated the work of human computers. As it was in the last century, when the “I.T.” wave displaced human clerical assistants and rendered the typing pool redundant, and desktop publishing closed the type shop. Thus we see today, that categorization systems based on “A.I.” are validated on their performance when compared with human categorizers.
Nothing about today’s computers is physically necessary for their function, although through a process of iterating discovery with development we’ve consolidated on a physical process (integrated semiconductor circuits) that has particular cost, power, performance, manufacturing, and staffing qualities. A more interesting question to ask would be: what are the human relations that software depends on in order to run? In other words, what was it about these computers, typists, typesetters, paraprofessionals, and so on that made their work the target of software? Can a different human relation be used?
The book Software: A Technical History has plenty of exercises and projects at the end of each chapter, to get readers thinking about software and its history and to motivate additional research. For example, here’s exercise 1 (of 27 exercises and 8 projects) from chapter 1 (Introduction to Software History):
Why does the definition of software in this text include “related methods and tools?” What does knowing about the methods and tools used to develop software tell us about the software?
Kim W. Tracy, Software: A Technical History (p. 43)
The definition is this: Software is the set of programs, concepts, tools, and methods used to produce a running system on computing devices. (p. 2)
Those tools and methods are sometimes “a running system on computing devices” themselves, in which case they trivially fall into the definition of software: an Integrated Development Environment is a software system that people use to produce software systems.
Sometimes, the tools aren’t themselves “a running system on computing devices”. Flowcharts, UML diagrams, whiteboards, card punches, graph-paper bitmaps, and other artifacts are not themselves applications of computing, and neither are methods and methodologies like Object-Oriented Programming, the Personal Software Process, or XP.
Both the computer-based and non-computer-based tools and methods influence the system that people create. The system is a realization on a computing machine of an abstract design that’s intended to address some set of desires or needs. The design itself is an important part of the software because it’s the thing that the running system is intended to realize.
The tools and methods are themselves part of the design because they influence and constrain how people think about the system they’re realizing, the attributes of the realized system, and how people collaborate to produce that system. As an example, software developers using Simula-67 create classes as types that encapsulated part of their design, and instantiate objects as example members of those types within their system. Software developers using Smalltalk do the same thing, but documentation about Simula-67 encourages thinking about hierarchical type systems within a structured programming paradigm, and documentation about Smalltalk encourages thinking about active objects within an object-oriented paradigm. So people in a community of Simula-67 programmers and people in a community of Smalltalk programmers would design different systems that work in different ways, even though using very similar tools.
When I read the definition of software and the exercise, I initially thought that it was wrong to think of the tools and methods as part of the software, even though it’s important to include the tools and methods (and the associated social and cultural context of their legitimacy, popularity, and importance) in a consideration of software history. I considered a tighter definition, that includes the running programs, the (intangible) artifacts that comprise those programs, and any source code used in creating those artifacts. Reflecting on the exercise and writing this response, I see that this is an arbitrary place to draw the boundary, and that a definition of software that includes the design and methods used in creating the artifacts is also workable.
Software people are always all up in the XY problem: someone asks about how to do X when what they’re really trying to solve is Y. I find the YX problem much more frustrating: where software people decide that they want to answer question Y even though what someone asks is question X.
I’ve seen a few different manifestations of this pattern:
The common advice on questions for software people is How to ask questions the smart way. The problem with this advice is that it’s written from the perspective of an asymmetric relationship: the respondent is a busy expert, the querent is an idle dilettante; the querent has a responsibility to frame their question in the optimum way for the expert to impart wisdom to the idler.
Frequently the situation is more symmetric: we’re both busy experts, and we both have incomplete knowledge of both the question domain and what we’re trying to achieve. Have some patience with other people (whichever side of the interaction you’re on), and assume good faith on the part of all involved until they present contrary evidence. That means starting from the assumption that someone asked question X because they want an answer to question X.
When people say “floating point numbers are weird”, they typically mean that the IEEE 754 floating point representation for numbers doesn’t meet their needs, or maybe that it meets their needs but it is surprising in its behaviour because it doesn’t match their intuitive understanding of how numbers work.
IEEE 754 isn’t weird, it’s just designed for a specific scenario. One where having lots of different representations of NaN makes sense, because they can carry information about what calculation led to NaN, so you shouldn’t do equality comparisons on NaN. One where having positive and negative representations of 0 makes sense. One where…you get the idea.
(What is that specific scenario? It’s one where developers need to represent real numbers, and need reliable error-signaling, and don’t necessarily understand the limitations well enough to design a good system or handle corner cases themselves. Once you’ve got an idea that IEEE 754 does something weird, you’ve probably graduated and are ready to move on.)
But you were sold a general-purpose programming language running on a general-purpose computer, and the only representations of numbers they support are actually not general purpose: fixed-width integers with optional 2s complement negatives, and IEEE 754 floating point. You don’t have to use those options, and you don’t have to feel like you must be holding it wrong because your supposedly general-purpose programming environment only lets you use specific types of numbers that don’t fit your purpose.
Check out decimal representations, arbitrary precision representations, posits, fractional representations, alternative rounding choices, and open up the possibility of general-purpose numbers in your general-purpose computer.
In his 1986 article No Silver Bullet—Essence and Accident in Software Engineering, Fred Brooks suggests that there’ll never be a single tool, technique, or fad that realises an order-of-magnitude improvement in software engineering productivity. His reason is simple: if there were, it would be because current practices make software engineering ten times more onerous than they need to be, and there’s no evidence that this is the case. Instead, software engineering is complex because it provides complex solutions to complex problems, and that complexity can’t be removed without failing to solve the complex problem.
Unfortunately, the “hopes for the silver” that he described as not being silver bullets in the 1980s are still sold as silver bullets.
Meanwhile, of his “promising attacks on the conceptual essence”, all have accelerated in adoption since his time.
Another day in which someone lamented to me the demeaning nature of the interview coding challenge. It is indeed embarrassing, when someone with more than two decades of software engineering experience is asked to complete a gotcha-style programming task under the watchful eye of an unhelpful interviewer. It ought to be embarassing for both of them if the modern IDE available to the candidate is a whiteboard, and a selection of coloured markers for syntax highlighting.
But here’s the problem: consistently, throughout those decades and longer, recruiting managers who hire programmers have been beset by candidates who can’t program. It’s such a desirable career path that plenty of people will try to enter, even those who hope to pick up on whatever it is they’re supposed to do once they get the job. And, indeed, that can be a good way to learn: what is Pete McBreen’s “Software Craftsmanship” other than an imperative for on-the-job learning and mentoring?
Many companies don’t have the capacity or ability to bring a keen learner up from scratch, or are hiring into roles where they expect more familiarity with the skill. Thus, the uncomfortable truth: to fix programmer interviews, you first need to fix programmer screening. Demonstrate that all candidates coming through the door are capable programmers at the required level, and hirers no longer need to test their programming skills.
Note: or do they? Maybe someone can program, but uses techniques that the rest of the team consider to be unnatural. Or they work best solo/paired/in a mob, and the team works best paired/in a mob/solo. Still, let's roll with it: remove the need for a test in the interview by only interviewing candidates who would pass the test.
The problem is that every approach to screening for programming comes with its own downsides.
The economic approach, as currently practised: keep people away by making the career less desirable, by laying off hundreds of thousands of practitioners. The problem here is plunging many people into financial uncertainty, and reducing the psychological safety of anyone who does remain.
Moving the problem upstream: sending out pre-interview coding challenges. This suffers many of the same problems as live coding, except that the candidate doesn’t have to meet the dull gaze of a bored interviewer, and the interviewer doesn’t know it was actually the candidate who completed the challenge. I suppose they could require the candidate to sign their submission, then share their key fingerprint in the interview. An additional problem is that the candidate needs time outside of the interview to complete the challenge, which can be difficult. Not as a difficult as finding the time to:
Maintain a public portfolio. This biases towards people with plenty of spare time, or who get to publish their day-job work, or at least don’t have an agreement with their day-job employer that they don’t work on outside projects.
Our last possibility is the more extreme: de-emphasize the importance of the programming skill, so that the reason employers don’t need to screen for it is that it’s less essential as a hiring criterion. This was tried before, with 1990s-style software engineering and particularly Computer-Aided Software Engineering (CASE). It didn’t get very far that time, but could do on a second go around.
When I use a word it means just what I choose it to mean — neither more nor less.
Humpty-Dumpty in Alice through the Looking Glass
In my recent round of TDD clarifications, one surprising experience is that folks out there don’t agree on the definition of TDD. I made it as clear as possible in my book. I thought it was clear. Nope. My bad.
Kent Beck in Canon TDD
I invented the term Object-Oriented, and I can tell you I did not have C++ in mind.
Alan Kay in The Computer Revolution Hasn’t Happened Yet
I could provide many other examples, where a term was introduced to the software engineering state of the art meaning one thing, and ended up meaning “programming as it’s currently done, but with this small change that’s a readily-observable property of what the person who introduced the term described”. Off the cuff: “continuous integration” to mean “running automated checks on VCS branches”; “Devops” to mean “hiring Devops people”; “refactoring” to mean “editing”; “software engineering” to mean “programming”.
I could also provide examples where the dilution of the idea was accompanied by a dilution of the phrasing. Again, just typing the first ideas that come into my head: Free Software -> Open Source -> Source Available; various 1990s lightweight methodologies -> Agile Software Development -> Agile.
Researchers of institutions and their structures give us tools that help understand what’s happening here. It isn’t that software engineers are particularly bad at understanding new ideas. It’s that software engineering organisations are set up to reproduce the ceremonies of software engineering, not to be efficient at producing software.
For an institution to thrive, it needs to be legitimate: that is, following the logic that the institution proposes needs to be a good choice out of the available choices. Being the rationally most effective or most efficient choice is one legitimising factor. So is being the thing that everybody else does; after all, it works for them, so why not for us? So is being the thing that we already do; after all, it got us this far, so why not further?
With these factors of legitimacy in mind, it’s easy to see how the above shifts in meaning can occur. Let’s take the TDD example. Canon TDD says to write a list of test scenarios; turn one item into a runnable test; change the code to make that test and all previous tests pass; optionally refactor to improve the design; then iterate from the second step.
First person comes along, and has heard that maybe TDD is more effective (rational legitimacy). They decide to try it, but their team has heard “working software over comprehensive documentation” so they don’t want to embarrass themselves by writing a list of test scenarios (cognitive legitimacy). So they skip that step. They create a runnable test; change the code to make that test pass; optionally refactor. That works well! They share this workflow under the name TDD (Red-Green-Refactor).
Second person comes along, and has heard that the cool kids (Kent Beck and first person) are doing TDD, so they should probably do it too (normative legitimacy). They decide to try it, but they notice that if they write the code they want, then write the tests they want, they end up in the same place (they have code, and they have tests, and the tests pass) that Canon TDD and TDD (Red-Green-Refactor) end up in. So where’s the harm? Now they’re doing TDD too! They show their colleagues how easy it is.
Now everybody is doing a slightly different TDD, but it’s all TDD. Their descriptions of what they do construct the reality in which they’re doing TDD, which is an example of what the researchers call performative discourse. TDD itself has become ceremonial; the first and subsequent people are doing whatever they want to do and declaring it TDD because the legitimate thing to do is called TDD.
This does give those people who want to change software engineering some pointers on how to do it. Firstly, overshoot, because everybody’s going to meet you a short way along the path. Secondly, don’t only talk up the benefits of your proposed change, but the similarities with what people already do, to reduce the size of the gap. Thirdly, make sure that the likely partial adoptions of the change are improvements over the status quo ante. Fourthly, don’t get too attached to the words you use and your choice of their meanings: they mean just what anybody chooses them to mean—no more and no less.
The book about all the things software writers do that aren't writing software.
