We are uncovering better ways of developing
software by doing it and helping others do it.
It’s been 20 years since those words were published in the manifesto for agile software development, and capital-A Agile methods haven’t really been supplanted. Despite another two decades of doing it and helping others do it.
That seems problematic.
I’ve spent about a year working on an app for a group in the University where I work, that needed to be available on both Android and iOS. I’ve got a bit of experience working with the Apple-supplied SDKs on iOS, and a teensy amount of experience working with the Google-supplied SDKs on Android. Writing two apps is obviously an option, but not one I took very seriously. The other thing I’ve reached for before in this situation is React Native, where I’ve got a little experience but quite a bit of understanding having worked with React some.
Anyway, this project was a mobile companion for a desktop app written in C# and Windows Forms, and the client was going to have to pick up development at the end of my engagement. So I decided that the best approach for the client was to learn how to do it in Xamarin.Forms, and give them a C# project they could understand at the end. I also hoped there’d be an opportunity to share some code from the desktop software in the mobile project, though this didn’t pan out in the end.
It took a while to understand the centrality of the Model-View-ViewModel idea and how to get it to work with the code I was writing, rather than bludgeoning it in to what I was trying to do. Ultimately lots of X.F works with data bindings, where you say “this thing and that thing are connected” and so your view needs a that thing so it can display this thing. If the that thing isn’t in the right shape, is derived somehow, or shouldn’t be committed to the model until some other things are done, the ViewModel sits in the middle and separates the two.
I’m used to this model in a couple of contexts, and I’ll give Objective-C examples of each because that’s how old I am. In web applications, you can use data bindings to fill in parts of an HTML document based on values from a server-side object. WebObjects works this way (Rails doesn’t, it uses code to fill in parts of etc). The difference between web app data bindings and mobile app data bindings is one of lifecycle. Your value needs to be read once when the page (or XHR) is rendered, and stored once when the user posts the changes. This is so simple that you can use straightforward accessor methods for it. It also happens at a time when loading new content is happening, so any timing constraints are likely to come from elsewhere.
You can also do it in what, because I’m that old, I’ll call rich client applications, like Xamarin.Forms mobile apps or Cocoa Bindings desktop apps. Here, anything could be happening at any time: a worker thread could be updating the model, and the user could interact with the UI, all at the same time, potentially multiple times while a UI element is live. So you can’t just wait until the Submit button is pressed to update everything, you need to track and reflect updates when they happen.
Given a dynamic language like Objective-C, you can say “bind this thing to that thing with these options” and the binding library can rewrite your accessors for this thing and that thing to update the other when changes happen, and avoid circular updates. You can’t do that in C# because apparently more typing is easier to reason about, so you end up replicating the below pattern rather a lot.
public class MyThingViewModel : INotifyPropertyChanged
{
public event PropertyChangedEventHandler PropertyChanged;
// ...
private string _value;
public string Value
{
get => _value;
set
{
_value = value;
PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(nameof(Value)));
}
}
}
And when I say “rather a lot”, I mean in this one app that boilerplate appears at least 126 times, this undercounts because despite being public, the PropertyChanged event can only be invoked by instances of the declaring class so if a subclass adds any properties or any change points, you’re going to write protected helper methods to be able to invoke the event from the subclass.
Let’s pivot to investigating another question: why is Cocoa Bindings a desktop-only thing from Apple? I’ve encountered two problems in using it on Xamarin: thread confinement and performance. Thread confinement is a problem anywhere but the performance things are more sensitive on mobile, particularly on 2007-era mobile when this decision was made, and I can imagine a choice was made between “give developers the tools to identify and fix this issue” and “don’t give developers the chance to encounter this issue” back when UIKit was designed. Neither X.F nor UIKit is wrong in their particular choice, they’ve just chosen differently.
UI updates have to happen on the UI thread, probably because UIKit is Cocoa, Cocoa is appkit, and appkit ran on an OS that didn’t give you an easy way to do multiple threads per task. But this has to happen on Android too. And also performance. Anyway, theoretically any of those 126 invocations of PropertyChanged that could be bound to a view (so all of them, because separation of concerns) should be MainThread.BeginInvokeOnMainThread(() => {PropertyChanged?.Invoke(...)}); because what if the value is updated in an async method or a Task. Otherwise, something between a crash and unexpected behaviour will happen.
The performance problem is this: any change to a property can easily cause an unknown amount of code to run, quite often on the UI thread. For example, my app has a data grid (i.e. spreadsheet-like table view) with a “selection” column containing switches. There’s a “select all” button, and a report of the number of selected objects, outside the grid. Pressing “select all” selects all of the objects. Each one notifies observers that its IsSelected property has changed, which is watched by the list view model to update the selection count, and by the data grid to update the switches. So if there’s one row in the grid, selecting all causes two main-thread UI updates. If there are 500 rows, then 1000 updates need to run on the main thread in response to that one button action.
That can get slow :). Before I understood how to fix this, some UI actions would block the UI for tens of seconds as they computed the update. I asked about this in some forums and was told the answer is “your users shouldn’t have that much data in a mobile app, design an app with less data” which is not that helpful. But luckily the folks over at SyncFusion were much more empathetic, and told me the real solution is to design your views and view models such that you can turn off updates while you’re doing some big change, then turn them back on and recalculate the state at the end.
Like I say, it’s likely that someone at Apple already knew this from the Cocoa Bindings times and decided “here’s a great technology, and here’s how to turn it off because it will get in your way” wasn’t a cool story.
In which I use this hacker news discussion on Agile to ask the question: has the poacher become the gamekeeper?
I definitely talked about Questioning Extreme Programming by Pete McBreen, and Ron Jeffries has a take on that book too. I referred back to my punk rock model of software engineering from Episode 25. And I certainly talked about Scrum, too.
But I also talked about the importance of telling your stories, and I can’t put those in the show notes. Yet.
I’d like to start by recapping the three distinct categories of interest in software freedom. This is definitely my categorisation, though only the third is novel and the first two have long histories of common recognition so this is hardly Humpty-Dumptyism on my part.
This is all pre-amble to a discussion of #uninstallman, the internet pressure mob removing the leadership of the Free Software Foundation over objectionable statements made by its founder, former president, and recently-surprise-reinstated board member, Richard M. Stallman (rms).
Let’s start with the obvious: the Free Software Foundation has not demonstrated good leadership over this matter. Clearly rms’s statements have distracted the conversation away from software freedom, and the FSF have not taken enough steps with enough publicity to resolve this issue and to get people talking about software freedom again[*]. The FSF has not even given clear enough separation between their policy and rms’s personal views for it to be obvious that anyone else on their board has any views, or control over policy.
It is right that the FSF take a critical look at their management, and ask whether the people who are leading the Foundation are the best people to promote the idea of software freedom.
Unfortunately, we are now at the point where whatever the outcome, software freedom has lost and Open Sores will fill the ideological vacuum. Because if software freedom is about the extension of existing freedoms into the online space, and the baying mob are calling for the blood of someone who said a thing questioning the definitions of words related to the actions of someone who was associated with someone who did known bad things, and for the blood of any other people who are associated with that person, it is easy to argue that the whole software freedom movement is hypocritical. You claim to support freedom of expression, and yet you actually deny the right for anyone to express views that disagree with your own? Where’s the unity of purpose?
Bradley Kuhn, policy fellow at the Software Freedom Conservancy, has talked about the damaging impact of rms’s personal views on the software freedom movement back in 2019, when this controversy was fresh; and in 2018 which is arguably where it started (to become public). He has also talked about the need to maintain a big tent; that being principled on your core issue gives you the legitimacy to take principled stands on other issues.
Taking an authoritarian, only-say-what-I-permit line on expression doesn’t leave any legitimacy to support freedom of expression in the software field. Unfortunately, if the FSF and more generally the software freedom community is unable to maintain principle on this argument, it will lose the right to be taken seriously on matters of software freedom. And then, the organisations who take the Open Sores line on software licensing will step up to fill the leadership vacuum. The business interest “Foundations” who think that software freedom means the freedom for big businesses to control the revenue stream while everybody gets to build their products for free. And then it may be decades before there is another software freedom movement with any legitimacy, and they may have to start from scratch.
[*] Arguably the software freedom movement was already in a difficult state, because the freedoms proposed were only really adopted by a small community with a technical interest in the details relating to those freedoms: a few tens of thousands of technologists, some intellectual property lawyers, and a small number of others. But that’s more about the difficulty of developing a mass movement and of translating the theory into activism, and doesn’t necessarily reflect badly on the characters or actions of any of the leaders in the movement.
One of the least teamy things I see with software teams is limiting the maximum and minimum number of items of work in process – tasks, stories, whatever you call them – both to the number of developers on the team. For some reason it’s always the number of devs, never the number of product owners, customers, QAs, or deployment people. Got four devs? Then there should be four tasks in process!
This approach is surprisingly backward given that we’re all supposed to have come so far as the leaders of the Agile Fourth Industrial Revolution 2.0 that we’ve internalised and transcended Goldratt’s theory of constraints. It’s the last holdout of the old Taylorian school of management. Everybody is working full-tilt, so if anyone runs into any trouble then everybody else is too busy to help them. If what they’re doing is upstream of anybody else’s work, then they are going to be blocked too, but rather than fix the blockage they’ll pull another task because one-person-per-task at all times!
So much is this at the core of software team thinking that when I’ve suggested in informal discussions that maybe we should do something else, people are confused. Are you saying that we should have a developer who isn’t assigned to a task, just in case? What does that person do the rest of the time, play Minesweeper? As if the only alternative to “one person per task” is “one person per task but perhaps there is another person”.
One person per task has the “nobody can help” disadvantage already mentioned. In fact, people are disincentivised from helping, because their task has their name on it and your task has your name on it. Did issue #1348 miss the release train? Bob is such a drag on the team, at least Karen managed to ace her ticket. Maybe we should reevaluate who leads on the next project.
You’ll see other effects of one person per task. Code reviews fall into one of two categories: “LGTM” and “axe to grind”. Only the people who are really invested in making sure that nobody ever misses off a const keyword, or uses function() where => would suffice, will take the time to commit to code reviews. Everybody else will skim-read, look at the CI output to see if the tests pass, and get back to their own task with their own name on it as quickly as possible. This loses both the review benefit of code review, and the shared-understanding-of-the-code benefit too. Everyone only really understands the features they worked on individually, there just happens to be a big ball of those features in one repo.
Code quality suffers. Each individual is too busy chopping down trees to sharpen the saw, because there’s always a next task for each developer to do.
Everybody else has been under-resourced. We need one developer per task, because what I can see is features in a UI and the only people shovelling features are the devs. QA is a cost centre, so if we can get one QA (or at most, one per team) then let’s do that. Same with ops. Infosec, coaching, UX, and other nice-to-haves can be consultants as needed. Weird how our devs are ticking off tasks like billy-O, and nothing’s getting through to release!
The alternative to “one person per task” is not “one person per task and some change”. It’s “one objective per team”. Set the goal, and let people work out what to do about it and how everyone contributes. As they used to say, “give them the environment and support they need, and trust them to get the job done”.
In which we investigate the question why are climate models written in programming languages from 1950?, the discussion some programmers had about it, and the phenomenon that programmers seem to think that everybody else’s programming work just can’t be that difficult.
If you enjoy this podcast, please consider joining others like you in becoming a patron!
A bunch of the topics I wanted to discuss all turned out to have a common basis, so I’m going to write the post about the commonality using a couple of examples from the specific topics for illumination. Maybe I’ll come back to those topics in more depth later, each one is itself interesting and valuable.
The common thing is the Vizzini Effect, named after the Sicilian in the Princess Bride. In the movie/book, Vizzini often describes events as “inconceivable,” to which Inigo Montoya replies “you keep using that word. I do not think it means what you think it means”. The Vizzini Effect in software engineering (and undoubtedly in other fields of endeavour too, I doubt we’re special) is when the same thing happens: a word or phrase seems to adopt a different meaning such that two different people, or two different groups of people, can mean it to use different things without either seeming malicious or disingenuous. In the examples I’m going to explore here, those groups are separated by time rather than space. But unlike with Vizzini, it’s not that one person is using a word in a weird way, but that collectively software engineers seem to have decided it takes a different meaning.
OOP is perhaps the ur-example here, and definitely the one with the most obvious dog-whistle. “I invented the term Object-Oriented Programming,” says Alan Kay, “and I can tell you I did not have C++ in mind”. To Alan and that early group of Smalltalk programmers at Xerox, ParcPlace, Tektronix etc., object-oriented programming was extreme late binding and decoupling through message sending. These days, it is often programming in any language that has a “class” keyword, or a straw man meaning any form of mutable state.
In the discussion do you think Agile/Scrum is beneficial for software delivery?, the first answer (at time of writing) says “The whole thing was designed to give non-technical people more power over the ones who spent a lifetime honing their craftmanship.” The question asks about the surprising rituals and the extra layers of bureaucracy. That’s the opposite impression than the one I have, where pre-existing software engineering methods tried to minimise or even automate away the programmer contributions. The lightweight methods, promoted by (among others) the agile alliance, sought to build projects around motivated individuals, giving the support they needed but leaving them alone to get the job done. The alliance members thought that the best architectures and designs were created by self-organising teams: a far cry from imposing methodologies to remove power from technical contributors.
Design Patterns in software used to refer to the Christopher Alexander idea of identifying repeating problems in architecture and building a shared language that succinctly communicates understanding of the problem, solutions selected, and trade-offs in those solutions. These days it seems to mean any of the examples of design patterns in the Gang of Four book on early-OOP implementation patterns, and no others.
The Free Software Foundation and the GNU project were created with the goal of extending desirable human rights and liberties to the world of computing. These days it seems to mean “open source, but said by a person who uses the word actually a lot”.
The Open Source Initiative was created to generalise the Debian Free Software Guidelines out from the Debian project to general business rules for software, based on the prior successful Open Systems movement and the liberties identified in the Free Software Movement. These days it pretty much means making the components needed to build SaaS subscription products available at zero cost.
In 1967, software engineering was a provocative term, meant to imply that the art of creating software would be somewhat improved if it had a socio-scientific basis. These days software engineering is two words anyone who gets or wants to get paid for programming uses on their CV/résumé.
My impression is that three things changed, and that two of them are almost the same. The origins of all of these phrases are in particular times in history, made by particular people, talking in specific contexts. Time has changed, which has changed the context (or at least the relevance of the original context), the people who said the things have changed, and so many new people have entered the field that a majority of practitioners no longer know who the original people were, nor have experienced the context in which they spoke.
It’s entirely possible, for example, that the agile methodologies which were lightweight reactions to software engineering around 2000 are oppressively bureaucratic in 2021. We expect to be able to release software multiple times per day now, using analytics and telemetry to understand in real time how it’s being used. The agile folks wanted us to release up to every two weeks and to talk to someone before doing it, ugh!
Some people talk about an Agile-Indu$trial Complex, suggesting that there’s some shady cabal of consultants and certification bodies conspiring to make us all agile so they can profit from it. Again, maybe true. Others talk of companies who “talk the talk without walking the walk”: they got the consultant in, decided which parts of this whole Agile thing sounded nice or relevant, and adopted those things then trumpeted their “fully agile workflow” on their websitesfax banners.
And, of course, there’s the telephone game. Even those of us who heard about it from the horse’s mouth—maybe worked on an XP team, or read “Free Software, Free Society”—will have learned a slightly different thing than what the originators were trying to teach us, or thought they were teaching us. When telling others, we’ll have misremembered, and adapted, and extemporised. And so will the people who learned from us, and so on.
The telephone game is subject not only to slow evolution, but to a Byzantine Generals attack. If someone wants, for example, OOP to die so that their preferred paradigm get used instead, they can inject a false message into the call graph. This is where Vizzini meets Lewis Carroll’s Humpty-Dumpty: I keep using this word, and I do not think it means what you think it means.
Take into account the fact that most people who work in software now didn’t work in software five years ago, and that this was true five years ago and so on, and you realise that the vast majority of people will have learned about any “classic” idea in software from a telephone conversation.
Well, the first question to answer is, does anything particularly need to be done? Maybe these are ideas that have had their time, and can just fizzle out. But evidently for all of the examples above enough people want the ideas to continue that they (well, we obviously) keep trying to dredge the original discussions out of the history books and put them back into contemporary discourse. To do this, they need recontextualising. For example, nobody cares that Richard Stallman couldn’t get a printer driver in the 1970s, but maybe they do care that there are things that they aren’t allowed to do with the smartphones they think they paid for. That’s how ideas of software freedom could be reintroduced.
Maybe the original phrase has become toxic and needs to be retired, without the original meaning being lost too. That is, whether you like it or not, the reason that “Open Source” was created as a term: to remove deliberate and accidental confusion over the word “freedom” in a business context, and to provide familiarity to people who had already adopted Open Systems ideas. It’s why Devops exists: to tell the stories of Agile again, but to those who didn’t listen the first time, or who listened and heard the wrong thing.
The telephone game can’t be avoided. You have to keep telling the stories if you want new people to hear them, and that means accepting alterations in their re-telling. And you need there to be more than one raconteur, even if they’re telling slightly different versions of the story. Don’t count messiahs, count prophets. Only don’t count prophets, count gospels. Only don’t count gospels, count churches. Only don’t count churches, count preachers.
You’re never going to get every programmer or software professional in the world to agree with your interpretation of some phrase. But you can use contextually-relevant stories to tell people things that might help them make better software, and you can follow up “I do not think that means what you think it means” with a conversation in which you both learn something. Maybe it’s your understanding that’s wrong?
Inconceivable!
I do a lot of writing, podcasting, presenting, and streaming about how to make software. Most of it has been free, still is free, and I don’t intend to change that. It’d be great if you are able and willing, for you to support that free work by becoming a patron. No obligation!
In which I discuss whether the reaction to traditional software engineering in works like Software Craftsmanship: the New Imperative didn’t throw out significant amounts of baby with the bathwater of waterfall processes and CASE.
I also mention my popular post reasoning about code is a scam, in which I explain that you should not be struggling to understand the software, but the problem.
And I discuss my Patreon! If you like this podcast and anything else I create and would like to help me to do more of it, please consider becoming a patron.
Another day, another developer explaining that they don’t follow some popular practice. And their reason? Nothing more than because other people do the thing. “Best practices don’t exist,” they airily intone. “They’re really mediocre practices”.
In one sense, they’re correct. Best practices need to be evidence-based, and there’s precious little evidence in software engineering. In a regulated profession, you could avoid using accepted best practice, but if something went wrong and you ended up on the receiving end of a malpractice suit, you would lose.
So best practice as an argument in software engineering has two weaknesses: the first is that there’s no basis in evaluation of practice; and the second is that being a monetised hobby rather than a profession there’s no incentive to discover and adopt best practice anyway.
But those arguments mean that best practices are indistinguishable from alternative practices, not inherently worse. If a programmer discards a practice because they claim it’s considered best practice, they’re really just stamping their foot and shouting “I don’t wanna!”
They’re rejecting the remaining evidence in favour of the practice—that it’s survived scrutiny by a large cohort of their peers—in favour of making their monetised hobby look more like their headcanonical version of the hobby. “We are uncovering better ways of making software by doing it and by helping others to do it” be damned: I want to use this thing I read a substack post about yesterday!
Dig deeper, and you’ll find only platitudinous justification based on thought-terminating cliche: I’ve already covered “Reasoning about code”, and maybe some time I’ll cover “Right tool for the job”. This time, let’s look at “things won’t advance unless some of us try new ways of doing it”.
People tried new ways of making new steam engines all the time, during the industrial revolution. People tried new ways of making chimneys all the time, during the 15th and 16th centuries. A lot of factories and trains exploded, and a lot of buildings burnt down. If you live in a house with a chimney now, or you have ever taken a train, it’s significantly less likely to have self-immolated than at earlier times in history.
It’s not, for the most part, due to misunderstood lone geniuses rejecting what everybody else was doing, but a small amount of incremental development and a large amount of theoretical advance. It’s no coincidence that the field of thermodynamics advanced leaps and bounds during the steam age. Brad Cox makes this point about software too, in almost everything he wrote on the topic: you don’t get as much advance from random walks in the cottage industry as you do from standardisation, mass production, the division of labour, and interchangeable parts that can be evaluated on merit with reference to a strong theoretical underpinning.
Of course, the “reason about code” crowd try to stop this from happening, because if that advance happened then the code-reasoning would quickly disappear to be replaced with the problem-domain-reasoning that’s significantly harder and less of a hobby. Hence the sabotage of best practice: let’s put a stop to this before anybody realises it’s more than sufficient to the task at hand.
Alan Kay once referred to a LISP evaluator written in LISP as “the Maxwell’s Equations of software”. But what software needs before a James Clerk Maxwell are the Gibbs, Boltzmanns, Joules and Lavoisiers, the people who can stop us from blowing things up in production.
The book about all the things software writers do that aren't writing software.
