Disownership, Pull Requests And De-Facto Architects

 · 

A while ago I got really triggered by by the following two tweets - one by Avdi and another by Pete

This essay was on my mind - and lying dormant - for a couple of years, but I think it didnt lose its relevance.

I must admit that I have been struggling with this a ton, and while the tweets do outline the problem I think we are missing out on a very important component. We’ve all seen more than our fair share of systems that were “difficult to work with”, or “too large to keep in one’s head”, or “too complex and needing focus to understand”. In a few instances there was indeed a lot of complexity present, usually because a system was very tied to a specific piece of hardware or an OS primitive: for instance, how do you ensure your OS really really does fsync() at the right point? What are the failure modes when this gets used with NFS? During shutdown? During a progressing storage media failure? and the like. Or when a certain computation neds to be optimized to the maximum, and you really need to take a lot of the minutiae of your language and runtime into account to get there?

But most of the “complex systems” I’ve encountered came not in the shape of being “complex” but of being “complicated”, or rather overcomplicated. They evolved into being complicated because there was, knowingly or unknowingly, a process of “complicationing” them throughout their lifetime. Very often these systems can be characterised by actually making money for the owning business too! since they have survived a number of organisational evolutions and transformations, changes in direction, changes in scope, changes in runtime or changes in personnel working on them. With a lot of these “complicated” systems there was an incredible amount of value to be had from analyzing the political context in the organization at the time those systems evolved or emerged. Very often this context can be second-guessed from the Git commit history. If you examine an application which has served for long enough, you will invariably see a few key names in the commits - those were the de-facto architects of that system. Formally or informally, it were them who have shaped the direciton of the system going forward, and have reacted to the organizational tantrums and undercurrents. Sometimes they left, like ancient civilizations whose scriptiures and wall carvings we admire so much. Sometimes they persevered, and stuck to the bitter end (the last example being a - to me - good natured trend of companies making their source code available when going out of business). What I did see happen, though, was that the system - if it is at least somewhat successful - outlives the tenure of those de-facto architects at the company. It will evolve, change, mutate, transform in different ways - but the end outcome would be similar, the system would be passed on to a new hire (or a team of new hires) who would then in turn become the de-facto architects. People do move on, after all.

This process is a living embodiment of today’s concept of “shipping your org chart”. At a certain moment person X was in charge of a system, and later on person Y took over from person X. Then person Y got fired over a scandal, or moved on for an interesting position at a FAANG, or got finally offered a coveted CTO title at a startup where they could call the shots. Or left to build a cabin in the Swedish forests. The outcome though: the resulting, “de facto” design of the system will then also be a reflection of the passions and tribulations of that person. If you scan through the commit logs of an application which has existed for a while, you will often see certain milestones, for example:

Although we all may strive for “collective code ownership” factually it will never, ever materialize. Even in Soviet Russia there was no “collective art”. Yes, you can have a collective artwork of 2 or 3 artists who have willingly decided to collaborate on something (a rare but incredible occurrence!) but they chose to get together out of their own volition! People hired into an organization, by different managers, at different developmental stretches as the org matures, are very unlikely to end up in a situation of spontaneous shared creativity. Spontaneous self-immolation might be more likely I’d say, especially if both of them hold opinions™.

And this is where we get to the crux of the issue. That difficult person “imposing their style” in a PR review – ask them about a couple of the milestones you saw in the codebase. How did they come to be? Was this person consulted? Were there overriding orders? Was there an extreme deadline imposed somewhere? is there a reference to one of those milestones in the review they left under your code?

If you dig deep enough, it will often become apparent that a lot of the situations where people are overbearing in code reviews - or have to deal with a complicated system and focus to “load it into their L3 mental cache” is because, well… what if they did do a lot as to make that system less complicated? What if they did flag it that an integration with an external service exponentially increases the code paths you have to cover? What if they did flag it that implementing a certain feature “without UI” was discarding a very specific, tangible use case for which now there is a 2K-line workaround in place?

The outcome of this conversation might surprise you, or change your perspective on “that difficult guy”. Yes, it could have been an asshole - but it could have, just as well, been a genuinely caring and forward-thinking person.

Also note that this political context is largely taboo for a group conversation, both in Western and in Eastern cultures. It can only be broken in a super-aligned, cohesive team (one might even say that this team would be a group-think-team) where an incredible amount of trust is present.

Consistently, from asking a few people who have been in those situations, but also from having been in those situations myself, a pattern was always present that a person had to be the de-facto architect, but with going into that role there is another card attached - the power to say “no”. And at some key junctions of the evolution of the org, this power was not available. Or taken away. Or never given to begin with. So in a way, you are not an “owner” of something but a “custodian”, and any “new” person or current (“Let’s use GraphQL for everything”, “Let’s rewrite everything from PHP to Ruby on Rails”, “Let’s use Redux everywhere”) may come into the garden under your custodianship and trample on all the vegetation you have tried so hard to preserve. It is not an enviable position to be in, and exactly one of the gut reactions to this is being a dick about “small things”. It comes from a position of insecurity. Because you know that nobody is going to listen to things you say about topics that really matter and they are not going to be addressed - you revert to, at least “in the small”, to the “if we keep just this rose, just in this little corner”, you might be able to prevent a nuclear disaster or a problem down the line. Or at least feel like you are able to perform your work with dignity – it can give you an illusion of having autonomy and control.

There are a few key forces driving this process. Firstly, organizations want to be independent of a creative professional - yet they consider themselves fully entitled to the fruits of that professional’s creativity. Secondly, nobody can guarantee that such a professional will be able to stay aligned with the organization along its evolution. Yesterday we had to “not be evil”, today we have to prioritize adverts for competing products over a product’s website. With the incredible amount of money flowing through the ecosystem it is only fitting that if, say, such a key person would decide to “say no” at that crucial moment (a new investment round maybe? a dilution? appointing a politician implicated with authorizing torture to your board of directors?) the business may be in jeopardy. I am deliberately picking some very extreme examples here and specifically hammering on ethics, very often the disputes will be much smaller and the “right side to be on” – much harder to pick. But the disputes will be there. Naturally, an organization will try to de-risk operation by removing as much authority from such a creative professional as possible, and make no mistake: making decisions about how a system evolves is also authority. Deny a person their voice enough times, and they will start behaving like a jerk. Or leave.

So don’t be surprised if you end up with a spaghetti monster instead of a clean design, and please do avoid generalisations about “people who do X also always do Y”. Maybe the actual problem is that we are so scarred by the stories of “John is the only one who may touch X” that we are denying software authors ownership or consultation? And that is where our overcomplicated systems come from?

Note that this is specifically not one of the other sutuations where pull-requests are useful even outside async, open-source work in low-trust environments:

Of course, what I am describing is inherently a “low trust” environment, as Avdi duly notes - but this is exactly how people lose trust in each other.