Do the Scariest Thing First

I know the pieces fit, cause I watched them fall apart

Kir recently wrote about fragmented prototyping which struck a nerve. I use a very similar approach for gnarly engineering and system design problems, so figured I could share while we are at it. I call it the maximum pain upfront approach. Another name could be do the thing that scares you the most first.

It goes roughly like this. When you need to design a system, make an inventory of the tasks/challenges you expect and make a list of them. Preferably list out all of them, in detail. Then look at that list, and find the thing you know the least about - or a thing that scares you the most. Then try to “run around” your system and design the least possible amount of “glue” around the piece you are worried about. It doesn’t have to be perfect, “just enough” is enough. Make your system do something sensible, provide just that little bit of output which proves your system is sane and can roughly do what it is supposed to do. This will be your “skeleton”.

Then comes the “maximum fear” part. Laser-focus on the part of the system which scares you the most. Something you never done before. Something you do not know the constraints of. Something that requires you going 2-3 levels “down” from what you normally consider comfortable. I’ll give a few examples from my experience where I hit those “maximum fear” aspects:

• In zip_tricks the part I was completely lost about was parallel compression. In retrospect it wasn’t that necessary because we ended up not using it (the cost of speculatively compressing all of the uploaded data proved too great when more than 80% uses lossy compression formats, and will thus not compress well)
• In SyLens it was matrix transforms
• In the download server, Ruby memory consumption was the item that scared me the most - especially because initial experiments were not encouraging at all
• In idempo the atomicity guarantees and data races were legitimately scaring me quite a bit

Here is how to recognise one:

• You never did anything like this before.
• You vaguely know that the thing will have behavior you do not completely understand - like race conditions, or numeric precision issues
• You are unable to visualise the code / design which will carry the feature ahead of time
• You know that the problem touches a theory topic you are unfamiliar with, and it will require you to upskill
• You try to search for readymade solutions, and they are vaguely close – but don’t look like “closed form” solutions, or you have doubts you wil be able to use them in your case

In all of these situations, I went about it roughly the same:

• Set up the “skeleton”, or “harness”, which will run the code that scares you the most - it can either be a minimum possible implementation of the “system around” that you are building, or just a test runner. Make sure it works first, make it accept input and provide some output! These “scary problems” often cause you to backtrack - or, in extreme cases, will need the work abandoned! Be prepared for this and do not get caught with a blank sheet of paper. Having a “skeleton” will allow you some holdfast to come back to if you get lost in the woods
• Try any approaches that could work. Test-driven might work well if you can isolate the part of the system well. Metrics work great - profile and measure and try to compare various solutions to give you perspective. Use visualisations, sketches, Matlab plots, Jupyter notebooks, Excel - anything that gets you closer to a solution is fair game! Doesn’t have to be in the target language even (sometimes).
• Time-box your effort. This is important! Scary problems are nerdsnipe nirvana. You can become so consumed that you will lose track of all the other work that needs to be done. If you do not have a time limit a scary problem can consume you for months and you won’t even notice.

In my previous work I had a few jobs where I would laser-focus on a particular part of the job, partially because it was the chunk I was unsure about the most. I would then spend most of my allotted time – without the “skeleton” setup in place – on “nailing” the part I was fearful of. As the time ran out, I would en up in a situation where only about 20% of the “scary thing” was done, but there was nothing besides it. No skeleton, no reversion strategy and no “big strokes” version to backtrack to. This was very embarrassing and painful for the clients/stakeholders too!

So, imagine you end up with a problem similar to what Kir encountered: “Make programming language X do Y bytes per second over interface Z.” You have never done this before, you know that this requires detail work and it might yield a negative outcome (“it is not possible to make this thing do that”). Time-box for it! Allow yourself a day or two just for that problem, and have a wrapper in place. If you fail, you can backtrack to the wrapper and begin again, or replace your yet-missing implementation with a shim of some kind. It will also give you space to try again later. On some of these problems, I had to make 2, 3 or even 4 attempts before the final solution emerged.

A few caveats are in order of course.

• When you see a problem like this - think about bypassing it outright. For example: you know that DynamoDB has write and read quotas, and you are afraid of hitting them. Think about whether your project needs DynamoDB to begin with. Could you do without it? Maybe using a datastore with different guarantees and tradeoffs can get you to the end solution faster, and will allow you to skip the problem?
• What will you do if you are unable to solve the problem? Have a plan B. If your system is moot without the scary component: congratulations, you have potentially blocked yourself.
• Have a buddy. You will, after some time in the field, have a list of names you could scroll through – of people who “know a lot about X”. Ask the person who knows the most about the part that scares you so much, maybe not immediately but it helps to have them in the back of your mind. This is where bona-fide networking becomes essential.
• This is not necessary everywhere. Sometimes there is no “scary component”, there is just… grind. Time pressure, temperamental client/stakeholder, shitty deployments, that sort of thing. Know to recognise and manage accordingly. Especially when we get bored out of our mind, we tend to create complicated contraptions, fight through their complexities and then admire the end result, while they should not have been applied in the project to begin with.
• The technique is usable when you can recognise the scary problem. It is going to be much, much harder for a junior to recognise those – and they are different for each and every person (there is an intersection of your skillset and problems you can attack). A crucial task for a mentor is to find those scary parts ahead of time, and either steer the mentee around them or try to attack them ahead of time to give the mentee some cover.
• And obviously: the bigger the team, the easier it would be to find people familiar with the topic of the scary problem. They might be able to crack it for you quickly – so divide work if you can. If your team can have good, hot, frank conversations about areas of expertise: you are in luck.
• It could be that the true challenge lies where you did not expect it instead. By investing time in the scariest thing you wil rob yourself of time that you could have used for discovering the “unknowns” you didn’t even think existed, and they would turn out even scarier. So again: the “scariest thing upfront” technique I would recommend to experienced users.

This approach has saved my bacon quite a few times. Use with moderation, and may you always succeed.