Julik Tarkhanov

Have less stuff

When I joined Cheddar I was pleasantly surprised to find a sharp, mostly minimalist development culture which was a pleasant departure from the ZIRP-era “platform and team sprawl” that was in full bloom at WeTransfer.

The mostly-Shopify-inspired mindset - the Shopify of roughly 2012-2014, when Bouke and Tariq were there, long before the sharded-the-world Shopify of today - was extremely poignant and felt like a breath of fresh air:

  • Let’s not have services, at most - we’ll have 2
  • Let’s not have Kubernetes and use GCP native VM scaling instead
  • Let’s not have frontend bundling of any kind and rely on Rails builtins for any internal web plumbing

Since then I took this approach to heart, and furthered it even when Bouke left for Monumental, and Emil chose for greener pastures as well. Time and time again, when we were faced with adding more systems and services to the stack, we resisted - and while I can’t prove we were right (we never ran the branch of history where we caved), not once did we later sit around wishing we had the extra stuff.

So what is it that made stuff my biggest enemy, and why should you consider joining the cause?

🧭 I am currently available for contract work. Hire me to help make your Rails app better!

In this line of work that is about as close to proof as you get. The overall list would have been long, but we can note the key turning points and decisions:

  • Instead of furthering the service portfolio, we actually fused the Go server app into the Rails app as a set of endpoints and modules
  • Instead of relying on Lambda running, we added responders to our monolith app
  • Instead of using GCP PubSub for job queues, we migrated good_job into our own database-based queue adapter called gouda as solid_queue was still not available back then
  • Instead of using Terraform for deployments we went on with simple scripts that call into the gcloud CLI to replace our fleet of machines
  • Instead of using Kubernetes we engaged the GCP autoscaler and scaled using raw VMs, using custom metrics for storing queue depth and other important parameters
  • Instead of creating a separate service for brand logos and matching, we embedded source-controlled YAML files and images into our base monolith image
  • Instead of using gRPC we managed to maintain strong API contracts with our Flutter app by subscribing to just the protobufs - by using twirp.

In general, at every turn of the crank - and at every addition to the system - we looked at the possible vehicles of implementing it and we asked what can we avoid using.

Introduction of stuff often comes from people

People have implicit identification with their work. And they do have a very natural fear of becoming redundant. It’s not often talked about in those terms, but a lot of Kubernetes evangelism comes from a fairly clear timeline started in the late 2010s, which people tend not to like discussing:

  • A web product has one (or several) people who were old-school “sysadmins” of the system. They were responsible for maintaining and updating servers, (not)giving others root access, keeping things patched and managed
  • Cloud comes along and a sweet package deal allows the company to move to the cloud, wholesale
  • There are no more servers to manage. In theory, the new reality allows every developer to have root on machines. There is little custom infra to do, and the permissions management is standardized on the level of the cloud’s IAM model
  • The sysadmins become worried: are we going to be made redundant? What is there for us to do? Where is our influence, our power, our job security going to go?
  • Can we adopt a sufficiently complicated technology that justifies our employment, and usurp its management so that we are safe from layoffs (you don’t fire the person who has root access to systems your company relies on and that is hard to understand)?

That last item is not a joke, and it’s not something I say just because of my anarchical views on power relationships in computing. I’ve literally heard “tomorrow this joint goes full-on Heroku, and what then remains for us to do here?” uttered not as a joke, but as a legitimate fear, an existential threat.

This mindset begets stuff. Stuff lets people keep power and control, it makes them feel useful, and it gives them a sense of security. The problem is, though, that:

  • Stuff tends to require maintenance
  • Stuff makes things harder to change
  • Stuff breaks in wild and interesting ways

So: it is important to realize that furthering of stuff is something that happens when there is no engineering doctrine and people want to feel valued and safe. And we all like to put our imprint on the world, our little “dent in the universe” as they say.

And before I get too pleased with myself: wanting to be the one who deleted all the stuff, who ran the whole company on two services and a fistful of shell scripts, is its own little dent in the universe. Minimalism can be a flex and a personality just as readily as a Kubernetes cluster can. I am not exempt from wanting my imprint - I’ve just decided I want a small one.

Stuff you love is still stuff

There is a harder case than the sysadmin guarding his job, and honesty demands we name it: people become genuinely, tenderly attached to their stuff - not because it keeps them employed, but because they built it, or chose it, or learned it at some formative moment, and now it is woven into how their engineer identity. The event bus someone designed over a hard fortnight is not a line on an architecture diagram to them; it is a small monument. Asking them to delete it feels like being asked to admit the fortnight was wasted - or worse, that they were wrong, in public. It hurts! And pretending otherwise is precisely how we - ruthless minimalists - earn our reputation for being cold.

And the attachment is usually well-earned. The thing is frequently clever. It probably did solve the problem elegantly on the day it was born - the regex corpus was a nice idea, the sidecar was a neat trick, the bespoke tracing layer genuinely shows you things. None of that changes the arithmetic. Stuff you love still breaks at 3am, still needs upgrading, still has to be explained to the next hire, still becomes the thing “only Joost knows” - and Joost adoring it does not make Joost any less engaged in that clinch with company HR regarding his burnout leave.

So be mindful that when you come around to cut stuff out - your stuff is also on the line. Moreso: if it is you doing the cutting, the very stuff you hold the most dear should be the first subjected to scrutiny.

The many horrors of stuff

It’s not hard to enumerate, but I will go through the motions anyway - because while we all know that “stuff is bad” (because complexity bad) we often do not like being reminded as to why exactly. So allow me the honor.

  • Stuff comes from vendors and gets sunset or changed at the vendor’s whim
  • Stuff often includes opaque pricing, and turns out to cost a ton of money in the long run. A simple Postgres which is just like any other Postgres in your stack can be relocated to another vendor in an aternoon. A customized sharded Postgres with QoS proxies and custom tunnels in front of it, managed by a vendor where you do not have access to the logs and where even producing a dump is difficult - well, good luck moving off of that. And vendors welcome this: the more stuff you have with them, the more you depend on them.
  • Stuff contains new things to learn, including undocumented things - which you still have to learn
  • Stuff created dependency rings (we can’t deploy X before we deploy Y, but Y calls into an event bus which is listened to by X, and also Z manages permissions on the event bus) - the more stuff you allow, the more dependencies you need to manage
  • Stuff begets distributed tracing - not because observability is bad (you want that in a monolith too), but because once a single request hops across seven services you can no longer watch it without stitching the hops back together by hand
  • Stuff fails in parts. What is your uptime for the last quarter? Well, considering that 7 of the stuffs we use have 99.5, one of the stuffs has 99.9 but one endpoint on it times out in 30% of requests… well nobody knows really.
  • Stuff creates information and management hoarding - the thing only Joost knows about, and Joost, we’ve established, is indisposed. Nobody else knows how to remove it or how to control it.
  • Stuff makes large-scale migrations difficult. If you are beset with a system composed of 20 different stuffs (including vendor cloud products) and you realize that the architecture can be simplified, moving 2 units of stuff from system A to system B is usually much easier than moving all the 20.
  • Stuff creates rifts between teams. If you have a team preferring tech X, and a team preferring tech Y, and the two teams are permitted to produce stuff - rest assured that they will, mostly out of spite and conviction - not because having stuff is useful for the business or for the tech doctrine of the firm, but because it allows them to not talk to each other. In reality, not talking to each other and furthering rifts by producing more stuff should be a punishable workplace transgression.

The stuff creep

Stuff is everywhere, and when you lower your guards: it will eat you alive, and start requiring substantial human resources to maintain. Plenty of sober teams still pick self-managed Kubernetes, and for real reasons - control, cost at scale, air-gapped environments, not wanting a cloud provider’s opinions baked into their cluster. While I have opinions, I would still like to lay out the bill if what they are actually subscribing to with this - because the bill is rarely shown at the point of sale. So: a simple chain of decisions to adopt stuff, a fairly popular one around 2018-2020:

  • To manage our deployments, let’s use Kubernetes!
  • Well, with Kubernetes we need to decide which one we will use - managed from our cloud provider or self-managed. Let’s do self-managed! We’ll have more control!
  • Oh, now we need to manage configurations. What do we have? Oh, let’s get Helm.
  • Oh, and we also need to manage interconnections between our services inside Kubernetes, so let’s get istio.
  • Oh, services need configuration - let’s use kustomize.
  • Oh, kustomize templates credentials - let’s get a credentials management system
  • Oh, credentials now do not update when we deploy our services - and besides, we should not allow services to reconfigure their own credentials - let’s write a custom tool that rolls credentials independently of deployments.
  • Oh, credentials are now encrypted in files using some weird keypair setup - we need to have the private key of the keypair in our credential store in the cloud platform, and export it into Kubernetes which is self-managed
  • Oh, now we need a tool to expose the private key to the pods - let’s write a sidecar pod which does credential import from our secrets store…
  • Oh, now our instances authenticate under the same IAM machine role, whereas every pod is supposed to be running under the role appropriate for the service - more kustomize, and we also need to expose the role setup to our services
  • Oh, the services use the AWS SDK to self-configure when running on AWS infrastructure - let’s get us an adapter that exposes machine info with appropriate roles into the pod - or, even better, let’s force all our services to explicitly authenticate using their assigned IAM keys!

This snowball, once it gets rolling, is nigh impossible to stop - and, crucially, you don’t just backtrack out of all this tech you have so valiantly accrued. Now you need someone on your team to know and manage kustomize. To manage Helm. To upgrade Kubernetes (you picked a self-managed one, upgrades are on you now). To manage IAM bindings. To manage networking between pods and instances.

This is the worst kind of stuff proliferation, stuff that is effectively forever. Most of stuff, once applied, is forever.

Resisting stuff

  • Instead of Kubernetes we will run our workloads on autoscaling machine groups. A service specifies which machine configuration it needs and what templates it uses for networking and other capabilities. The service definition automatically pulls the correct Docker image with the correct tag and spins it up
  • To serve credentials, we will specify encrypted environment variables which will be used to start Docker containers with our service inside the machine. The only environment variable will be the master key for the secrets
  • Secrets will be stored encrypted in the service repositories. Secrets updated -> service redeploys -> secrets get loaded from the state they were in in the branch

There, you have just saved yourself from a proliferation of stuff, and the only thing that you had to compromise on were the egos of your “right tool for the job” engineers.

And Kubernetes is just the most proverbially known cesspit of stuff. Simpler examples:

  • Want LaunchDarkly? Just use Rails and install Flipper No, seriously - just install Flipper
  • Want gRPC? See whether a properly encoded JSON API will do the job just as well. Just use Rails. Why? This frees you of protoc, of running 2 web servers in parallel, of forking bugs, of hung services, of installing more dependencies on developer machines, it shows the payloads in the Chrome dev tools.
  • Want a separate service for matching bank transactions to merchants? Just get a corpus of regular expressions and store them in YAML files. Vet the YAML files for Norway (yes, you can absolutely do this). Generate unit tests from those YAML files. Do not create a service.
  • Want sophisticated ETL? Don’t get sophisticated ETL, just bulk-export CSVs from your main DB on a fixed cadence.
  • Want a build-time React-based interface setup? Write Rails helpers instead, dress them with vanilla CSS and a bit of Stimulus. Just use Rails. This allows you to escape package.json, tsc, deliberations on type signatures and disjoint union types, it gives you native backtraces in the browser…
  • Want to have Lambdas? Just write HTTP handlers or functions inside your product. Use autoscaling, wrap your function calls with decent observability. This frees you from polyglot stacks, from distributed tracing, from debugging slow cold start, from debugging broken streaming, from having a separate versioning system for artifacts (remember deploying with ZIPs?)…

A fair objection, and I’ll meet it head on: isn’t gouda stuff? Aren’t the deploy scripts, the autoscaler metrics, the corpus of regexes - aren’t those all my stuff, with the same bus factor and the same “only Joost knows” (well, Julik then - for the same amount) risk? Yes. Homegrown things are stuff too, and I won’t pretend otherwise. The difference I’ll defend is one of degree: a few hundred lines of Ruby that call gcloud are legible to anyone on the team in an afternoon, they carry no vendor, no separate control plane, no pricing page, and I can delete them the day they stop earning their keep. That is not the same beast as being married to a self-managed cluster. It’s not nothing, but it satisfies 2 criteria of acceptable mini-stuff:

  • It is small and easy to understand
  • It is easy to remove (gouda can be replaced with solid_queue in an afternoon)

LLMs permit smaller (and less!) stuff

For a long time the honest argument for stuff was cost of construction. Yes, you only needed a sliver of what Kafka does - ordered, durable-ish fan-out for a handful of topics - but writing even that sliver correctly was a fortnight you didn’t have, so you swallowed the whole broker, the whole operational manual, the zookeeper, the mother and her kitchensink - to get your 10%. Nowadays, that equation has quietly flipped. When an LLM can draft, test and harden the small version in an afternoon, the reason to adopt the big version - “building it ourselves is too expensive” - mostly evaporates, and what’s left is the part nobody wanted to admit: you were dragging along 90% of a system you were never going to use. You adopted it as stuff just because “those people know better”.

They don’t - often.

So build the subset. You can stand up a perfectly serviceable tiny Kafka inside your Rails app on top of Redis - or, honestly, on top of sqlite - the moment you know your own constraints: how many topics, what ordering you truly need, whether “at least once” is fine, how long you must retain. You stop reaching for SQS or GCP PubSub, because most of what you wanted from it is fan-out and retries you can bolt onto solid_queue as an extension and be done. At Cora we run a homegrown autoscaler for much the same reason - not because scaling is a solved-by-us miracle, but because these days you can just do things, and the thing we needed was a few hundred readable lines rather than a platform. The trick was never building the whole of X; it was knowing which tenth of X your problem actually touches, and now the wall between wanting that tenth and having it is thin enough to walk through.

Conclusion

If you are in a position to manage the introduction of new stuff - if you are a VP of engineering, a director, a CTO: make new stuff expensive. Not forbidden - forbidding everything outright is just the mirror-image ideology, and I’ve no interest in trading one dogma for another - but expensive. Fight each new piece tooth and nail, make someone stand up and defend it, and put it through a process rigorous enough that only the stuff which genuinely earns its place survives contact with you. There is a scale, and a moment, where a given piece of stuff pays for itself - your job is to make quite sure this is that scale and that moment before you let it in.

Stuff bad. Complexity bad. Have less stuff.

Even if those wonderful people try to convince you otherwise.

By the way

If you have a Rails app and a nagging sense you’re oversubscribed on stuff, hit me up and I’ll help you chop some of it away. Not going on staff makes me the ideal villain: pin all the grief on the consultant who swept in, took the knife to everyone’s darlings, and left. Your colleagues keep their dignity, you keep a smaller app, and I’ll gladly be the bad guy - it washes off by the next train home.