Julik Tarkhanov

Serving your packages with Paquette

geneva_drive is the first thing I’ve put out for money on my own, without an org behind my back - and a rollercoaster on many fronts. One of the less obvious ones is the actual distribution: how does geneva_drive get to your servers or dev machines, once you have that ticket?

It works via paquette - my little engine for serving libraries. Here’s how it got made, and why.

🧭 I am currently available for contract work. Hire me to help make your Rails app better!

The idea is simple: you purchase access to some distribution point for a commercial package, you point your package manager to it, and your package management system (bundler or npm or what have you) does the rest.

This is pretty much how it was done by the pioneers in the scene - namely sidekiq and Avo.

When I tried to figure out how it actually worked under the hood, though, it turned out to be simpler than what I needed. Most of those are just copies of an existing codebase for distributing gems - either geminabox or gemcutter fronted by a basic authentication via nginx or Apache. And in many cases, you don’t need much more!

There are subscriptions and subscriptions

The problem was that I did need a bit more. First of all, it feels plain wrong to me to have people hooked on a subscription with a risk of losing access forever unless that subscription invoice is paid on time. It’s great for MRR and pretty bad for the user. Adobe gets a ton of hate for this model, and I believe it to be justified.

For geneva_drive, I wanted to try something different: it sells as a base package with indefinite access. The only thing you pay for is continued access to updates. Newer Rails version compatibility, schema updates, support for more databases and backends, major features… that sort of thing. I believe that while this model is less lucrative for me it makes geneva_drive a fairer, better proposition for you - the user - while still allowing for additional income and financing further development.

Furthermore, I had something in the works on the JS front as well - I do, after all, have UI opinions and they did turn into some pretty nifty software down the line - that also could be distributed.

Therefore, what I wanted was to offer my packages in a way that you would get access to a subset of versions you have actually paid for. For example, if I publish foo version 2.0.1 and you only have a license for 1.0.0 your package manager won’t even see that 2.0.1 exists - and won’t propose that you update to it.

Every copy is your own

Another item that I wanted was personalization. Different licenses have different conditions, and I wanted the copies that get served out to actually contain those conditions in the package - and, perhaps, a serial number attached.

This implies that every package that gets served should be personalized to a specific customer. Nearly all package formats are archives of some description - PyPI packages are zips, npm packages and gems are gzips - so personalizing a package involves “uncorking” the master package that is in the registry, modifying/adding certain files inside, and repackaging it for distribution.

One of the side-effects of that is that the checksum of the package will change, in nearly all distribution scenarios - and the system must produce a stable checksum for all your packages so that your package manager does not start yelling at you when you do an install. More on that below, because it turned out to be the most interesting part.

Paquette

And here we bump into a challenge: none of the off-the-shelf servers support these two features (version and library gates and personalization). First and foremost - because those features are a bit esoteric. Second - because they conflict with efficient storage and caching! Public package repositories can be so fast exactly because they can offer caching that serves many users, with the same deliverable - and many users with the same indexes and API responses.

No jungle please

There is, of course, the grown-up answer to all this. Once an org is big enough, it buys itself an Artifactory - or a Nexus, or whichever one the procurement department had lunch with in 2010. The Artifactory then serves gems and wheels and Docker images and Maven jars and Debian packages, proxies every public registry on Earth, speaks LDAP, has a license server and a plugin marketplace, and costs about as much as a mid-level engineer. You wanted a banana but you got yourself a shelf full of JARs. Somewhere deep inside the Spring calls of the concoction there is, in theory, a dependency injection hook you could use for a custom permissions model, which could express “this customer sees versions up to 1.x” - but first you will need to get Eclipse installed.

The thing is, the jungle (of banana fame) is built the wrong way around for what I wanted. With an Artifactory your app is a client of the registry: the registry owns the users, the groups and the permissions, and your storefront has to be taught about them through some SAML dance or a sync job. But all of that data is already available in my app itself, in a plain old orders table. I don’t want to replicate it into a second system with its own idea of who you are. I want the registry to ask my app.

Upon some examination, none of the options in the Ruby scene - at least - offered what I wanted. Besides, I didn’t want to maintain a .htaccess file for the basic auth - I wanted something better. I wanted the gem server to be integrated into my storefront directly - for authentication, entitlements and other features that should not involve some kind of weird format juggling at all.

Existing gem servers answer “how do I host gems?”. Paquette answers “how does my app decide, per customer, what a package registry looks like?” - and for that the registry has to be a component. And that’s how Paquette came to be.

Architecture at a glance

A package repository is, in essence, a fancy way to expose a filesystem directory on the Web. You pick a directory, run a server over it, and that server scans the packages you store in that directory and allows the package managers to optimize for their needs:

  • Give me all the packages for geneva_drive above version X
  • Give me all the dependencies of amazing_drag_and_drop
  • Show me all the binary prebuilt versions of dreadful_glw_bindings

So, the original brief for my robotic lieutenants was pretty simple: examine the RubyGems API for gem servers and produce me the most minimal thing that could work. Quite a bit of Paquette got built this way, with me pointing an agent at a spec and then arguing with the result - the compact index format is exactly the kind of thing where you want a tireless reader and a skeptical reviewer. Once that was in place - and it became possible to actually install a gem from my little server - came the “complications”. After some thought, the design that emerged is relatively simple:

  • You pick a directory where your packages are stored
  • You then set up an authenticator object which wires into whichever token authentication system your Rails app has (or you write your own)
  • And you customize your per-user entitlements and personalization by wrapping your package repository object with wrappers
  • Once that is done - you create a Rack app that is your gem repository and you serve from it

Kinda like this:

class GemServer
  class Authenticator
    def read_permitted?(auth_token:) = Token.where(t: auth_token).exists?
    def write_permitted?(...) = false
    def identity(auth_token:) = Digest::SHA1.hexdigest(auth_token.to_s)
  end

  def self.call(env)
    repo = Paquette::GemServer::DirectoryGemRepository.new(Rails.root.join("storage/packages/gems"))
    authenticator = Authenticator.new
    Paquette::GemServer.new(repo, authenticator:).call(env)
  end
end

# routes.rb
constraints subdomain: "gem" do
  mount GemServer, at: "/"
end

Paquette does not need Rails but it can exist within your Rails app - it’s just Rack, remember? That is the baseline. Now, for some more spice - what makes Paquette unique:

def self.call(env)
  repo = Paquette::GemServer::DirectoryGemRepository.new(Rails.root.join("storage/packages/gems"))
  authenticator = Authenticator.new

  token = Paquette::AuthToken.from_env(env)
  owner = Token.where(t: token).first&.owner

  gate_key = authenticator.identity(auth_token: token)
  restricted_repo = Paquette::GemServer::ReadGatedRepository.new(repo, gate_key:) do |name:, **|
    owner.has_package_access?(name)
  end

  Paquette::GemServer.new(restricted_repo, authenticator:).call(env)
end

Just adding this allows us to only serve a package named name (any version) to a user who has_package_access?. This includes serving the actual package file, but also manages whether the file ends up in the compact index, in the full index, and so on.

Personalization is one more wrapper on the same stack:

personalized_repo = Paquette::GemServer::Personalizer.new(restricted_repo,
  license_key: owner.license_key,
  personalization_key: owner.id,
  files: {"LICENSE" => owner.render_license_text},
  cache_dir: Rails.root.join("tmp/personalized_gems"))

Paquette::GemServer.new(personalized_repo, authenticator:).call(env)

Every gem that leaves the server now carries that customer’s LICENSE file, has the license key stamped into its gemspec metadata, and can have magic comments in the source swapped for whatever you want to say to that particular customer. Which brings us back to the checksum problem.

The same bytes, every time

Bundler, like every package manager worth its salt, does not trust the file it downloads. It trusts the checksum in the index - and if the gem it just fetched hashes to something else, it stops and tells you somebody is tampering with your supply chain. So if I hand you a repacked gem, the index I hand you has to carry the checksum of that gem. And the next time you run bundle install - tomorrow, on a different machine, after I have redeployed the server - the same gem has to come out with the same checksum, or your Gemfile.lock is suddenly lying to you.

The naive version of “repack a gem” breaks this immediately. gem build stamps the current time into the gem. Two repacks of the same gem, a second apart, produce two different files. RubyGems does honour SOURCE_DATE_EPOCH for reproducible builds, but that is an environment variable - process-wide - and a server repacking gems for two customers at once on two threads cannot be setting the same global to two values.

So Paquette’s repacker pins the build time to the original gem’s own date. It takes care to preserve file modes verbatim, and writes the metadata in a stable order. In the end, a repack outputs both the deliverable and the new hash the server can then furnish to Bundler, and the output is repeatable - it will produce the same deliverable every time, provided the inputs stay the same. On any machine, any day of the week.

Once that holds, the rest is bookkeeping. Gems that have nothing to personalize are served straight off the disk, bytes untouched, so their checksums never change in the first place.

The HTTP caching layer gets the same treatment: the ETag of your index has your personalization digest mixed into it, so a cached index for one customer can never satisfy a request from another. npm gets the equivalent - the dist.integrity in the packument computed per licensee.

Needless to say, if you have a company-internal package (or several) you can easily use Paquette for those as well - no need to do the GitHub PAT dances anymore. Paquette also supports package publishing, with proper two-factor auth based on OTP codes. No need for your JFrog install either anymore.

Closing the loop

So, if you decide to buy geneva_drive here is what will happen:

  • You get an account, and the account gets an access token (or several)
  • You put that token in your Gemfile, with gems.stanquette.nl as the source for geneva_drive and geneva_drive_admin
  • The server checks your credential, builds the gem serving stack for you, and hands you an index which only contains the packages and versions you have access to - with your copy of the gems, repacked once and cached from then on

Any “heavier” requests are fast on subsequent access; you should never be waiting for a repack twice.

Feel free to use Paquette for your own projects - we need more libraries worth paying for! Just don’t make it into a JFrog and do not sell Paquette itself.

And, should you decide to buy geneva_drive, enter paquette for a discount - now that you know how it gets delivered to you!