Julik Tarkhanov

Batching the billing ledger (but with less)

Batching the Billing Ledger: Small Trade-Offs with High Impact is a great article about “systems assembly”, and microbilling is part and parcel of nearly any web app today, certainly when your users call into LLM models. Gotta keep’em tokens counted!

Since the start of the agentic era, I’ve been curious about “how much software” we need for workloads like this. After all, my doctrine is less stuff and this looked like a use case too perfect to pass on. Thank you, Vicente, for the pitch-perfect nerdsnipe!

So, here’s bab - but I am getting ahead of myself. First, we need the parts.

🧭 I am currently available for contract work. Hire me to help make your Rails app better!

The work with software is sometimes similar to being a crazy village inventor in his garage. You try to mash pieces together until they fit, and - sometimes - you need to concoct your own, for nothing ready-made can fit the mold.

Vicente’s system includes the following key components:

  • Ledger mutations packaged into protobufs - a serialization format used (and designed) by Google
  • Redpanda - a Kafka-compatible broker - for forwarding the ledger mutations, with WaterDrop producing them and Karafka consuming them in batches
  • DuckDB for writing every sealed batch out as Parquet, through its appender and a single COPY
  • Parquet as file format for the exported segments
  • S3-compatible output sink (MinIO when running locally)

Up until fairly recently, at least 2 - and, by some accounts, 3 components of this system would be painful for a Rubyist. There was hardly anything to write Parquet decently. Protobufs would require a Google binary gem, that would usually be not up-to-date on the latest VM features and niceties. Kafka bindings would need to be… sophisticated. Something like Maciej’s karafka - which Vicente uses, and which is pretty much the default game in town if you want to do Kafka from Ruby seriously.

But - nevertheless - the article piqued my interest. Could we do the same without all of this stuff? Why do I need DuckDB, a distributed log broker (be it Kafka on the JVM or Redpanda in C++), and a whole framework to talk to the said broker - all for the sake of pushing messages down the line? And why would I need to have google-protobuf just to decode a pack of bytes coming down the wire?

Assembling the puzzle: Parquet and herringbone

As luck would have it, my recent pursuits have been directed at exactly the bits of that little puzzle. First came herringbone. It is built under a very simple maxim:

  • Parquet is just a convoluted, overly-complected™ binary encoding (Thrift) supplemented with modern compression algorithms and a few pre-aggregations to make the files easier to use as SELECT sources
  • Implementing an easy-to-use, feature-complete Parquet library is not a question of Ruby being a bad language - it is a question of will. Implementing that complected stuff is just difficult, painful, and takes a lot of time.

Luckily, these days we have the robobrains of Claude and ChatGPT to carry the burden of complected BS. If you know what you want, realizing things - that used to be the issue of will - becomes the issue of tokens. Since I do not get over my limits all that frequently, one of the things I have always wanted to have in Ruby was a decent library for Parquet files. Decent, in this instance, implying that:

  • It would not require some baroque set of C++ headers that only works on Linux, on good days (hello GObject)
  • It would not require a whole additional compiler stack
  • It would interop well with other parts of the Ruby stack, such as IO-ish objects that can be read from and written to
  • It would not copy the extensive, monumental chunks of C++ to implement the complected serialization of Thrift
  • It would work perfectly with all the parts of the Ruby concurrency/parallelism story: Fibers, Threads and Ractors alike!

If, just a few years back, I had to crawl back into my cave and take my peace with exporting GZip-compressed CSVs, these days I could just wish it into existence. Mind you: it’s not a one-shot vibe project. You need to art-direct. You need to think what the API should look like. Where to make concessions and which subset of features to support - Parquet, with all its extensions, is vast.

Thus - came herringbone. It turned out great: I believe this is about how a Parquet library made for a Rubyist should be, to the dot. All the standard facilities are there, it is Ractor-safe, it doesn’t need you to install any binary dependencies (ZSTD and Brotli come from optional gems, if you want them), it integrates well with YJIT and Ractors, and the whole gem is a ~240KB download of plain Ruby.

The first missing bit was, thus, in place.

More puzzle pieces: protobufs and protostar

From my work at Cheddar I have retained a mixed feeling towards protobufs. As a format they are passable and quite effective. While we steered well clear of grpc and the scandalous disaster that is its Ruby rendition, we did partake in Twirp - and even did some decent work making it play nice with Rails. They also use complected encoding, but - more than anything else - they force you into the world of extra tooling. That tooling, for a Rubyist, is usually two shotgun-sized appendages: the google-protobuf gem and protoc, the venerable Protobufs compiler.

protoc is a C++ monstrosity which you have to install separately, which is meant to transform .proto files into polyglot source files in a myriad of languages. Its Ruby output is so-so - it doesn’t have type annotations, and its format changes from release to release of protoc. But - for all its warts - it is workable.

google-protobuf, however, is a different beast. Every Protobuf message you decode or create with it is a handle to a native object. Is it properly write-barrier-protected for the MRI GC? God knows. Can you look inside it? Only as far as #inspect will print - there are no ivars, the fields live in C memory. Can you marshal it, for example - if you want to store it in caches? Also no.

While, in practical terms, a protobuf message is just a bag of bytes - a binary String in Ruby parlance - working with them had all the warts of very special native object handles without there being any reason whatsoever for those warts to be present, at all. A Proto message should be just a Ruby object, with ivars and accessors - that you can marshal, serialize, store in caches, cross-reference between threads - and, in general, it presents zero merit to be considered as special as google-protobuf makes it.

Moreover, these “native handle” messages have other challenges too. You want to find out what fields your protobuf definition has, from an instance of the message? Off you go to the separate descriptor API - the object itself will not tell you much. You want to create a jbuilder like facility for filling those messages in? Good luck with that, I had to maintain such a contraption and it was anything but pleasant.

So, the next mission is fairly simple: let’s have protobufs, but without treating Ruby as a third-rate language and runtime. It could be slower (due to non-native encoding and decoding), but it has to feel nice:

  • You should not need a compiled compilations compiler written in C++ to turn one piece of text (.proto) into another (.rb)
  • You should not have to look for workarounds to have proper introspection, proper reflection and proper streaming builders - as protobufs are quite streamable if you don’t do stupid stuff
  • You should have facilities for cheap partial reading if you get a message and you only need to read a couple of fields
  • You should not need a massive gem with native shared libraries just to decode some bytes on the wire

Hence - protostar came to be. Since Google forces you to use google-protobuf for every single little thing they ship for Ruby (like working with their cloud SDKs) one of the components is a shim that replaces the standard google-protobuf with something more decent, at a fraction of the weight and complexity. It installs as google-protobuf straight from git, and the gems that pin an older version of it get taken care of by the new Bundler 4.1 override - still in beta.

See how pieces are coming together? But we are not done yet.

The fabric: Kino and Ractors

Just a year ago, a person saying that you should be able to load all the cores of your machine with one Ruby program would be laughed out of the room. Of course you need half of Apache Commons big-data stuff, and some Scala, and a modern JVM, and Kafka and some Go to get even close to the purported “proper performance”.

Not so now. Ractors have made some remarkable progress in recent years, and even though they can’t run Rails yet - they can be used to design exactly the type of a nimble system we see here. We just have to hold them right. Namely:

  • We need a proper app server, which supports Ractors from the get go
  • We need all the participating libraries to be Ractor-friendly

Both protostar and herringbone are Ractor-enabled, so the missing bit is the server. The server thus. And as it happens, my friend Yaroslav has recently produced exactly that - a Ractor-native Ruby web server, called kino. The network side of it is Rust (tokio and hyper), and your Rack app runs in parallel Ractors behind it.

If I want to pump data fast enough - faster even - I would want proper parallelism, and the only carrier fabric for this - at the moment - are Ractors. And since the other libraries are in place, and we are not held back by 50 layers of C++ template metaprogramming and a few dozens of staff engineers at the Big G - why don’t we make the thing?

Dispensing with Kafka

Kafka is a marvelous piece of tech. To be honest, it is the only piece of the cambrian explosion of “Apache big data stuff” systems I consider genuinely great - and unique. Its main feature, for all intents and purposes, is durability - if you hold it just right (acks=all), you know that your append will gawddamn be on more than one machine before you get the ack. Not necessarily on disk, mind you: by default Kafka leaves the fsync to the OS and relies on replication instead. Redpanda, which Vicente uses, does fsync before it acks.

But a real Kafka comes at a price. Several servers, each with a huge JVM process to run. A controller quorum to keep them in check (that used to be ZooKeeper, until Kafka 4.0 went KRaft-only). A huge vocabulary of things to learn - topics? quorums? at-most-once receive? Are these really necessary if we are looking for an append-only ledger of mutations?

Here be tradeoffs. If whatever processing you place downstream from your billing event ingestion turns out faulty - and fails - you do want to preserve a stream you can replay somewhere. Luckily, there are alternatives these days which are - to describe it in a simplified manner - Kafka, but smol.

Such as...

Ruby-native, in the database you already have. No new server at all:

  • Rails Event Store - append-only streams with linearized writes on top of your ActiveRecord database, subscriptions and an event browser. Built for event sourcing, but a ledger of mutations is exactly that. Actively maintained.
  • Message DB - the Postgres message store extracted from Eventide: streams, categories, consumer groups and pub/sub, in plain Postgres functions. The Ruby gem has not seen a release since 2022, but it is the kind of thing that is done.
  • pgmq-ruby - by Maciej himself, a driver for the PGMQ Postgres extension. Mind you, this one is a queue, not a log: a message is gone once acked, so no replaying from an offset.
  • litequeue - the same idea on SQLite, by Moe of litestack fame. Also a queue, and quiet since 2023.

Not Kafka, but with a proper Ruby client:

  • NATS JetStream - persistent streams with replay and pull or push consumers, inside the single Go binary of the NATS server. nats-pure is, as the name says, pure Ruby, and supports JetStream in full.
  • Redis or Valkey Streams - XADD, consumer groups with XREADGROUP, all through redis-rb. Chances are you already run one. Retention is bound by RAM, though, and partitioning is up to you.
  • RabbitMQ streams and LavinMQ streams (the latter written in Crystal - a Ruby cousin!) - append-only queues you can replay with x-stream-offset, over plain AMQP 0-9-1. So Bunny or amqp-client work today, even though Bunny will never speak the native stream protocol.
  • Durable Streams - ElectricSQL’s protocol for append-only, offset-resumable streams over plain HTTP, with a young Ruby client.

Speaks the Kafka protocol, so karafka just works (it rides on librdkafka, which these support):

  • Redpanda - Kafka in one C++ binary, no JVM, and it fsyncs before it acks. This is what Vicente uses.
  • Nisshi (née Tansu) - a stateless Kafka-compatible broker in Rust which keeps the actual data in Postgres, SQLite or S3 - and can even write Parquet or Iceberg out by itself. I did not try karafka against this one.
  • AutoMQ, WarpStream and Bufstream - “diskless” Kafkas which put the log on object storage. Not exactly smol to run, but no disks to babysit. WarpStream belongs to Confluent now (and so to IBM).

No Ruby client yet: Apache Iggy (Rust, with SDKs for eight other languages and an open issue asking for Ruby), Fluvio (Rust) and S2 (streams as a serverless API, with a self-hostable s2-lite).

I do, however, believe that if you want to pursue the doctrine of less stuff - and this doctrine has served me well - Kafka is not the system to reach for. And, since we are working with a system having some degree of academic experimentation, we will replace it with something much smaller and simpler. Which will be this:

  • You insert a mutation into the log by POSTing to an endpoint - one or more messages. Just a protobuf in the HTTP body, like Twirp. A confirmed 2xx request counts as ack’d.
  • We will be storing the messages in some form of WAL. It can be a SQLite database (it has a native WAL, after all), or - the horrors - just an append-only file

This will remove a huge piece of infrastructure we would need to maintain, update and manage (or pay for). Yes, it will be less reliable than a Kafka cluster - but what we lose in reliability we get back in simplicity. And you can always put a restic or other backup over your directory of .wal files - or use SQLite and one of its many replicating filesystem drivers.

Sketching the system

Thus, on a train ride somewhere in deep Lincolnshire, I tasked my agent with cobbling the puzzle pieces together. The mission and the ask: can we do 10K messages per second, on that stack, without any native code except for zstd compression (which is what Vicente’s DuckDB COPY compresses with as well) and Kino’s Rust network front? And what can we use in lieu of Kafka to complete the dirty deed? The other constraints were quite simple:

  • All received events properly decoded and collated into a single set of second-length segments
  • Same Parquet file per second output from the system
  • For simplicity - no S3 upload yet, though herringbone is capable of doing those without any tempfiles at all (bonus points to GCP Ruby team for never shipping a proper SDK stub for streaming chunked uploads out of the box: their love for the language has, yet again, been noted. gcs_put fixes that)
  • Use Ractors to maximum effect, fanning in: several Ractors to accept data, and one to collate the received billing events into Parquet and compress them.

And, needless to say: why 10K? Can we do 20K? 50?..

Turn on the cannon

So, here is the architecture. TL;DR: one Ruby process and a directory.

flowchart LR
    H["bin/hammer<br/>POST /events"] -->|"length-delimited<br/>UsageEvents"| K["Kino front-end<br/>Rust: tokio + hyper"]
    subgraph P["one Ruby process"]
        K --> W1["worker Ractor 1<br/>protostar: decode, validate"]
        K --> W2["worker Ractor 2"]
        K --> WN["worker Ractor N"]
        W1 -->|"append, fdatasync"| L1[("wal/w1/S.wal")]
        W2 --> L2[("wal/w2/S.wal")]
        WN --> LN[("wal/wN/S.wal")]
        W1 -.->|"sealed through S"| C["cutter Ractor<br/>herringbone"]
        W2 -.-> C
        WN -.-> C
        L1 --> C
        L2 --> C
        LN --> C
    end
    C -->|"tmp, fsync, rename"| PQ[("ingested_on=.../S.parquet")]

Kino’s Rust front-end owns the sockets and hands every request to one of the worker Ractors. A worker decodes every event in the body with protostar - one broken event and the whole request gets a 422 - and then appends the body, as is, to its own WAL. The WAL is the dumbest thing that could possibly work: one append-only file per worker per wall-clock second, with the length, a CRC32 and the ingestion time in front of every record. Then comes an fdatasync, and only then do you get your 202. At about 34µs a pop on my laptop, the fsync is the cheap part. SQLite did not make the cut - a file and an fdatasync are all the WAL we need.

The clean cut-off is the only tricky bit. Every worker runs a little ticker thread which, every 100ms, closes the previous second’s file and tells the cutter Ractor “I am sealed through second S”. The ticker and the appends hold the same mutex, so once a worker has said it is done with a second, nothing can sneak into it anymore - not even when the wall clock steps back. Once all the workers are past S, the cutter reads all of their WALs for S, sorts the events by team, and writes one ZSTD-compressed Parquet file with herringbone: into a temp file first, then fsync, rename, fsync the directory, and only then do the WALs get deleted. If the process gets kill -9‘d, the next boot finds the WALs that were left behind and cuts them the same way, dropping a torn last record by its length or CRC.

Parquet and proto schema, of course, are perfectly shareable between Ractors.

What are the numbers like? Well, the 10K events per second is achieved with no effort at all: a p99 of 2.6ms with 50 events per request, and even with one event per request - so 10K requests and 10K fsyncs every second - the p50 is 0.17ms. Better: on my laptop with both the hammering script and the solution running at the same time (cardinal sin of benchmarking) I could push it to 80K events per second, with not a single request rejected and every acked event in exactly one Parquet file. A kill -9 in the middle of a run didn’t lose any acked events either. The files come out at 21.2 bytes per event with event_id stored as a Parquet UUID, against the 22.1 Vicente measured with DuckDB.

80K is where we reach the limits of the cutter: about 0.8 seconds to cut one second of 80K events (with YJIT). The bottleneck is the Parquet output: of those 0.8 seconds, decoding takes about 140ms and the Parquet writing about 540ms. That writing has to be serialized if we want to combine all of our output into those neat per-second packages.

We can, of course, remove that bottleneck and burst our Parquet writing over a few Ractors in parallel, just using a modulo. But I was more interested in compaction - instead of pushing a Parquet file per second we can use the wonders of Herringbone.combine to splice the 60 seconds of data into a single minutely file. This is at a marginal extra cost with an extra Ractor to do the work, provided you have enough cores - and makes for 60x reduction in the S3 objects you will be producing.

Minutely combines

The per-second files now go to a landing directory first. Once the cutter has cut all 60 seconds of a minute, it nudges a third kind of Ractor - the compactor - which glues those 60 files together with Herringbone.combine into a single ingested_on=.../<minute>.parquet. combine is designed specifically to exploit the intrinsic features of the file format - where Parquet feels overengineered, it sometimes compensates in features. Combining does not decode anything: the column chunks get copied byte for byte, statistics and all, and only the offsets in the footer get rewritten. Every second simply becomes a row group of the minute file.

flowchart LR
    C["cutter Ractor"] -->|"one file per second"| LD[("landing/S.parquet")]
    C -.->|"cut through S"| M["compactor Ractor<br/>Herringbone.combine"]
    LD --> M
    M -->|"tmp, fsync, rename"| PQ[("parquet/M.parquet")]

The compactor sticks to two rules. It never combines a partial minute - so if a minute file exists, it holds every one of its seconds, and per-second files still lying next to it after a crash can be thrown away safely. And it counts: the rows in the minute file must add up to the rows of its inputs before a single landing file gets deleted. On shutdown, an unfinished minute just waits in landing for the next run. Whoever reads the parquet directory only ever sees whole minutes, and each of them appears there with one rename.

The cost really is marginal. A full minute at 80K events per second - 4.8 million events - gets combined in about 0.6 seconds, once a minute, in a Ractor of its own. Ingestion did not notice, and the files even got a smidge smaller (20.9 bytes per event), since 60 footers become one.

Could you just go ahead and find me that cost center?..

Billing queries are rarely “everything”. Mostly they are “everything for this one team, this month”. team_id already was dictionary-encoded - which is Parquet’s version of an enum - and the events in every second are sorted by it. But a second of 10K events fit into a single 20K-row page, so there was nothing for a reader to skip, and asking a minute file for one team meant reading all of it.

Two writer options fix that. Pages of 1,000 rows let the page index narrow a team down to about one page per second, and a bloom filter on team_id lets a reader skip the seconds the team was not in at all. combine copies both into the minute files untouched, and DuckDB, Spark, Arrow and herringbone’s own where: all know how to use them:

WRITER_OPTIONS = Ractor.make_shareable({compression: :zstd, page_rows: 1000, bloom_filters: ["team_id"]})

Measured on a minute file of 4.8M events (80K events per second, 500 teams):

  Before Now
Lookup of an active team reads the whole file 1.4% of rows, 146ms
Lookup of a team with no events that minute reads the whole file 0 rows, 0ms
Cutting one second ~820ms ~710ms
Combining one minute 634ms 908ms
Bytes per event 20.9 21.3

At 10K events per second the same lookup goes from 365ms to 39ms. And it makes cutting a bit faster, probably because smaller pages are cheaper to encode.

If team lookups were all we ever did, we could go further and re-sort every minute by team, bringing a lookup down to about 0.3% of the rows. That means decoding and rewriting every row once a minute, though - fine at 10K events per second, heavy at 80K. Small pages and a bloom filter get us most of the way for next to nothing.

And the bill please

Vicente’s article is, at its heart, a bill: what does it cost to keep up with 10K events per second? So let’s run ours - in us-east-1, at on-demand prices, for 10K events per second around the clock, and with the same 30 days of retention he modeled.

The setup:

  • Two VMs in two availability zones, behind a Network Load Balancer. Either of them has to carry the full 10K alone while the other one is gone.
  • Local NVMe, not network disks. Every request waits for an fdatasync, and on a local NVMe drive that takes tens of microseconds - no burst credits to run out of, no network round trip. That means a d instance: a c8gd.2xlarge has 8 Graviton4 cores (on Graviton a vCPU is a whole core, not a hyperthread), 16 GiB of RAM and 475 GB of NVMe.1 8 cores is enough for 6 worker Ractors, the cutter and the compactor. I have not benchmarked it on Graviton, though - for headroom towards the 80K my laptop did, the 4xlarge doubles everything, the price included.
  • Only minute files go to S3, one per VM per minute - so two objects a minute, and the object key needs the name of the host in it (a small change we haven’t made yet, just like the upload itself). The uploads go through an S3 gateway endpoint, which is free - unlike a NAT gateway, which charges for every byte passing through it.
  • A replicated bucket. S3 Standard already keeps every object across at least three availability zones of the region, so “replicated within the region” is what you get by default. On top of that, Same-Region Replication into a second bucket - ideally in another account - protects you from the fat finger and the leaked key. It doubles the storage and the PUTs; moving the bytes between buckets in the same region is free.2
  • A backup of the directory. Instance store goes away when the instance stops or dies, and it can’t be snapshotted. AWS’s own advice is to copy the data onto EBS or into S3.3 Our directory only ever holds what hasn’t made it to S3 yet - a minute or two of events - so we mirror it every second onto a small gp3 EBS volume (gp3 gives you a flat 3,000 IOPS, there is no burst bucket to drain), and let Data Lifecycle Manager snapshot that volume every hour. Running restic into S3 every few seconds would work too, but every run costs a handful of PUTs - and getting rid of PUTs is the whole point of the exercise.

That last one is a trade-off worth spelling out. The 202 still means “fsynced”, but onto the local NVMe. Should a VM die, the events acked in the last second before the mirror caught up can go with it. If that second matters to you, put the WAL onto the EBS volume itself and pay a network round trip for every request - at 50 events per request that is perfectly fine.

At about 89 bytes per event on the wire, 10K events per second is 3.4 GB an hour through the load balancer. Here is the month:

  Monthly
2 × c8gd.2xlarge (8 Graviton4 cores, 475 GB NVMe each), $0.392/hour $572.17
2 × 20 GB gp3 root volumes $3.20
2 × 50 GB gp3 volumes for the mirror of the directory $8.00
Hourly snapshots of the mirrors, a day’s worth kept ~$2
Network Load Balancer, $0.0225/hour plus ~3.4 capacity units4 $31.15
S3 Standard, 30 days of minute files (514 GiB at 21.3 bytes per event) $11.83
86,400 PUTs (2 VMs × 1 per minute) $0.43
Same-Region Replication: the second copy and its PUTs $12.26
Total ~$641

That is about 2.5 cents per million events.

Drilling down with options:

Nine tenths of that bill is two computers, and a Savings Plan or reserved instances would take a good bite out of that line. The S3 part - which is what Vicente’s article is about - comes to $25 a month, replica included. His batches of 10,000 events or 5 seconds come out at one object per second at this rate, so 2.59 million PUTs a month; minute files from two VMs make 86,400. And for the compute, keep in mind what it replaces: a replicated Redpanda cluster wants at least three brokers of its own, before a single consumer has written a single byte to S3.

A few things that would move the numbers. If your producers send one event per request, the load balancer processes about three times as many bytes. If they sit in another availability zone, cross-AZ traffic costs $0.01 per GB in each direction - for 2.4 TB a month, that’s real money, so keep them close. And if you keep the files longer than 30 days, S3 grows by about $24 a month (both copies) for every month you keep.

However, if we intend to store on S3 - and AWS is the picked platform from the start - we can partake in some proprietary platform fun instead. Firehose can be used to stand in for our Kafka and provide a durable log. It cannot be read from - it has no consumer API, it only pushes. But one of the places it can push to is an HTTPS endpoint: it buffers whatever producers send it, then POSTs the records to you in batches (base64 inside JSON - Bab would need a small endpoint for that), and only lets go of a batch once it gets a 200 back.5

You would also need a dedicated public IP for your Bab endpoints, due to intrinsic Firehose limitations.6

If the endpoint is down, Firehose keeps retrying for up to two hours, and after that drops the batch into an S3 backup bucket for you to replay.7 So Firehose becomes the durable, multi-AZ front door, and one VM will do as the POST destination. If it dies, we start another one, and Firehose redelivers whatever it never got a 200 for. Bab still acks only after its fdatasync, and we keep the EBS mirror for what was acked but hasn’t reached S3 yet.

The catch is in Firehose’s own meter: it bills every record rounded up to 5 KB.8 Send it our 89-byte events one by one, and you pay for 5 KB each - 130 TB a month, $3,758. Firehose has a per-event tax of its own, just like the S3 PUTs in Vicente’s article. Put 50 events into one record - which is exactly what a body of delimited protobufs already is - and it comes down to 2.6 TB, and $75:

  Monthly
Firehose Direct PUT, 50 events per record (2,592 GB billed at $0.029) $75.17
1 × c8gd.2xlarge $286.09
20 GB gp3 root volume, 50 GB gp3 mirror and its snapshots ~$6.60
Public IPv4 address $3.65
S3 Standard, 30 days of minute files $11.83
43,200 PUTs (1 VM × 1 per minute) $0.22
Same-Region Replication $12.05
Total ~$396

That is $245 a month less - nearly all of it the second VM and the load balancer - and the “what if the box dies” problem moves over to AWS. It does have a price beyond the $75. Firehose buffers before it delivers, so events reach Bab in bursts, seconds to minutes late depending on the buffering you configure, and the “ingestion second” of a file becomes the second Firehose delivered it in (occurred_at still carries the producer’s clock). AWS also says that additional data transfer charges can apply to the delivery; for a VM in the same region I would not expect any, but check. And if you would rather pull than be pushed: Kinesis Data Streams in provisioned mode, with the same 50 events per record, is about $30 a month for two shards and their PUT units,9 with 24 hours of replay thrown in - but then Bab has to become a consumer which tracks its own position in the stream, and the KCL for Ruby runs on top of a Java daemon.10 It’s one of those horrid DynamoDB-era contraptions, from the same department of “we hate Ruby” engineering that gifted us with gRPC, just from a different provider.

Why even bother?

Easy. First: you use a language you know and understand, and have full command over. Second: your only installation for your infra is a bundle install. Third: you don’t have nearly as much stuff to manage, to update, to learn (or to have your agent operate) and control. Deployable in one command.

And, most importantly: we, yet again, prove to people that nobody ever does that with Ruby has become a bullshit excuse. Enough. Of course we can do it. We should. Now is the time.

  1. On-demand price and specs from Vantage and cloudprice.net, the 4xlarge from Economize. These are third-party trackers, so check them against AWS’s own price list before you budget. ↩

  2. There is no charge for moving data between buckets in the same region; what replication does cost you is the second copy and its PUTs. ↩

  3. The AWS Knowledge Center on backing up an instance store volume, and the EC2 best practices on keeping only temporary data on instance store. ↩

  4. Elastic Load Balancing pricing, with the US East figures from the breakdowns by Cloudchipr and Spendark. One NLB capacity unit covers 1 GB of processed bytes an hour. ↩

  5. Firehose’s HTTP endpoint request and response specification: the records arrive base64-encoded in a JSON body, and only a 200 counts as delivered. ↩

  6. In AWS’s own words: “Currently Amazon Data Firehose does NOT support data delivery to HTTP endpoints in a VPC” (Controlling access with Amazon Data Firehose). The endpoint has to be public HTTPS - the IP behind it does not have to be a fixed one, though. ↩

  7. Handle data delivery failures in the Firehose documentation, and the S3 backup mode setting for what lands in the backup bucket. ↩

  8. Amazon Data Firehose pricing: Direct PUT is $0.029 per GB for the first 500 TB a month, metered in 5 KB increments. The FAQ adds that the rounding is per record, not per API call. ↩

  9. Kinesis Data Streams pricing: in provisioned mode, $0.015 per shard-hour - a shard takes 1 MB or 1,000 records a second - plus $0.014 per million 25 KB PUT payload units. ↩

  10. “Using the Amazon KCL for Ruby package requires the MultiLangDaemon which is provided by the Amazon KCL for Java”, says the KCL for Ruby README. ↩