Event streams aren't event sourcing

September 15th, 2022

Analytics was painful at Dropbox. Partially because the tools were homespun and partially because it never really was a priority. Dropbox wasn’t Facebook: growth was a tactic, not a strategy. Nonetheless, while I was there, I often desperately wanted high visibility into product usage without having to liter the codebase with event triggers. Of course that meant I stumbled into event sourcing.

Event sourcing

Event sourcing is a powerful idea that has never gotten much traction. It’s benefits (audit log, replay, rollback) aren’t compelling at the development stage where all that matters is viability and in the early days of a product, event sourcing really is just misdirection. Early tech decisions matter a lot, but only insofar that they might paint you into a technical corner. Mediocre decisions are ok as long as you’re still able to refactor later because, turns out, computers are really fast. Audit logs can always be tacked on (usually at the HTTP or database interface layers). Rollbacks within transactions are usually all you need. Rolling back and replaying don’t account for side effects. Even something as simple as undelete is almost never actually used because of “non-data side effects”. Event sourcing has “solutions” for this, but writing “gateways” for any side effect only adds to the weight.

I think Abramov’s article You might not need Redux put it best: Local state is fine. In the case of event sourcing: Mutable state is fine. This [SO discussion]https://stackoverflow.com/questions/34385243/why-is-immutability-so-important-or-needed-in-javascript or the follow up post further articulates why immutability might be overblown.

Event streams

Event sourcing might be a bad idea but event streams are not. There are no single player games, only multiplayer cross-play.¹ The same can be said of SaaS apps, where sharing and collaboration are the norm. Even “campaign mode” will often involve your marketing, payment, CRM and support ticketing software, each with their SaaS apps representing their own source of truth. Beyond that, products are increasingly expected to directly integrate with other products: GitHub PRs in your issue tracker, Slack alerts in your CRM. These integrations are a pain to build and the API platforms even harder.

Event stream shouldn’t be considered a source of truth either. They’re not meant to replace the database. But for businesses, there never has been and never will be a single source of truth. Integrations will always be a part of business. You can’t buy integration, nor should you build it too deeply into your main app, as 3rd parties breaking their interface shouldn’t create downtime in your deploy process. Truly great integrations require software and that means choosing the right language, protocol and abstraction for state synchronization across services. The right language is JavaScript. The right protocol is HTTP. The right abstraction is a stream of events.

A few nits:

Mutability is negotiable

Immutability is a lovely concept, a lot like pure functional languages. Immutable should be the default, but mutability should be an option. Regulation is coming to data and things like the right to be forgotten mean that nothing can be truly immutable. If nothing else, it’s just better to be able to make quick fixes. Local state is fine, mutable streams are fine too. “Make easy things easy and the hard things possible.”

Eavesdropping on the interfaces

Local state is fine. Interfaces between systems (particularly the HTTP API) are pretty obvious places to attach event listeners. Listening for POST, PATCH, DELETE requests to your own API seems like a decent way to get started with events.

For instance, accounts.patch seems like a fine event type when making a PATCH fetch.

fetch('api.pipet.io/accounts/1', {
  method: 'PATCH',
  body: JSON.stringify({
    a: 2,
  })
})