Introducing Collie: nine months of DCB event sourcing in Go/Postgres, rounded up into a library

Today I’m releasing Collie, my Go implementation of Dynamic Consistency Boundaries (DCB) event sourcing, with a Postgres event store. It started as a late-night experiment in February. Nine months later it runs a project with tens of millions of events, and I think it’s ready for other people to try.

In my day life at Eurostar, I built the core event store of my current project as a hybrid: part classic aggregates, part DCB. (Collie has nothing to do with that work. It’s a personal side project, built in my own time, but that experience is what inspired it). The more I learned about DCB, the more it felt like it was solving problems instead of creating new ones. Its flexibility lets you simplify complex projects in ways that regular event sourcing always struggled with.

To me, it combines the best of both worlds: the freedom of a CRUD application, with the history and guarantees of event sourcing.

Why I built my own

When a colleague, Jef, introduced me to DCB, I was hooked pretty quickly. We talked about it endlessly and kept sending each other ideas. At some point I wanted to stop talking and put those ideas into practice.

As an experienced Go developer, Go felt like the natural choice. I love clean Go applications, and the DCB interface is clean too: read, decide, append. For storage I wanted a database everyone already knows, one that every cloud provider offers as a managed service. That’s Postgres, which makes Collie usable pretty much anywhere.

More importantly, I didn’t want to vibe code it. I went old school: I designed and tested the interfaces myself. I did use Claude to quickly visualize the ideas I had in my head, but Collie is handcrafted. It’s not one of those projects where nobody has read the code the AI wrote…

And honestly, I enjoyed it. That kind of programming doesn’t happen that often for me anymore.

What Collie is

Classic event sourcing fixes your consistency boundaries at design time: one aggregate, one stream, and a business rule that spans two aggregates needs a saga. DCB gets rid of aggregates and streams. There is one global log, every event carries tags, and every decision names the events it depends on with a query. An append is refused only if a matching event arrived after the decision read the log. The consistency boundary is exactly what the decision read, nothing more.

Collie DCB event sourcing mascotte

That’s also where the name comes from. Instead of fixed fences that keep the sheep together, a border collie decides on the spot which sheep to round up. The mascot completes the picture: a collie herding sheep through a gate, while a gopher ticks them off on a clipboard.

I wrote about the ‘why’ in more depth in Dynamic Consistency Boundaries (DCB) for Event Sourcing. In code, a decision is three steps. Here’s a username that can only be claimed once:

func signUp(ctx context.Context, store collie.Store, username string) error {
	// 1. Read the events this decision depends on.
	taken := usernameTaken(username)
	condition, err := collie.BuildDecisionModel(ctx, store, &taken)
	if err != nil {
		return err
	}
	// 2. Decide.
	if taken.State {
		return fmt.Errorf("username %q is already taken", username)
	}
	// 3. Append, on the condition that nothing changed since step 1.
	event, err := collie.NewEventFrom("UserSignedUp", UserSignedUp{Username: username})
	if err != nil {
		return err
	}
	return store.Append(ctx, []collie.Event{event}, condition)
}

If two requests claim alice at the same time, both pass step 2, but only one append succeeds. The other gets ErrAppendConditionViolated, and when it retries, it sees the winner’s event and is refused in step 2. No locks on aggregates, no sagas.

From February to now

The first version was easy to write. It was also nowhere near enough. It used a global lock, which is of course not exactly fast, and the database wasn’t optimized at all.

So I went down the rabbit hole. Trying to replace the global lock with locks on the consistency boundaries, I ended up deep in the world of auto-increment IDs and the moments transactions actually commit. In the end I settled on a staging approach, which I explain (with numbers) in Can DCB event sourcing be fast and flexible? A Postgres benchmark. Getting the right indexes in place did the rest.

It was never a project I worked on every day. It was more of a slow burner: for days I wouldn’t touch it, but it was always somewhere in the back of my mind.

Then I needed a solid backend for a new project, belgiancompanies.be, and using my own Go implementation was the obvious choice. It now holds tens of millions of events, and that number grows every day. Running it myself showed me what to optimize next, and slowly turned an experiment into something other people could use. That’s why I’m open sourcing it now.

Along the way I also wanted the things a real application needs to work out of the box, like attachments and encryption. These days, handling personal data properly shouldn’t be an afterthought. So those are also part of the first version.

What’s in this release

This release covers the core DCB paradigm, with two store implementations:

  • In-memory, for tests. It enforces the same rules as Postgres, so your tests catch mistakes before production does.
  • Postgres, for the real world. Collie is a library that uses Postgres as its event store directly. And yes, multiple instances can append at the same time while staying consistent. I checked 😉

In the future, when my own projects need it, I want to add a third option: a client-server event store that several applications can easily share.

Creating events is meant to feel natural. You tag an event with struct tags, and the same goes for encryption and attachments:

type UserRegistered struct {
	UserID string `json:"userId" dcb:"tag:user"`
	Name   string `json:"name"   dcb-encrypt:"tag:user"` // encrypted with this user's key
}
type DocumentUploaded struct {
	DocumentID string `json:"documentId" dcb:"tag:document"`
	FileName   string `json:"fileName"`
	Content    []byte `json:"content"    dcb-attachment:"blob"` // stored in a blob store
}

Encryption uses a key per data subject. Deleting that key makes their data permanently unreadable, so you can forget a person without rewriting history (crypto-shredding, for GDPR). Attachments move large fields to a blob store and put them back when you read the event.

On top of the read, decide, append loop I also built a command framework. It handles retries on conflicts, authorization, and work that has to happen outside the loop, such as calling an external service. Working with the store then comes down to implementing a small interface per command:

err := command.Execute(ctx, store, &SignUp{Username: "alice"}, command.WithMaxAttempts(3))

The rest, including projections for the read side, is in the README.

Is this thing production ready?

I run it in production on a side project, but that doesn’t automatically make it production ready. I’m quite confident it works well, but let’s talk again in a year 😉

In the meantime I’ll keep improving it. That’s also why I don’t promise a stable interface yet: I’m pretty sure I’ll introduce breaking changes whenever I find a way to make things better. The optional packages (reactor, envelope, encryption and attachments) are younger than the core, so they’re the most likely to change.

Thanks

Thank you to everyone who challenged my ideas and reviewed my code. Special thanks to Jef Wauters, for all the long conversations about DCB and all the knowledge sharing.

I hope some of you find Collie useful and give it a try:

go get github.com/DenBeke/collie

I’m looking forward to your feedback, issues and ideas on GitHub. Happy herding!

Leave a comment