Immutable projections

Published on Friday, August 09, 2019

TL,DR; Evolving projections can be difficult. That is why I propose to copy existing implementations and to run them side by side in a separate data store. Hide both implementations behind a feature flag and gradually start routing traffic. This method offers you a safety net, eases trunk based development, and encourages frequent changing.

The first time I took an Event Sourced domain model to production it was tough. In part because I applied CQRS and Event Sourcing as a top level architecture. You probably shouldn’t do that. Unless you know what you’re doing and no supporting subdomains are in scope.

But we were young and foolish. So by the time our understanding of the domain had grown, we had to modify a whole bunch of models that were in production. This isn’t a bad thing but it can be a tough job if you haven’t done it before. Because how do you change a projection without causing problems or downtime for your customers? And how do you know it works as intended? Back then we tested a lot on a staging environment, we timed well and we crossed our fingers #startuplifeyolo.

But please, save your hair and don’t use that approach. Use feature toggles instead to evolve your models with ease.

The problem with evolving projections by simply changing them are easy to sum up:

• How expensive are the rebuilds of this projection? How can you replay it without downtime?
• What if you were wrong? Can you revert easily to the previous version? Or would this entail another costly replay of the entire event log?
• How representative is the data you used for testing on your non-production environments? Are you going to encounter edge-cases when you roll out with your production dataset?
• And what about A/B testing or a canary release, is that even possible when you replace existing projections?

Evolving models with feature toggles

Feature toggles offer a simple approach to add new behavior to the system or to use different models within our system. If the feature toggle is enabled then your application offers an additional behavior or uses a different model to handle the current request. Whether or not a toggle is enabled can be determined by a simple boolean value or by more complex logic based on the incoming request(s).

I’m not going to try to do a better job explaining it than Martin Fowler and his fellows, there is a great article on his website about the matter.

All I want to tell you is the approach that is possible thanks to feature toggles: Immutable Projections It is the projection equivalent of Immutable Infrastructure.

It is very easy to explain: You never alter a projection once it has been taken into production. Or in a more full form: You never alter the structural representation of the state model that powers a projection nor the mechanics that project it once it has been taken into production.

An example

Imagine some e-commerce SaaS platform that allowed customers to place a single order per transaction to achieve the best conversion rates. Earlier this month the company tested a very effective way to sell multiple orders without sacrificing conversion.

This change has implications for the CustomerOrderProjection located in the customer support dashboard. It is effectively a list of CustomerOrder models indexed by orderId and transactionId.

In the current situation the CustomerOrder contains information about both the transaction and the order.

In the new situation this should become a 1-N relationship between Transaction and CustomerOrderV2. Each model should contain their own respective data.

That calls for a new projection

The steps to take are straightforward:

Copy, paste, rename

Rather then rebuilding the whole thing from scratch we will utilize what we have. Copy the entire module of the existing implementation and give each element a version number. You copy CustomerOrderProjection into CustomerOrderProjectionV2. You copy CustomerOrder into CustomerOrderV2. Etc.

Adapt your test and implement

Adapt your tests to the new desired behavior. In our example that means three things:

• The introduction of the Transaction model
• The move of transactionData
• The ability to place multiple orders in a single transaction

Now that our tests are guiding us, it is time to implement our new model and make it pass our tests.

Wire up the new projection

Time to connect the projection to the infrastructure so that we can test it in the dev environment. Keep this in mind:

1. The new version needs its own private copy of the projected data
2. We need the ability to flip the feature toggle once we are certain that the projection is working correctly and has caught up with the live stream of events

Therefore we should only route queries to the new projection if the feature flag is enabled. In a typical setup that means we need to modify our QueryDispatcher to do so.

final class HardCodedQueryDispatcher implements QueryDispatcher {

    private $projectors;
    private $featureToggles;

    function __construct (
        ProjectorFactory $projectors,
        FeatureToggles $featureToggles
    ) {
        $this->projectors = $projectors;
        $this->featureToggles = $featureToggles;
    }

    function dispatch (Query $query): State {

        if ($query instanceof GetOrder)
        {
            if ($this->featureToggles->isEnabled('CustomerOrderV2'))
            {
                return $this->projectors->customerOrderV2()->ask($query);
            }

            return $this->projectors->customerOrder()->ask($query);
        }
        
        // other queries omitted for brevity

        throw SorryUnsupportedQuery::queryUnknownToDispatcher($query);
    }
}

Exploratory testing

Time to respect our QA process: let’s do some exploratory testing. After that it is time to bring this to production to let the new projection catch up with the live stream of events. During this time you may or may not want to monitor the projection depending on the importance of it and its expected replay time.

Once it has caught up with the live stream you can enable the feature for internal users. This allows you to “test in prod”, it doesn’t get more certain than that.

When you are certain it’s time to flip the bit you can. And when you do get support requests poring in you can easily revert. This is the power of immutable projections: it is the best safety net you can get.

What is next?

There is obviously some cleanup to do. This is what I typically schedule at a later point in time. You cannot revert if the code and the data are gone.

Be mindful of the types of changes you have made, use your domain expertise to come up with a reasonable moment to delete old versions. As an example: for a financial report that is used every quarter you should wait until some time after the current quarter.

Reflection

Immutable projections are an easy concept to grasp: You never alter a projection once it has been taken into production. The process is easy, it involves copying the existing implementation and versioning its elements. We alter the projection as we need and push it to production.

The benefits are great:

• Always being able to rollback with the flick of a switch
• No downtime due to projections rebuilding state
• Acceptance testing with production sets
• Trunk based development :-)
• Which all will contribute to a culture where experimentation is the norm

But keep in mind that whenever you use this approach you should first stop and analyze why you need a new version to begin with. Often new versions of projections correspond to problems in the associated write model. Address those too if that is the case.

In case you disagree, or if you have been doing this too, let me know on Twitter.

Or join my workshop in Milan.