Microservices Distributed Transactions

In this article, we are going to discuss Microservices Distributed Transaction Management when performing CRUD operations. As you know that we learned practices and patterns about Microservices Data Design patterns and add them into our design toolbox. And we will use these pattern and practices when designing e-commerce microservice architecture.

By the end of the article, you will learn how to manage distributed transaction in Microservices Architectures with applying Microservices Data Design patterns and principles.

Microservices Distributed Transactions

We will learn how to perform transactional operations across microservices and which patterns we should apply that we learn together.

We saw that querying data across microservices with some patterns and it was not easy stuff. So now we are going to learn how to implement transactional operations across several microservices which is more complex. Its complex because of the working on distributed architecture and working with polyglot database on microservices can cause to network concerns as naturally.

In other words, There is no distributed transaction managements on microservices, we should handle it by manually implementing some patterns and practices.

Normally in monolithic architectures are single database and manage transactions and ACID principles in database level and provide strict consistency, But when it comes to distributed microservices, we are mostly working with eventual consistency.

Lets check the image. First of all let me talk about the use case. Here use case is creating order, or you can say order fulfilment process.

There are 5 microservices that need to interact each other during this order fulfilment use case, and this interactions should be distributed transaction that creates an order. Since every microservices has its own database, they can manage internal transactions with their databases.

But in this create order process, it needs to successfully finished all microservices internal transactions with sequential order.

If any problem happened in one microservices, the process should be rollback for processed microservices.

That means microservices which finished their parts should abort processes and rollback their internal operation into individual databases.

So this create order process is a distributed transaction and in distributed systems we should manage this distributed transactions in order to keep data consistency between several microservices.

Of course we should follow some patterns and principles when implementing distributed transactions. Lets see what patterns can be applied for this problem.

Saga Pattern for Distributed Transactions

The saga design pattern is provide to manage data consistency across microservices in distributed transaction cases. Basically, saga patterns offers to create a set of transactions that update microservices sequentially,
and publish events to trigger the next transaction for the next microservices.

If one of the step is failed, than saga patterns trigger to rollback transactions which is basically do reverse operations with publishing rollback events to previous microservices.

By this way it is manage Distributed Transactions across microservices.
As you know that its used some principles inside of the Saga pattern like publish/subscribe pattern with brokers or API Composition patterns.

The saga pattern provides transaction management with using a sequence of local transactions of microservices. Every microservices has its own database and it can able to manage local transaction in atomic way with strict consistency.

So saga pattern grouping these local transactions and sequentially invoking one by one. Each local transaction updates the database and publishes an event to trigger the next local transaction.

If one of the step is failed, than saga patterns trigger to rollback transactions that are a set of compensating transactions that rollback the changes on previous microservices and restore data consistency.

There are two type of saga implementation ways, These are “choreography” and “orchestration”.

Let me explain Choreography way of Saga pattern.

Choreography Saga Pattern

Choreography provides to coordinate sagas with applying publish-subscribe principles. With choreography, each microservices run its own local transaction and publishes events to message broker system and that trigger local transactions in other microservices.

This way is good for simple workflows if they don’t require too much microservices transaction steps.

But if Saga Workflow steps increase, then it can become confusing and hard to manage transaction between saga microservices. Also Choreography way decouple direct dependency of microservices when managing transactions.

Orchestration Saga Pattern

Another Saga way is Orchestration. Orchestration provides to coordinate sagas with a centralized controller microservice. This centralized controller microservice, orchestrate the saga workflow and invoke to execute local microservices transactions in sequentially.
The orchestrator microservices execute saga transaction and manage them in centralized way and if one of the step is failed, then executes rollback steps with compensating transactions.

Orchestration way is good for complex workflows which includes lots of steps. But this makes single point-of-failure with centralized controller microservices and need implementation of complex steps.

Rollback of Saga Pattern

The below image shows a failed transaction with the Saga pattern.

The Update Inventory operation has failed in the Inventory microservice. So when it failed to one step, The Saga invokes a set of compensating transactions to rollback the inventory operations, cancel the payment and the order, and return the data for each microservice back to a consistent state.

We should careful about when using saga pattern in distributed microservices architecture. If our use case required data consistency across several microservices, and required rollback when one of the step is failed, than we should use Saga pattern.

The Outbox Pattern

Simply, when your API publishes event messages, it doesn’t directly send them. Instead, the messages are persisted in a database table. After that, A job publish events to message broker system in predefined time intervals.

Basically The Outbox Pattern provides to publish events reliably. The idea of this approach is to have an “Outbox” table in the microservice’s database.

In this method, Domain Events are not written directly to a event bus. Instead of that, it is written to a table in the “outbox” role of the service that stores the event in its own database.

However, the critical point here is that the transaction performed before the event and the event written to the outbox table are part of the same transaction.

In example, when a new product is added to the system, the process of adding the product and writing the ProductCreated event to the outbox table is done in the same transaction, ensuring that the event is saved to the database.

The second step is to receive these events written to the outbox table by an independent service and write them to the event bus.

As you can see the above image, Order service perform their use case operations and update their own table and instead of publish an event, it is write another table this event record and this event read from another service and publish and event.

Why we use this Outbox Pattern ?

If you are working with critical data that need to consistent and need to accurate to catch all requests, then its good to use Outbox pattern. If in your case, the database update and sending of the message should be atomic in order to make sure data consistency than its good to use outbox pattern.

For example the order sale transactions, it is already clear how important these data. Because they are about financial business. Thus, the calculations must be correct 100%.

To be able to access this accuracy, we must be sure that our system is not losing any event messages. So the outbox pattern should be applied this kind of cases.

So we should evolve our architecture with applying other Microservices Data Patterns in order to accommodate business adaptations faster time-to-market and handle larger requests.