In the context of any software application, a workflow (also known as a business process) is a repeatable sequence of steps to fulfill a certain business use case. Typically, a workflow can be a long-running process (from seconds to days) and involve calls to unreliable external systems (such as the payment system) introducing a failure scenario. In the case of microservice, built as a collection of small, independent, and loosely coupled services, this failure can be very hard to handle as workflow steps can span many services.
In this article, we’ll understand how to implement resilient and fault-tolerance workflow in a microservices architecture using Temporal, a durable, reliable, scalable workflow orchestrator engine. We will understand the core concept of Temporal and see code examples of Spring Boot microservices and Temporal.
In this article
What is Temporal?
Temporal is an open-source, workflow-as-code, scalable, and distributed workflow orchestration platform. One of the main benefits of the workflow as code approach is that all dev tools like IDEs, debuggers, compilers, and unit testing frameworks work out of the box.
In Temporal, you can define workflow steps in code using constructs of language-specific client SDK. The client SDKs are available in many languages including Go, Java, Python, PHP, and TypeScript. You can check the full list of supported SDKs at Temporal SDK.
Once the client application submits the workflow execution to Temporal Server, it runs workflow in a distributed manner and handles all failure scenarios such as retries, state management, deadlines, backpressure, etc. Temporal Workers execute workflow and Workflow steps (called Activities).
The Temporal server maintains the state of workflow execution in event sourced database. Because of the replay feature of event sourcing, the Temporal server can continue execution from the point it left in case of an outage in the Temporal Server or Workers.
One important difference between Temporal and other workflow engines, such as Airflow and Netflix Conductor, is that it doesn’t need special executors; workers are part of your application code. Thus, it fits nicely with microservices applications.
If you have used Cadence earlier, you will find many similarities between Temporal and Cadence. In fact, Temporal is a fork of Cadence with some notable differences.
Temporal Architecture
On a high level, Temporal consists of two distinct components – Temporal Server and Temporal Client components (running as part of your application’s deployment). The client applications communicate to Temporal Server using the client SDK via gRPC.
The Temporal Server’s internals and communication protocol are completely hidden from developers. As a result, application developers need to make themselves aware of the APIs of the client SDKs.
The Temporal server primarily consists of a web console, to visualize and troubleshoot workflow execution, a bunch of services written in go, and an event-sourced database.
On the client application side, the main Temporal components are – Workflow, Activities, and Workers.
Temporal Workflow
In Temporal, we define workflow in code (also called Workflow Definition). Temporal Workflow is an instance of workflow definition (executed by a Worker).
When a client application submits the Workflow to the Temporal Server, it becomes the responsibility of the Temporal Server to execute the Workflow and all Activities associated with the Workflow in the appropriate Worker.
Example of a Workflow defined in Java:
public void createOrder(OrderDTO orderDTO) {
paymentActivity.debitPayment(orderDTO);
reserveInventoryActivity.reserveInventory(orderDTO);
shipGoodsActivity.shipGoods(orderDTO);
orderActivity.completeOrder(orderDTO);
}
The above Workflow definition consists of four distinct steps (Activities).
Temporal Activities
An Activity is a function that executes a single, well-defined action (either short or long-running), such as calling another service, transcoding a media file, or transforming data. You can think of an Activity as a distinct step (business action) in a workflow execution.
In the above workflow definition, there are four different Activities – Debit Payment, Reserve Inventory, Ship Goods, and Local Activity completeOrder.
A Local Activity is an Activity that executes in the same process as the Workflow that spawns it. You should use Local Activities for very short-living tasks that do not need flow control, rate limiting, and routing capabilities.
Temporal Worker
A Temporal Worker is responsible for executing Workflows and Activities. A Worker connects to the Temporal server via client SDK and polls the server for the workflow or activities tasks. You can register Workflows and Activities with the Worker using APIs provided by Client SDK.
Use Case and Code Example
The working code example of this article is listed on GitHub. To run the example, clone the repository, and import order-fulfillment-workflow as a project in your favorite IDE as a Gradle project.
In this code example, all microservices and integration with Temporal are implemented in Spring Boot and Temporal Java client SDK.
Code structure
The sample use case covered in this article is order fulfilment workflow in the microservices application (as shown below).
In this use case, once a user checks out goods from the cart, UI calls the Order microservice to create an order. The Order microservice first creates an order in Pending status and then submits the Order fulfilment workflow to Temporal (steps 3 to 6 in the above diagram).
As this article is to show the concept of distributed workflow orchestration, we’ll focus our code on workflow orchestration and not on the actual business logic. Also, we will not cover any failure and compensation scenarios.
Order Fulfillment Workflow Definition
In Temporal, workflows constituents an interface, annotated with @WorkflowInterface and exactly one workflow method annotated with @WorkflowMethod, and its implementation.
@WorkflowInterface
public interface OrderFulfillmentWorkflow {
@WorkflowMethod
void createOrder(OrderDTO orderDTO);
}
Once we have defined the workflow interface, we can implement workflow with appropriate business action (Activities) as:
public class OrderFulfillmentWorkflowImpl implements OrderFulfillmentWorkflow {
//...................
@Override
public void createOrder(OrderDTO orderDTO) {
paymentActivity.debitPayment(orderDTO);
reserveInventoryActivity.reserveInventory(orderDTO);
shipGoodsActivity.shipGoods(orderDTO);
orderActivity.completeOrder(orderDTO);
}
}
As mentioned earlier, an Activity represents a single, well-defined action. Think of an Activity as a step of the business process (as represented by the workflow).
Before using Activities in the Workflow we need to create an Activity stub by calling Workflow.newActivityStub(Class,..). For example, you can implement OrderFulfillmentWorkflowImpl with DebitcPaymentcActivity as:
public class OrderFulfillmentWorkflowImpl implements OrderFulfillmentWorkflow {
private Logger logger = Workflow.getLogger(this.getClass().getName());
//..................
private final ActivityOptions paymentActivityOptions =
ActivityOptions.newBuilder()
.setStartToCloseTimeout(Duration.ofMinutes(1))
.setTaskQueue(TaskQueue.PAYMENT_ACTIVITY_TASK_QUEUE.name())
.setRetryOptions(RetryOptions.newBuilder().setMaximumAttempts(3).build())
.build();
private final DebitPaymentActivity paymentActivity =
Workflow.newActivityStub(DebitPaymentActivity.class, paymentActivityOptions);
@Override
public void createOrder(OrderDTO orderDTO) {
//...
paymentActivity.debitPayment(orderDTO);
//...
}
}
In the above code,
- Start-To-Close Timeout is the maximum time allowed for a single Activity task execution. We should always set this value to be longer than the maximum possible time for the Activity execution.
- A Task Queue is a lightweight, dynamically allocated queue that Worker(s) poll for Tasks (Workflow and Activities). In the above code example, Payment Activity is sent to the PAYMENT_ACTIVITY_TASK_QUEUE, and the Payment Service Worker can subscribe to the task queue as
workerFactory.newWorker(TaskQueue.PAYMENT_ACTIVITY_TASK_QUEUE.name(), workerOptions). - The
RetryOptionsdetermines how an Activity is retried in the case of failure.
In the above use case,
DebitPaymentActivityis called by the Order microservice but its implementation is at the Payment microservice codebase. How can the Order microservice refer to the classes from the Payment microservice? One way to solve this problem is by using an untyped stubWorkflow.newUntypedActivityStubor we can package the Activity definition as a common library that can be referred to by required microservices.
Workflow Constraints
One of the most important aspects of implementing Workflow in Temporal is ensuring it exhibits certain deterministic traits – i.e., making sure that the rerunning same Workflow produces the same result. For example, using random values, local time, etc. causes non-deterministic behavior.
In general, avoid using database calls, random values, time-clocks, external service calls, shared mutable states, etc. in the Workflow implementation.
Spring and Temporal: as Temporal controls the Workflow life cycle, we need to ensure that a Workflow implementation is not created as Singleton Bean and that there is no shared mutable state defined in the Workflow.
Workflow Activities
The Activities in Temporal is defined as an interface annotated with @ActivityInterface. For example, you can define DebitPaymentActivity as:
@ActivityInterface
public interface DebitPaymentActivity {
void debitPayment(OrderDTO orderDTO);
}
Workflow Activity Implementation
In general, Activities perform domain-specific tasks and thus are associated with domain-specific microservices. For example, all payment-related tasks are the responsibility of Payment microservices. So, it’s natural for us to consider the implementation of DebitPaymentActivity as part of the Payment microservice.
In Spring Boot, you can define Activities as any normal singleton bean along with dependency injection of other beans such as Services, and Repositories.
For example, we can implement DebitPaymentActivity as:
public class DebitPaymentActivityImpl implements DebitPaymentActivity {
private final PaymentService paymentService;
private final PaymentRepository paymentRepository;
@Override
public void debitPayment(OrderDTO orderDTO) {
log.info("Processing payment for order {}", orderDTO.getOrderId());
double amount = orderDTO.getQuantity() * orderDTO.getPrice();
//Call external Payment service such as Stripe
var externalPaymentId = paymentService.debit(amount);
//Create domain object
var payment =
Payment.builder()
.externalId(externalPaymentId)
.orderId(orderDTO.getOrderId())
.productId(orderDTO.getProductId())
.amount(amount)
.build();
paymentRepository.save(payment);
}
}
Where DebitPaymentActivity bean is defined as:
@Bean
public DebitPaymentActivity debitPaymentActivity(PaymentJpaRepository paymentJpaRepository) {
return new DebitPaymentActivityImpl(paymentService(), paymentRepository(paymentJpaRepository));
}
Worker Implementation
In Temporal, Workers are responsible for executing Workflow or Activities tasks. Workers poll Task Queue for any tasks and execute tasks once available.
You can register all Activities executed by a Worker as worker.registerActivitiesImplementations(Object..).
Similarly, you can register Workflow with the Worker as
worker.registerWorkflowImplementationTypes(Class<?>...);
A Worker can execute Activities or Workflow or both. Also, you can register more than one Activity or Workflow with a Worker.
As Workers are part of your application code, there is nothing special needed to create a Worker except getting WorkflowClient and registering Activities or Workflow with the Worker.
Below is the code of PaymentWorker that executes DebitPaymentActivity.
public class PaymentWorker {
private final DebitPaymentActivity debitPaymentActivity;
private final WorkflowOrchestratorClient workflowOrchestratorClient;
@PostConstruct
public void createWorker() {
var workflowClient = workflowOrchestratorClient.getWorkflowClient();
var workerFactory = WorkerFactory.newInstance(workflowClient);
var worker =
workerFactory.newWorker(TaskQueue.PAYMENT_ACTIVITY_TASK_QUEUE.name());
worker.registerActivitiesImplementations(debitPaymentActivity);
workerFactory.start();
}
}
And code for workflowOrchestratorClient.getWorkflowClient().
public WorkflowClient getWorkflowClient() {
var workflowServiceStubsOptions =
WorkflowServiceStubsOptions.newBuilder()
.setTarget(applicationProperties.getTarget())
.build();
var workflowServiceStubs = WorkflowServiceStubs.newServiceStubs(workflowServiceStubsOptions);
return WorkflowClient.newInstance(workflowServiceStubs);
}
Where the target is the URL of the Temporal server. If you are running Temporal locally using docker-compose, then you can define the target as target=127.0.0.1:7233.
Unlike Workflow, a Worker can be created as a normal Spring Bean, and there is no restriction about using dependency injection.
Submitting Workflow
To submit workflow execution to the Temporal server, we first need to create a stub of the Workflow by calling workflowClient.newWorkflowStub(Class<T>,..) and call WorkflowClient.start(..) for asynchronous execution of the workflow.
public class WorkflowOrchestratorImpl implements WorkflowOrchestrator {
private final WorkflowOrchestratorClient workflowOrchestratorClient;
private final ApplicationProperties applicationProperties;
@Override
public void createOrder(Order order) {
var orderDTO = map(order);
var workflowClient = workflowOrchestratorClient.getWorkflowClient();
var orderFulfillmentWorkflow =
workflowClient.newWorkflowStub(
OrderFulfillmentWorkflow.class,
WorkflowOptions.newBuilder()
.setWorkflowId(applicationProperties.getWorkflowId() + "-" + orderDTO.getOrderId())
.setTaskQueue(TaskQueue.ORDER_FULFILLMENT_WORKFLOW_TASK_QUEUE.name())
.build());
// Execute Sync
// orderFulfillmentWorkflow.createOrder(orderDTO);
// Async execution
WorkflowClient.start(orderFulfillmentWorkflow::createOrder, orderDTO);
}
Where WorkflowOrchestrator is called from the domain command object OrderCommandImpl.
public class OrderCommandImpl implements OrderCommand {
private final OrderRepository orderRepository;
private final WorkflowOrchestrator workflowOrchestrator;
@Override
public Order createOrder(Order order) {
log.info("Creating order..");
order.setOrderStatus(OrderStatus.PENDING);
var persistedOrder = orderRepository.save(order);
workflowOrchestrator.createOrder(persistedOrder);
return persistedOrder;
}
}
It’s always a good idea to abstract infrastructure-specific code from core business logic by using design practices such as Hexagonal Architecture. This gives us the flexibility to change the Workflow Orchestrator, for example, Temporal with Airflow, without affecting business logic.
The workflow submission sequence diagram looks like this.
You can start the workflow execution by calling the REST API of the Order Service (using Curl or Postman) as:
curl --location --request POST 'localhost:8081/orders' \
--header 'Content-Type: application/json' \
--data-raw '{
"productId": 30979484,
"price": 28.99,
"quantity": 2
}'Temporal with Spring Boot
At the time of writing this article, official support for native Temporal Spring Boot integration is in progress. But, integrating Spring Boot with Temporal is not that difficult, provided we care for certain things.
Firstly, we need to make sure that the Workflow implementation, when registered with Workers as worker.registerWorkflowImplementationTypes (Class<?>..), doesn’t contain any mutable state or Spring dependency (such as any Repository, Service, or any other object managed by spring).
Secondly, use prototype bean when Workflow is registered with the Worker using the Factory method.
worker.addWorkflowImplementationFactory(
WorkflowImplementationOptions options, Class<R> workflowInterface, Func<R> factory) {
There is no restriction in Activities and Worker implementation, i.e. Activities and Workers can be created as default singleton Spring Bean.
Installing Temporal
The fastest (and perhaps easiest) way to install Temporal locally is by using Docker as,
git clone <https://github.com/temporalio/docker-compose.git>
cd docker-compose
docker-compose up
Once Temporal Server is started, you can open Temporal UI by visiting http://localhost:8080/namespaces/default/workflows
Temporal User Interface
Temporal UI provides some important detail about the Workflow and Activities execution.
Visualize all Workflow and its status.
Details of individual Workflow.
Worker and Task Queue.
Details of Activity and its input.
Summary
- Temporal is an open-source, workflow-as-code workflow orchestration platform.
- In Temporal, you can define workflow steps (called Activities) in code using constructs of language-specific client SDK.
- An Activity is a function that executes a single, well-defined action (either short or long-running), such as calling another service, transcoding a media file, or transforming data.
- A Temporal Worker is responsible for executing Workflows and Activities. A Worker connects to the Temporal server via client SDK and polls the server for the workflow or activities tasks.
