Below is an article I’ve written with Sander Bockting, a CRM Service Line colleague at Avanade, about designing maintainable workflows in Microsoft Dynamics CRM.
Designing Maintainable Workflows in Microsoft Dynamics CRM
The powerful and user-friendly workflow solution in Microsoft Dynamics CRM can be leveraged to easily build relatively straightforward business processes. However, when business processes become more complex, the effort required to translate them into a maintainable workflow increases disproportionately. This two-part article provides practical guidelines for translating complex business processes into maintainable, easily testable workflows.
Helping the business get their complex workflows
While working on a large enterprise Dynamics CRM solution, we were handed seven PowerPoint slides containing nearly one hundred human tasks. The tasks were part of two key business processes. Many of the tasks were to be executed in parallel, some were to be executed conditionally, and others functioned as approval tasks – only to be executed once the preceding tasks were canceled or completed. The processes even contained multiple recursions. Of course, each task had it is unique details, such as the owner to whom it should be assigned and due dates that were dependent on fields in CRM.
It quickly became apparent that simply building a large workflow would not result in a maintainable solution, should it even be possible at all. A more structured approach was required that would be resilient to change and where the workflow execution state could be understood at a glance.
In an attempt to make effective use of our computer science background, we decided to take a more academic approach to formalize complex business processes – written by domain experts – into a model decomposition where each model could be implemented using a CRM workflow. Here we share our methodology and experiences.
Reduced maintainability of larger workflows
Dynamics CRM workflows provide great flexibility and extensibility for automation of system-driven business processes. Conditional checks, waits and branches provide for enough expressiveness to implement most business processes. Custom workflow activities can be leveraged when things get really complicated. These tools are especially applicable to smaller workflows; larger workflows tend to come with challenges in maintainability. Workflows can be considered maintainable when:
- It is possible to quickly grasp its functional intent
This becomes increasingly difficult when depth of the workflow steps increase or when there is recursion involved in the workflow steps. Recursion means steps may need to be looped until a certain condition is met.
- Grasping the current execution state of a workflow (set) does not take much effort
Dynamics CRM provides out-of-the-box functionality to see the current execution state for one workflow. But this does not help the system administrator in great detail when multiple workflows depend on each other.
- Small changes do not impact a large portion of the workflow
Many developers that have built workflows with CRM are familiar with the frustration where you have to delete and rebuild a workflow branch, because the “Insert Before step” did not do the trick in accommodating the required change.
- Long running and complex (multi-staged) workflows can be changed without disrupting operation
Without proper planning ahead, it may be necessary to let the old workflow instances finish while the new instances run in parallel. This is not always acceptable for the business.
To overcome the maintainability issues of larger workflows and to help in the process of implementing complex business processes, we have adopted a six-staged structured methodology:
- Capture the desired functionality of the workflows in a model using a functional language that can be easily understood by the customer and the developers.
- The functional model is used to break up the monolithic large workflow into smaller and more maintainable workflows.
- Create a status record that tracks completion details of the individual workflows. The record shows the total progress of the combined workflow.
- Each of those workflows is described in a technical model.
- Each technical model allows for translation into an actual CRM workflow.
- Create the Starter Workflow to initialize the composite workflow.
A more detailed explanation of each stage is given below.
Stage 1: Design Functional Model
The idea for a workflow frequently arises at non-technical users. Our experience is that these users are perfectly capable of creating a schematic view of their desired business processes. Their schematic views can be used to discuss the flow and act as a starting point for the technical workflow. Among many different customers, we have found that they all came up with comparable notations. We collected them into a uniform set of functional process symbols.
Functional Process Symbols
Functional process symbols describe the business process from a functional point of view. The representation is used as an input for the translation from a business process into a workflow. The symbols should be seen as an accessible notation for the customer to define their processes. Although not obligatory, getting a design based on these symbols may speed up your work in translating it into a technical design.
To increase diagram readability, we chose to let the customer describe details of actions or tasks (for example, start date, priority, subject, etc.) in a separate document. Letters and digits were used as a reference between the two design artifacts.
Figure 1: Legend of functional process symbols
|Start of diagram
||Indicates start of the business process.
||A state indicates a task or action to perform or finish to be able to take the next transition.
||A branch models the paths and their preconditions that could be traversed. Only 1 path can be traversed simultaneously. A branch does not represent an action or task.
||A transition models a possible one-way path from one state to another or to a branch. The transition could hold a guard that models the precondition required to traverse the path. For example, the completion of a task, the passing of a certain amount of time, etc.
|Parallel activity streams start
||This one-to-many fork models multiple activity paths that take place in parallel.
|Parallel activity streams convergence
||This many-to-one fork models the convergence of multiple parallel paths. The outgoing path can only be traversed when all parallel paths are finished.
Using Functional Process Symbols to create schematic views
The customer uses the functional process symbols to create schematic views of their business processes. The most common situations and their compositions are provided below.
|Sequential execution of workflow items.
|Parallel execution of workflow items.
|Repeat a workflow item until a condition is met (e.g. ensure that an end-user entered a valid value in a record).
|The first state is a task or an action to be performed by a person. The second tasks is assigned to a supervising entity, and is meant to make a judgment about the outcome of the first state. If undesirable, the state or action is to be performed again (e.g. useful in an approval flow).
|Repeat a set of workflow items from the beginning each time the outcome of a task or action was not desirable (e.g. a complex approval flow with multiple approvers).
|Wait until a specific point in time is reached.
Stage 2: Box Identification – Identifying child workflows
When a schematic view of the desired business process is complete, the next step is to break up the view into smaller, workable components. The task is to identify the areas that must be modeled as child workflows within the main workflow to encapsulate the entire business process. These areas are called boxes. The procedure of finding them in a business process flow is called box identification. Box identification is characterized by finding the right start and end of a box. Let’s look at the mechanics of this process in more detail, as well as an example.
Finding the start of a box
A box should have one incoming and one outgoing transition. Breaking up the business process into smaller boxes is done by traversing the business process path from the start. Three situations have been identified as good fits for a new box. While traversing the path:
- A branch element is encountered.
- A state is found that has an incoming transition from a state that has not been traversed yet; the transition is a recursive call from a child element.
Finding the end of a box
The end of the box depends on the situation that started the creation of a box.
- For a box that was created after encountering a branch element, the end of it is marked by the first transition that is traversed by all possible paths that started after the branch element.
- For a box identified by a recursive call, its end is identified recursively:
- Include all possible states between the start state of the box and the state making the recursive call to the start state, including the start state and the state making the recursive call.
- If this set of states has any state making a recursive call to a state that is not yet in the box, repeat step A for these states, also including them in the box.
- Repeat this process until there are no more recursive calls found.
The resulting set of states together form the box.
Box identification example
Figure 3: An example of box identification that breaks down one large functional process design into smaller boxes. Each box eventually results in a CRM workflow.
Figure 3 shows a large functional process design. The black box covers the entire business process or Complete Workflow; it is the main workflow and will be the first box (1) (see Figure 4). Within the black box, traverse the path until the branch element is found, indicating the need for a new box (2): Subflow A. The end of Subflow A is found by the first single transition going out of the box.
Figure 4: Breakdown into three boxes
Within Subflow A, observe there is yet another branch element in the business process. The blue box indicates a child workflow (Subflow A’) of Subflow A: box 3. The end of Subflow A’ is also found by identifying the single arrow going out of the box.
Stage 3: Design and Build Status Record Entity
Due to the lack of a native CRM mechanism to have a workflow pause while a child workflow runs, we have devised our own method to track the execution state of a child workflow. We have done this by means of defining a custom entity for the composite workflow, called the status record entity. The idea is to have this status record entity contain a number of datetime fields equal to the number of workflows identified in stage 2 that are to be waited upon. The names of these fields on the status record entity should preferably be descriptive for the child workflow that is to be waited upon for better readability.
After initiating a child workflow, the parent workflow will pause until the status record field corresponding to the child workflow is filled with a datetime.
After execution, the child workflow will set the value of its status record field to the current date and time and then end. This signals the parent workflow to resume duty.
The primary entity of the workflow is linked to the status record by means of a new field preferably hidden from the entities forms. The status record is initiated at the start of the overhead workflow.
It can frequently occur that a parent workflow resume after waiting (because the child workflow has set the date and time in the appropriate field), and see that the result of the child workflow is not satisfactory. When the child workflow is to be run again, the value in the status record field should be emptied before initiating the child workflow.
Note that when a child workflow is a ‘fire and forget’ workflow, it will not need representation in the status record entity.
Stage 4: Create a technical model for each identified box
The previous stages described the identification of boxes that function as separate (child) workflows and the creation of a status record entity that maintains the connection between each of the (child) workflows. Next, each box is translated into a technical model. Just like the Functional Process Symbols are used to create functional models, the Technical Process Symbols are used to create technical blueprints of the workflows.
Technical Process Symbols
Besides functional process symbols – to capture business processes in diagrams interpretable by your customer – a set of technical process symbols is given below. The symbols serve as intermediate language and allow you to translate the functional diagrams into technical diagrams in a more formal way. They provide just the right amount of expressivity to create diagrams for implementable workflows.
Figure 5: Legend of technical process symbols
||Indicates start of workflow.
||Recursive call to current workflow.
||Indicates end of workflow.
|Record Update / Create Operation
||Models a create operation of a data model record (for example, a Task) or an update operation on a data model record.
|Status Record Update
||Models an update operation on the status record related to the primary entity.
||The Composite state models a child workflow. It is a call to a child workflow.
||A branch models the paths and their preconditions that could to be traversed. Only 1 path can be traversed simultaneously. A branch does not represent an action or task.
|Guard on workflow records
||The guard describes the condition that must have been met in order to traverse the path it guards. A guard can consist of multiple conditions joined by logical operators. A guard can also be a mix of workflow record guards and status record guards, however the labels will then be written in blue to show the guard’s dependency on the status record.
|Guard on status record field(s)
||The guard describes the condition on one or multiple fields of the status record that must have been met in order to traverse the path it guards. A guard can consist of multiple conditions joined by logical operators. A guard can also be a mix of workflow record guards and status record guards.
||Parallel paths show actions and states that take place in parallel.
Creating technical models using the symbols
For each identified box, a technical model is created using the Technical Process Symbols. An example is given that shows how a parent workflow (Workflow X) is connected to a child workflow (Workflow Y). With the use of a status record, Workflow X will only continue processing once Workflow Y filled the appropriate status record field with a date.
Figure 6: Workflow X – it depends on a subworkflow Y that should be finished before it can continue. It waits until the appropriate status record field contains data.
Figure 7: Workflow Y stores the current date and time in a status record field when it has successfully reached the end. Workflow X waits until the appropriate status record field contains data.
Stage 5: Translate technical models into CRM workflows using the editor
Translating the technical models created in stage 4 into CRM workflows should be pretty straightforward if a sufficient breakdown of boxes has been made. It is assumed the reader has a good understanding of creating processes in Dynamics CRM 2011. If not, the internet has various tutorials on how to build a workflow.
All child workflows in the composite workflow need to be defined for the same primary entity. When creating the workflows in Dynamics CRM, be sure to uncheck the ‘Start when’ options. Also, make the workflows ‘available to run’ only as ‘child workflow’, unless you are testing or debugging a specific child workflow. In this case, do not forget to manually attach a new Status record. Otherwise you run the risk of either not having a Status record attached yet, or that some datetime fields have already been filled with data.
Translation of the technical symbols is done with the following components.
- The ‘recursive call’ symbol is implemented by the Start Child Workflow step, calling itself as a child workflow.
- The ‘status record update’ symbol is implemented by the Update Record step.
- The ‘record update / create operation’ symbol is implemented by either the Create Record step or the Update Record step.
- If the create operation is used and the outgoing transition of this state has a guard, the ‘record create operation’ is followed by a wait step on the guard.
- The wait step is omitted in case the outgoing transition of the composite state is connected directly to the end point of a ‘parallel paths’ step.
- The ‘composite state’ symbol represents a child workflow and is therefore implemented by a Start Child Workflow step, starting the appropriate child workflow.
- If the outgoing transition of this composite state has a guard, the Start Child Workflow step is followed by a wait step on the guard.
- This wait step is omitted in case the outgoing transition of the composite state is connected directly to the end point of a ‘parallel paths’ step.
- The ‘branch’ symbol is implemented by a Check Condition step.
- The ‘guard on workflow records’ and ‘guard on status record field(s)’ symbols are implemented by a Wait Condition step, waiting on the designated field to fulfill the guard defined on the step.
- The ‘parallel paths’ symbol is implemented by creating all of the steps for the individual paths and then defined a wait step that guards all of the waiting conditions in each path. In case an individual path contains more than one symbol, it should be boxed into a new workflow as described previously.
- The ‘workflow start’ symbol needs no translation.
- The ‘workflow end’ symbol is implemented by the Stop Workflow step. If this is the last step of the workflow, it does not need to be created.
Stage 6: Build Workflow Starter
The sixth and final stage is to build the workflow that starts the composite workflow. Now that all of the workflows have been designed and defined, a starter workflow must be constructed. This starter workflow is the workflow that will be defined to run as the initial workflow for the primary entity.
When run, the starter workflow constructs a new instance of the Status Record entity. After creating the new Status record, the Workflow Starter starts the workflow that is defined for the outer most box of the business model. This, in essence, starts the desired workflow.
In our solution, the starter workflow is also used as a prerequisites test. It tests if the conditions are met that allow the workflow to run. Although not required, it could abstract the actual workflow from the logic that checks whether or not the workflow should be triggered to run. One could even imagine defining one starter workflow for starting one or more workflows from a set of possible workflows.
The introduced workflow design methodology introduces several major advantages. It allows for special constructs that are not possible with large monolithic workflows such as loops. It also allows for easier maintenance when workflows should be changed. Changes are limited to shorter-running workflows, which allows the support team to replace these workflows by new versions without disrupting the long running process. When making changes to child workflows, it is required to have a thorough understanding of the dependencies between the different child workflows, and this is where the technical designs come in play. If the workflows would not have been split up into smaller components, it would have been a lot more difficult to modify the large workflow and several changes may require the removal of entire branches (e.g. when an extra branch condition should be added before a tree of workflow items). The smaller child workflows also improve testing possibilities because smaller sets of workflows can be tested individually and with less associated test conditions.
Each set of workflows requires a new entity in the form of a workflow status record. Although this implies additional work, the workflow status record allows for direct insight workflow execution progress, removing the need for tooling such as BAM (Business Activity Monitoring) when there are no complex requirements in this respect. And when requirements become more complex, the status record can be leveraged to measure workflow execution durations.
The design methodology lowers the threshold to create complex workflows in Dynamics CRM. However, workflows are still complex so it is wise to be cautious to prevent certain problems:
- It is a Microsoft best practice to create one single long workflow instead of child workflows called by a parent workflow when it comes to maximum throughput. If workflow definitions do not change frequently and the workflows are not that complex that they should be split up into separate workflows, it is worth considering one single workflow.
- An important step in the design methodology is decomposing the large functional model into smaller child workflows through the process of box identification. When mistakes are made during this boxing process, it is possible that you end up finding yourself unable to create the actual Dynamics CRM workflows because a recursive call was required at a level where it cannot be implemented through standard Dynamics CRM workflow steps. In that case, all stages starting from stage 2 will have to be redone for the affected workflows.
- To reduce complexity, this methodology ‘forces’ you to create many child workflows. This could mean that you run into the Maximum Workflow Depth (default set at 8), meaning that your workflows will stop when a chain of child workflows is called. When this occurs, increase the Maximum Workflow Depth. This can be done using PowerShell.
This article described a structured approach to translate complex business processes into maintainable workflows. Two modelling notations were introduced that can be used to create a functional model and a technical model. A methodology was given to translate the functional model into a technical model. Finally, this technical model is the basis for the actual Dynamics CRM workflow set implementation. The child workflows have dependencies with each other that are managed through a status record. The status record makes it easy to grasp the current execution state of the workflow by looking which child workflow completed last. This setup ensures more flexibility; a business process change likely affects a (small number of) child workflow(s), minimizing the need to modify a long running workflow with the corresponding maintenance support issues.