Peter HarrisonThe Gravity Vision
The Gravity Vision brings together from various sources of inspiration for a complete picture to a business workflow automation system. It takes onboard the lessons learnt from use of other business workflow tools that failed to meet business needs.
The essential purpose of a workflow automation system is to manage business processes in the real world that may take days or weeks to complete, and that may involve complex human interactions and perhaps real world manufacturing, packaging and delivery.
Computer systems are traditionally good at clean problems that can be broken down into clear sequential programs that can be followed, but terrible at processes which involve a high degree of uncertainty, complexity and failures that take place over extended periods. Batch processing 10,000 phone calls is easy, connecting a single customer to a phone system may be complex and involve physical installations and supply of physical goods.
Gravity has been developed to handle this class of complex real world process. It has not been written as a solution that will only solve a specific kind of specialty problem. For example, although it has been utilized for provisioning of phone services there is nothing specific in Gravity which couples it with phone provisioning. It is a true universal business process workflow automation system.
Gravity as a Kanban Board
One of the inspirations for Gravity was the kanban board approach. Originally kanban was a system of cards which move across a physical board as each process is completed. It helped implement just in time manufacture. The success of the kanban approach lead to it being introduced to the software development field. From there the kanban approach was incorporated into software project management software.
One such system was AgileZen, which was initially used to manage a provisioning system. The AgileZen system became an initial proof of concept for the approach to automation that would later be used in Gravity.
This AgileZen system had cards with a few standard fields such as owner, estimated time and job description, but not specific fields for the project domain. Essentially a card in a kanban system is a free form text entry. Gravity borrowed the concepts of boards and cards from the kanban approach. Cards are shifted between phases as a business process progresses. A card is only ever in one phase, which makes visualization of the board easy to understand for users.
Gravity has not followed the kanban approach for the detail of cards however. Most business processes require well formed data with specific fields and values. In Gravity a card is based on a template which contains a definition of the fields. Therefore a card will contain definite fields and values rather than one large free form text field.
Not only does Gravity store fields, but it stores the data in a powerful data store that allows users to search for the cards they need. This technology allows us to produce filters to exclude cards from those displayed.
Gravity Automation
The inspiration for the Gravity automation system derrives from work in 2004 at Nothing But Net (NBN) on dependency trees. In 2004 the NBN Workflow system was used to automate a property sales process at a investment company. The property sales process includes multiple stages such as creating the sales and purchase agreement, having the agreement signed, preparing mortgage applications, and considering LIM reports.
A dependency tree automation system examines which automation actions have their dependencies met, and if so they will then execute the action. A dependency may be the successful completion of another action, the current phase of the card or whether a certain value is set. Dependency tree automation is event based, in that events kick off the process of examining whether dependencies are met.
The dependency tree approach was used to automate the a provisioning process managed by AgileZen. The new combination of AgileZen and automation was a response to issues of inflexibility experienced with the JBoss JBPM system. The nature of existing popular process engines such as JBPM means they do not handle complex processes in a adaptable way.
The AgileZen with automation proved to be effective to an extent, but it was clear that there were a number of issues with this solution that we needed to resolve.
Performance
AgileZen was not designed to handle thousands of cards. As the number of cards increased the performance began to degrade because the number of cards to display at one time was too large.
We needed a system that would be able to handle large numbers of active cards without degrading the users experience. Gravity has solved the performance problem in two ways. The first is the technology being used to store data, which makes search and filtering more flexible and powerful. The second way has been to reduce the volume of data being returned by maintaining a focus on returning only what is really needed to the client. When there is a large set of cards to display they are loaded in batches as the user scrolls down the screen.
Well Formed Data, Search & Filtering
AgileZen stores most of the card data in a single text field. While we found ways to store fields inside this text field it meant that there was no way to search or filter based on these fields. We needed a system that would allow us to search and filter based on any of the fields stored.
But we didn't want a static hard coded set of fields. We needed to have a template, but one which could be created and modified by users. As described under performance, Gravity was designed from the ground up to use a advanced data store that allows us to search and filter cards based on fields. This data store does not require us to define tables or data schemas, nor to define indexes. The practical implication of using this technology is the ability to deliver the search and filtering capabilities we need without tying ourselves to specific data structures or domains.
Visualisation
The AgileZen view of a board can contain a large confusing amount of data. In addition to taking a long time to load it was confusing and difficult to navigate. We needed a display that allowed users to focus on the subset of cards they really cared about, whether that was the cards in a specific phase, or met some kind of filtering criteria. We wanted users to be able to easily switch between views to find the cards they needed quickly.
Furthermore we didn't want to show all the data for a card, but rather allow the user to select a view that would provide a subset of fields that are useful to them. Again the idea is to reduce the amount of data to that which is useful for the user to get the job done.
In order to deliver these features we have developed a REST based user interface for Gravity which only pulls back the minimum amount of data required. In combination with an advanced data store we deliver a rich user experience that performs well.
Because the user interface uses REST all the web services to obtain data are exposed, which means that it is quite possible that other visualizations could be built on top of these web services to optimize the user interface for other use cases.
Defining Business Rules
With all prior workflow automation engine solutions it would require a major deployment to make changes to business rules or workflows. With JBPM and its relatives you need to deploy a new process at which point only new instances of the process will begin following the new process, while the old ones continue following the old workflow.
However, even in the NBN Workflow solution and the AgileZen automation system the business rules and actions were defined in XML and could not be changed without a software deployment. Deployments mean downtime and pain for system administrators.
What we needed was a way to make changes to business processes and actions without redeployment. We needed a way to have these changes take effect right away on cards that were already in process.
Gravity opened the door to adaptability and runtime modification by storing all the business rules and actions inside the data store. When an administrator changes a business rule the changes are put in place immediately and cards will begin to follow the new business rules.
This has dramatically changed the turn around time in addressing production issues, with administrators able to make modifications often within hours of an issue being identified as opposed to days or even weeks.
Currently we have a API for making changes to business rules, with no user interface. The intention was to build a easy to use user interface for administrators and managers which would allow them to create and modify business processes. Time constraints has meant we have been unable to build this new user interface. We have built a process visualization system which allows us to generate a graph of the process as above.
Running Automation
The AgileZen approach to automation involved downloading all the cards periodically to evaluate the business rules and trigger actions. Cards would be downloaded and examined even if no changes had been made to them. This had two impacts. The first was that automation ran infrequently, perhaps once every hour. If a change was made it would take a while for the automation to pick it up and evaluate it. The second issue was the huge load that this approach takes on AgileZen web services. We needed an approach that would be far more efficient, only running business rules on cards that have changed.
Gravity has been designed to receive notifications from the data store. When a card is modified the data store triggers a notification to Gravity to let it know there has been a modification. This way Gravity knows which cards have been modified and can then run the business rules against the modified cards. In this way any change to a card will trigger the business rules to run. This approach dramatically reduces load placed on the data store servers.
Gravity Architecture
The term 'Enterprise Class' is often thrown around, but in the case of Gravity the architecture has been designed to be performant, scalable, flexible and highly available. It has been used in anger to handle the core mission critical systems of a ISP.
Performance
One of the core limitations of prior systems was the user experience being very slow. A critical business system needs to not only be technically available, it must have reasonable response times for users. At every stage we have examined and optimised the performance to ensure Gravity is performing to users expectations.
Scalability
Performance is closely related to scalability. The real key to being able to maintain performance as processing load and the number of users increases is to implement a way of scaling out. There are three different clusters in Gravity, and each can be scaled up or down to meet demand. The first cluster is the worker nodes for Gravity which handle both the user interface web application and automation workloads. There is also a messaging cluster to handle delivery of messages between all the worker node. Finally there is the data store cluster that actually stores all the data.
Flexible
The Gravity Application supports plug in services which allow you to add business specific services to your Gravity installation. Gravity can already connect to a variety of external services such as SOAP and REST.
Highly Available
Due to the clustered nature of Gravity you get high availability out of the box. Even when you do need to restart Gravity instances you will never need to take down the whole cluster. All the services Gravity needs are also clustered, giving you a high degree of confidence that your business critical processes will not see downtime.
