The Onus To Comply

It is like a barber’s chair that fits all buttocks,
the pin-buttock, the quatch-buttock, the brawn
buttock, or any buttock.

William Shakespeare (All’s Well That Ends Well, II.2)

Enterprise software companies have earned hundreds of billions of dollars making and selling software integration products and services.

Why are enterprises spending such amounts; and what are software integration technologies?

In my first article in the Process Integration Trilogy, I explained the term “business process”, and discussed how software applications and systems were scattered in enterprises and thus resembled an archipelago. Further, the islands in the archipelago needed to communicate with each other in order to achieve certain higher level business outcomes (often termed as “complex business processes”). In the case of enterprise software systems, an archipelago of islands is the preferred architecture as opposed to a monolithic “vast mainland”. The reasons for this preference are intuitive and logical. Each modular software system performs a definite collection of use cases and is responsible for maintaining and owning certain collections of enterprise data. This makes it easier to develop and maintain each modular software system. A monolith would cause an administrative and architectural logjam. Organizations evolve and various new capabilities are added periodically. A monolithic software model would make it impossible for companies to add and update various new products and capabilities into their organization.

In real-life, almost all organizations are based on a collection of hundreds or even thousands of modular heterogeneous software systems.

The primary drawback of this scheme lies in the issue of interoperability among these component software systems. In a previous post, I had explained how business processes may require multiple software systems to participate in a workflow. The software applications need to communicate with each other in order to complete the business process.

Process integration is a term used to denote the concept of integrating business process use-case steps from multiple applications in order to achieve a complex business outcome. The purpose of application integration is primarily to achieve process integration.

Hundreds of billions of dollars have been spent in acquiring products and technologies that enable application integration — in order to achieve process integration.

Enterprise Application Integration (EAI) is the general term used in the industry to classify software products that enable multiple software applications and systems to get connected.

If you search the Internet for the phrase “Enterprise Application Integration”, you will find several hundreds of results. Many have attempted to classify the various types of technologies that enable EAI. These include categories such as point-to-point integration, “bus” integration, broker-based integration, to name a few. I don’t intend to provide yet another classification of EAI in this post. It is a very mature topic with plenty of high quality information already available. However, I must also add that I was very disappointed with the Internet search results on this topic. The initial few pages returned by popular search engines (at the time of this writing: early 2022) point to some really poor material. Now that may sound as if I have contradicted myself; but the fact is that EAI has been around for a long time and in my career of nearly thirty years I have come across plenty of high quality literature on EAI. Therefore my observation about the poor quality of the “top” Internet search results on EAI should be taken as a comment on the state of search engine optimization (SEO). While SEO has spun a whole sector in the digital marketing world, and a source of livelihood for many, I feel that it has created more avenues for digital information corruption and favoritism. We have unwittingly given control to HTML-indexing-companies to impose their “rules” about how Websites should behave in the digital world. Instead of ordering search results on the basis of lexical and semantic proximity of searched terms with the indexed content, we have created avenues for corruption by relying on “scores” computed on the basis of irrelevant aspects such as the number of external links pointing to a given page. When you “rank” a page, there is an implicit notion of ordering based on “quality” of the content in the page. That quality is based on how close the page is to the phrase that was searched for. It should not be based on how many external links point to that page. Such measures lead to people gaming the search engine and other corrupt practices.

Anyway, coming back to EAI, the purpose of this article is to highlight an important aspect about this topic and to show how UI-integration (a.k.a RPA, or robotic process automation) fits into the EAI context.

In recent years, there have been several articles and posts that classify RPA as another entry in the list of EAI technologies. I have discussed in other articles on how this term called “robotic process automation” or RPA got coined in order to spread awareness about UI-Integration technology.

I will summarize it here: UI-Integration technology, when used for automating Software Driven Labor, can be termed UI-Automation. RPA is just another name for UI-Automation, which got coined by a certain technology analyst firm and a particular UI-Automation product company.

Therefore, RPA’s entry into the EAI list is synonymous with UI-Integration’s entry in the the EAI list.

Let us now look at the characteristics of conventional EAI (before RPA/UI-Integration got included in the list).

Software applications that need to interoperate with each other must find a way to perform this communication. For example if software A needs to trigger some action in software B, then one possible method could be a “point to point” communication. This could be accomplished via some network communication method that is set up between the software applications A and B. If you consider the seven-layer OSI Reference Architecture for network communications, these types of point to point communications could be performed via a variety of methods, starting from the base ‘physical’ connection all the way up to ‘application’ layer mechanisms. Over generations more successful mechanisms have been ‘session’ layer methods such as Sockets, and ‘application’ layer methods such as HTTP/HTTPS. In particular, there have been various application and presentation protocols that have been formed on top of Sockets.

The applications that intend to participate in such communications have to agree on the communication protocol. The application that responds to external requests has to be prepared to do so, and it usually publishes its capabilities via some form of documentation. These can range from implicit arrangements between the developers, to specification documents available in paper format or hosted online, and via electronic interrogation of metadata.

The other type of “traditional” EAI, comprises of technologies such as enterprise service bus (ESB), message oriented middleware (MOM) and several others. The characteristic of this format is that both requestor and responder applications are programmed to be able to communicate via a common mechanism (such as ESB, MOM, etc).

The figure above depicts this situation. Each application that interoperates contains relevant code modules that enable communication with external systems. These forms of communication could be uni-directional or bi-directional. In either case, it is clear that the onus to comply rests with the software applications. They need to comply with either a point-to-point protocol or subject themselves to a protocol dictated by some common Integration Technology.

Why is this important? This paradigm causes several significant issues for process integration.

In order to compose a complex process by integrating simpler process-steps from individual applications, we need to automate the flow of data and control between the participating applications. From the above discussion and diagram it is clear that such flow of data and control between applications requires some form of integration technology to be available in the participating applications. I had mentioned earlier that for ease of evolution, ease of control, separation of concerns, and agility in adopting changing and adopting newer software techniques, it is best to continue with the “archipelago” model for enterprise software systems. Large monolithic systems will be, by definition, difficult to manage, and slow to adopt changes and embrace newer software technologies.

So, in such a world of loosely coupled software applications, it is imperative for each application to have the necessary capabilities to interoperate and thus enable process integration. Organizations struggle with this because individual software applications were not constructed to enable this form of interoperation. Adding integration capabilities often requires expensive projects and this leads to the long tail of integration graph, which was explained in another post in my blog.

The conclusion, therefore, is that in order to compose complex automated business processes across software applications, we need these applications to interoperate with each other. The onus to comply with an integration technology or predetermined protocol lies with each participating software application. Even if one of the participating applications is unable to fulfill its onus to comply with the selected integration technology, the end-to-end automation gets broken and the organization resorts to Software Driven Labor.

Thus the inability of software systems to abide by the onus to comply leads to Software Driven Labor. This situation has lead to the $400 billion process outsourcing and shared services market.

Now let us look at UI-Integration technologies. The figure below shows a schematic of how UI-Integration fits into the scheme of things.

UI-Integration software’s key capability is to be able to “hook” into the user interface screens of applications as they execute in some desktop environment. The “desktop” can be running directly on a physical device or on a virtual machine (VM) that is deployed in a physical device. The VM based deployment offers, in my opinion, the best model to deploy UI-integration. Once it gets access to the UI of a software application (on a desktop), the UI-Integration tool can perform any kind of action on the UI of the target application. I have discussed this in my blog posts titled Origin and Evolution.

The utility of this technology is that with it, we can create composite workflows spanning actions on multiple applications. In the diagram above, composite processes F, G, and H comprise of workflows that are executed on the UI-Integration tool. These processes (F, G, H) may use actions performed on any or all of A, C, D, E applications.

Those familiar with RPA may not find anything new in the above. However, the key matter to be realized is that by laying out the model as a stack: caller above and callee below, we can see that the onus to comply flows from bottom to top. It is then evident that conventional Integration Technologies and point-to-point integrations impose the onus to comply on the individual software applications that seek to interoperate. But UI-Integration technologies relieve the application software from this responsibility — instead they take charge of this aspect by implementing various techniques and adapters that can connect with a wide range of application types.

Although seemingly obvious, this is a very important matter to be understood in terms of the new “Application Integration Reference Model”

This is the real reason why UI-Integration is an important addition to the New Enterprise Application Integration scheme. Traditional EAI has shortcomings in that it imposes its standards on the participating applications. This leads to cost-prohibitive changes having to be made on individual applications to enable them to comply with the EAI technology deployed in organizations. EAI products tend to evolve over time; and requiring every participating application to keep pace with the evolution of the underlying EAI can be very expensive and time-consuming. The obvious benefits of traditional EAI is that once it is configured appropriately, it offers a very robust and reliable mechanism for application interoperation. In general, the performance of processes using EAI is several orders of magnitude faster than processes using UI-Integration technologies.

When changes are made to applications’ code, the developers ensure that the integration “contract” is not broken. If breaking changes are unavoidable in certain cases, the developers will ensure that these interface changes are published so that all the affected applications’ developers can also modify their systems accordingly.

In the case of UI-Integration, the tool takes responsibility of the “connection logic” and the process flow definition assumes responsibility of knowing what to do on each participating application. But since the onus to comply is placed at a layer “above” the applications, we get the benefit of not having to force these applications to include code to integrate with each other. The impact of this is far-reaching. It allows better manageability of enterprise software applications. Software driven labor can be eliminated, and the long tail of software integration can be implemented.

The key drawback with the UI-Integration mechanism is that due to the onus to comply with all participating applications, the integration tool as well as all the process flow definitions that are deployed over the tool (F, G, H in the example above) need to constantly keep themselves updated to adjust for any changes that might occur in the target applications (A, C, D, E in the example above).

In the case of conventional Integration Technologies, if the integration software itself undergoes breaking changes, then the “onus to comply” rule kicks in and all components “above” it in the stack are forced to make changes in their code to adjust for the change. In real-life enterprise deployments, such events are announced in advance and even if the application developers don’t get sufficient time to make the changes, they will at the very least know that a breaking change will come into effect on a given date.

With UI-Integration, this form of communication strategy has not been formally established. In conventional integration, the specifics of the breaking changes are precisely known. But changes in the UI of applications are not precisely communicated. In fact, in many situations, the developers of the applications (A, C, etc.) are often unaware that a higher level UI-Integration tool is operating upon them. Broadly, there are two types of breaking changes in UI-Integration. The first, is one in which the underlying technology for UI rendering is changed. For example, an application that was previously based on a Windows thick client UI technology (such as WPF) could migrate in a subsequent version to a browser based Web application. Many years ago SAP™ application UIs were based on Windows thick-client technologies. However, as of this writing (2022), they have moved all their UIs to browser based Web applications. A less drastic by equally “breaking” change can happen when the version of the rendering technology used by the application is updated. When this happens the mechanism and specification for selecting and identifying the target UI Elements in the workflow will have to be updated. The second type of breaking change occurs when the developers of the underlying application make significant changes to the screens such that expected UI Elements are no longer available, or when new UI Elements spring up on the screen that need to be used in the workflow. For example, in an updated version of an application, the developers might remove a text area Address field and replace it with Unit, Building, Street, City, Country, Pin fields. Process flows that used the old Address field would no longer be valid and they will all have to be modified to now make use of six new fields.

At present, this lack of formal change management poses a problem from a business continuity perspective. The remedies and workarounds include establishing a test environment on which all the UI-Integration process flows (eg. F, G, H) are periodically executed using test data. If any of the flows break due to one or more of the above mentioned causes, it would flag an investigation of the cause, which would then lead to understanding the “offending” application and the likely best remedy to fix the problem. Clearly, these are after-the-fact remedies and during the time the fixes are created and deployed, the affected process flow(s) will be non-functional.

I believe that this is a transitional state and eventually there will be sufficient mechanisms to communicate imminent UI changes to consumers of the UI screens. This process will accelerate as the adoption of UI-Integration technologies increases in the coming years.

The aim of this post is to justify the placement of UI-Integration as a first-class citizen in the Application Integration “Reference Architecture”. It is no longer the old “screen scraping” technology that had earned much disrepute in the 1990s. Contemporary UI-Integration tools (RPA tools) do offer OCR based screen scraping features. I do not advocate the use of these mechanisms for anything other than handling extreme cases; and most certainly not for document digitization!

I foresee the future of UI-Integration as follows:

• Over time, most UI screens will become browser based. The need to integrate with non-browser UIs will eventually disappear. This will significantly simplify the nature of UI-Integration products. Tools built on top of frameworks such as Selenium and Puppeteer would be able accomplish all the UI-Automation requirements of the future.
• Microservices, APIs, ESBs will continue to be preferred ways of accomplishing high-performance integration between software systems. But a new class of interactions will emerge in which UI-like, “chatty”, back-and-forth actions would be needed in order to accomplish certain types of business processes. Such types of actions would be too cumbersome to specify via APIs and too slow if attempted via Message Oriented Middleware. This will lead to a new class of UI-based applications that are not primarily intended for humans to use.
• Process integration will be “tapped” upstream and will not require discovery via fancy AI tools. Future applications will have system generated annotations that can be picked up by UI-Integration tools.
• Features such as “bot analytics” will wither away. At present they offer some form of measurement against the “old manual way”. All business data can and should be held in business software systems and not in workflow management systems.
• The use of paper-based and scanned-image documents for B2B will massively decline with the increase in online self-service applications and verticalized SaaS applications. The use of RPA as a sling-shot for OCR will end.
• Almost all the features of contemporary RPA products (including ‘bot’ management) will become commoditized and freely available.
• The term RPA may (regrettably) survive and continue to be used as a synonym for UI-Integration.

This is not a bleak picture of the future of UI-Integration. In fact, it is the most appropriate trajectory for this technology. As I had pointed out in another post, we have lost our way and wandered into all kinds of crazy schemes such as applying AI (artificial intelligence) and ML (machine learning) to mimic human behavior. We adopted tech-analyst-generated jargon such as “intelligent automation” and “hyperautomation” to describe an amorphous blob of unnecessary features.

The hidden gem is UI-Integration technology, and by using it wisely with other components in the Application Integration Reference Model we will finally be able to achieve full end-to-end automation of business processes.

Originally published at https://madhavsivadas.com on April 19, 2022.