Service Bus Communication

From SAM
Jump to: navigation, search

A service bus communication is a system which allows the communication between heterogeneous components in a platform based on service-oriented architecture (SOA). It should provide connection capabilities between these components and a set of features that enable the management and monitoring of interactions between applications.

Introduction

In order to design a system based on SOA the ESB approach could be a good solution. An ESB enables interoperability between heterogeneous technologies systems. The need for an ESB arises from the complexity of the organizations must coordinate and integrate their business processes, operational systems and data without sacrificing the essential technological innovation to be competitive. An ESB is the implementation of SOA, an architecture that allows maintaining integrated systems in a way totally distributed and interoperable.

Relevance to SAM

For the different modules in the SAM Platform to be connected, a communication system is needed that will enable transport of information between them. To allow this exchange of data between the different SAM modules the Interconnection Bus could be used. This bus offers the possibility to connect multiple federated instances of SAM (SAMs). Therefore, each SAM could share its own resources with the rest of SAMs in a homogeneous way.

To reach the objectives of this section in particular, a prototype of internal communication system could be created so that it allows the exchange of data and connection services in a flexible and efficient way. SAM platform could use TIE SmartBridge (TSB) as an approach in order to allow communication between the different SAM modules that compose the architecture. TSB is a B2B message exchanging broker that, in a very efficient way, is able to transport a message from a source point to its destination, performing the adequate message transformations and routing. The TSB will also cover the federation approach of SAM in such a way that multiple TSBs can be connected together providing interlinking and interoperation at a limited level between them. This interconnection bus will coordinate and facilitate the communication between the different SAM components, providing with build-in facilities such as message routing, formats transformation, message queuing, etc.

State of the Art Analysis

Federated Architecture

Due to the spectacular progress of communications and the increasing need to cooperate between heterogeneous and independent systems is necessary to think in Federated Architecture as pattern to design application architecture. In a Federated Architecture all entities involves in the system are be able to share all or part of its data allowing to support sharing data among the different application users. This kind of system offers also access control to the data based on access rights allowing transparent communication between physically distributed components, possibly heterogeneous and autonomous.

SAM aims at creating a Federated Architecture [1] a prototype, allowing in that way the distribution of infrastructures, but also the creation of advanced business models. The communication and Federated Infrastructure will allow the exchange of data and “plugging” of services in a fast, secure and scalable way.

Federated Architecture

Enterprise Service Bus (ESB)

An interconnection bus will be implemented using the ESB [2] approach based on TIEs Commercial Message Bus/ESB technology: SmartBridge. Furthermore, different SAM ESBs will be able to interconnect to compose a federated infrastructure of SAM instances, specially its assets Marketplaces. This interconnection bus will coordinate and facilitate the communication between the different SAM developed components, providing with built-in facilities such as message routing, formats transformation, message queuing, security, access control, etc.

The basic features that should present an ESB are:

  • Routing and forwarding messages.
  • Style of synchronous and asynchronous communication.
  • Multiplicity of modes of transport and link protocols.
  • Content transformation and translation of messages.
  • Orchestration and choreography of business processes.
  • Event processing.
  • Presence of adapters to multiple platforms.
  • Tools integration design, implementation and deployment.
  • Properties guarantee quality of service (QoS), such as transactional, security and persistence.
  • Audit logging and metrics.
  • Management and monitoring.

Interoperability

Service oriented architectures (SOA) [3] are the de facto standard of today's Enterprise Integrating Architecture (EIA) [4] and are widely used by developers and users. SOAfosters best practices paradigms which include: modularity, encapsulation, fine-grained granularity and the publication and discovery of services and data . It ensures the organisation interests are met by efficiently and cost effectively putting into operation corporate decisions from strategy to execution. The Enterprise Service Bus (ESB) is a technology that leverages best practices from EAI mechanisms and service oriented architectures. It is a messaging backbone, based on open-standard, which enables the implementation, deployment and management of SOA-based solutions. ESBs exploit several paradigms such as Web services, Message Oriented Middleware (message routing and message transformation).These systems provide integration capabilities and deployment models that support large numbers of highly distributed set of services via its safe, secure, scalable and manageable infrastructure.Essentially, it acts as a mediator between service providers and consumers by allowing:

  • Dynamic services to be discovered (via a registry service)
  • The choreographed and orchestrated of business process and services (via an orchestration engine which is the core integration mechanism of this bus)
  • The communication between services and applications through messages (Messages are stored in a queue ensure guaranteed delivery)
  • The correct encoding, mapping and message routing to the intended destinations (mediation patterns for routing, transformation, encoding and mapping are available .

Tools, Frameworks and Services

TIE SmartBridge (TSB)

In essence: Receive, Transform, Send

SmartBridge makes seemingly incompatible software exchange information with each other:

  • SmartBridge receives documents, for instance from your ERP system or from your business partner.
  • SmartBridge transforms these documents into a proper format for its destination.
  • SmartBridge sends the document to its final destination.
SmartBridge is like a postal service

To explain SmartBridge, we can use an analogy: SmartBridge is like an automated postal service for software systems, that sends and receives documents between companies.

The three phases of SmartBridge processing
  • SmartBridge does not produce documents; SmartBridge can only transform documents that enter the system: it needs input, in the form of incoming documents. Incoming documents come from your ERP system or from a business partner. SmartBridge can pick them up from some location, or receive them using a mailbox. Either way, this is called ‘receiving incoming documents‘.
  • SmartBridge will transform these documents into a form that is suitable for their destination. This is called ‘translating documents‘.
  • SmartBridge will send them to a partner. This is called ‘sending outgoing documents‘. It could also output the documents to a designated location instead, e.g. from where your ERP system could pick up and import these documents.

Standards and Policies

Extensible Messaging and Presence Protocol (XMPP)

Extensible Messaging and Presence Protocol (XMP) [5] is a protocol based on Extensible Markup Language (XML)[6] and intended for Instant Messaging (IM)[7] and online presence detection. It functions between or among servers, and facilitates near-real-time operation. The protocol may eventually allow Internet users to send instant messages to anyone else on the Internet, regardless of differences in operating systems and browsers.

Web Services

Web Services is the most used connectivity technology for service-oriented architectures. Web Services are modular applications, self-described, self-contained which can be accessed through a network. Based in open standards, they allow the construction of web applications using any platform, object model and programming language. Web Services modularity and flexibility make them appropriate for applications' integration with a minimum programming effort. This is why we propose the use of Web Services as the connectivity mechanism between the federation and the sources.

SAM Approach

The Interconnection Bus plays a central role in the SAM architecture allowing reliable transmission of information between the different SAM components and federated instances. It directly interacts with the different components of the Platform and is able to communicate with auxiliary external systems through the different pluggable Communication Adapters mechanism. It also provides advanced queuing mechanisms (allowing event-driven approaches), services orchestration, message routing and transformations.

Architecture and Dependencies

The Interconnection Bus is the component that provides the communication infrastructure between all the SAM components in the SAM platform. The major features in the communication module are explained in the following points:

  • Each component in the SAM Platform is developed as an independent service using different technologies. Therefore, it has been implemented a communication system which allows the interoperability between these heterogeneous services. For this reason, the Interconnection Bus component is based on an ESB (Enterprise Service Bus) system which supports SOA (Service-Oriented Architecture) and Event-driven SOA (SOA 2.0 or SOA+) approaches. SOA 2.0 combines event-driven architecture with traditional SOA allowing the use of events in the communication between SAM components.
  • The component allows basic features such as routing and forwarding messages, sending the same message to several recipients at the same time (Multi-recipient messaging), controlling the different information queues, where the messages are temporarily stored (Messages/topic queues), and finally synchronous and asynchronous communication.
  • Audit Logs about the component communication activity is available for tracing the messages. Besides, using these logs, it possible to get statistics about this activity, providing information for the Business Intelligence component.
  • A graphical user interface is available which allows easy configuration of the different services and processes. It also allows the monitoring of current activities and management of component logs.
  • Orchestration features is provided in order to define complex operations such as interactions between different technologies and data format transformation. A specialised Workflow Designer and its execution engine constitute the core mechanism of the Interconnection Bus.
  • The Interconnection Bus offers a pluggable mechanism in order to extend the communication protocols and formats supported.
IB Architecture

Implementation and Technologies

Frontend Technologies

Backend Technologies

Subcomponents

A summary of the tasks carried out for each subcomponent of the first version of the prototype is shown in the following table.

Subcomponent Task
Communication Adapter The Source and Destination components can use different types of communication protocols. The Interconnection Bus implements a pluggable mechanism that allows implementing and publishing different Communication Adapters so that they can be used by the different Source and Destination components.. Examples of these protocols are JMS (Java Message Service ), SOAP (Simple Object Access Protocol) or XMPP (Extensible Messaging and Presence Protocol). In that way, the different components can define which their favourite protocol and data format are, and the Interconnection Bus will be able to receive, transform and deliver the messages in the correct format and protocol
Queue Subcomponent in charge of controlling the different information queues, where the messages are temporally stored. The Queue subcomponent will ensure that messages are not lost and additionally will implement topic based publish-subscribe mechanisms
Workflow Editor The Workflow Editor is the graphical user interface to define and configure the orchestration steps including the configuration of the different services or processes to be executed in the Interconnection Bus
Workflow Processor The Workflow Processor is the subcomponent in charge of controlling and orchestrating the different services and actions described through the Workflow editor, e.g. before delivering a message, for instance, specific transformations, access to SAM Component services, digital signature services, etc
Routing Once the messages are received from the Queue, this subcomponent is in charge of processing them and instantiate the correct workflow
Logger To register the information of the messages flowing inside the interconnection bus, the Logger subcomponent stores the necessary information in the Cloud Storage
Bus Management Console The Logger information can be consulted by the SAM system administrators by using the Bus Management Console (BMC) or by other SAM components for accountancy reasons. In addition, the BMC allows the management of the processes, schedules, users, registered components services, etc
Federated Instance Registry The different SAM instances will be registered by the Federated Instances Registry sub-component
System Admin Person in charge of managing and controlling the SAM Platform
Federated Instances Through the Interconnections Bus, different SAM Instances can interact and exchange information

Functionality and UI Elements

This section explains how to execute the SAM TSB instance in order to use the features adapted for this 1st Prototype and the SAM CL in order to manage the messages types, components, services and routes.

SAM TSB Instance

For the first prototype, it has been set up a new TSB instance (SAM TSB) and adapted for the SAM requirements. Furthermore, several performance improvements have been carried out. In the following subsections is described the current available functionalities for the SAM Administrators.

Login

Through this interface different SAM Administrators can enter their credentials. Once validated the SAM Administrators are logged into the system.

TSB_Login.png
Dashboard

The dashboard contains different widgets in order to provide quick views about the following topics:

  • Errors in the System: This widget shows the number of undeliverable and unrecognised documents
  • System Health: This widget shows the information about the status of the different parts of the system such as CPUs (TIEKinetix Server has 2 CPUs), memory, disk space, services and archiving (external storing service for messages). The green button means that the health of component is good and red button means that something wrong is happening. In order to see what is exactly happening, the user can put the cursor over the circle and a tooltip message will show the detailed information.
  • Last 6 Documents: This widget will provide a quick view of the last 6 documents processed in the SAM TSB instance
  • Document Statistics: This widget shows different stats about the transmission of the messages such as number of received messages, sent or in process
TSB_Dashboard.png
Documents

In SAM TSB, all the messages are contained in documents and the documents are received, analysed and send to the destination. For this reason the document can have several statuses (received, ready to send, sent, undeliverable, in error, unrecognized, in process or archived). By default, it is defined several folders which contains the messages for each of the several statuses. Clinking on the different folders, it is possible to see the list of messages in each status. This list can be filtered to see the documents processed in the same day, 7 days before or 30 days before.

TSB_Documents.png

In case that the SAM Administrator wants to see the details of the message, he/she have to click on the message. After that, the details will be shown in the bottom part. It is possible to see the information for each one of the transmission steps (Received, Analysed and Sent). The information of the document is split in two sections: “General” section and “Document Contents”. The “General” section provides the following information:

  • Id: This is a unique document identifier
  • Created: Date when the message was created
  • Processed: Date when the message was processed
  • State Type: State of the Message
  • Stage: If the message is processed correctly or not
  • Format: Document Format (XML, JSON, etc.)
  • Type: Type of the message, which value should always be “SAMRequest”
  • Subtype: Name of the service, e.g. “CreateBucket”
  • Sender: Component which sends the message, e.g. “ContentGateways”
  • Recipient: Component which receives the message, e.g. “CloudStorage”
  • Extra Information: Some extra information, normally avoid
TSB_DocumentsDetails.png

The “Document Contents” section shows possible error codes and the associated message. It is also possible to see the original message clicking on the eye icon.

TSB_MessageView.png
Search Document

By clicking on the loupe icon, it is possible to introduce multiple filters to search for a specific set of messages. The filters are based on the document information and are subdivided in several groups:

  • General: General information filters
  • Partners: Senders and Recipients filters
  • Date & Time: Time filters
  • Advanced: Filters about state, stage, format, type or subtype
TSB_SearchDocument.png
Smart Folder

Clicking on the “+” icon in the bottom of the screen, it is possible to create Smart Folder. Based on the search fields, it is possible to create smart folders to have a quick view of the document that meet with the search specifications.

TSB_SmartFolder.png

SAM CL

As part of the Communication Adapter subcomponent, a user interface has been implemented, which provides functionalities to create message types, components, services and routes. The general idea is that the different SAM components communicate with the Interconnection Bus by using a specific XML format (the SAM Component Message Format) and the Interconnection Bus will be then in charge of forwarding these messages to the predefined destinations, keeping control of the events that occur during the transmission, message transformation, reception and acknowledgement (if necessary) of the information.

SAMCL.png

References

  1. Federated Architecture http://en.wikipedia.org/wiki/Federated_Architecture
  2. ESB http://en.wikipedia.org/wiki/Enterprise_service_bus
  3. Service oriented architectures (SOA) http://en.wikipedia.org/wiki/Service-oriented_architecture
  4. http://en.wikipedia.org/wiki/Enterprise_application_integration
  5. Extensible Messaging and Presence Protocol (XMPP) http://en.wikipedia.org/wiki/XMPP
  6. Extensible Markup Language (XML) http://en.wikipedia.org/wiki/XML
  7. Instant Messaging (IM) http://en.wikipedia.org/wiki/Instant_messaging