Enterprise Service Bus with BizTalk Server and Windows Azure : The ESB Toolkit

The ESB Toolkit now exists as a set of pre-written components, services, and architectural guidance and best practices that extends the capabilities of BizTalk Server by providing the functionality needed to implement the Enterprise Service Bus pattern.

We will shortly look at some of the specific parts of the ESB Toolkit, but first, let’s describe the set of core services it natively provides:

• Resolver service– This service allows external consumer programs to leverage the resolution mechanism. The Resolver service can be used to abstract service registry access and make it broadly available in a heterogeneous environment.

• On-ramp service– This service provides a means for Web service consumers to send messages to the ESB. Web service SOAP headers become message context properties as the message passes through a context setting component in a receive pipeline.

• Transformation service– This service allows non-BizTalk applications to access and leverage the BizTalk transformation engine. Specifically, it allows access to all Web service consumers, including those not running on the Microsoft platform.

• Exception Management service– By publishing the fault schema using the default BizTalk schema publishing mechanism, this service enables consumers to submit messages so that non-BizTalk (or non-.NET applications) can participate in the ESB exception management scheme
  

• BizTalk Operations service– This service returns information about the current state of BizTalk artifacts.

These core services are available as orchestrations and helper classes, and are also exposed as Web services.

It is important to note that the ESB Toolkit does not in any way change the way that BizTalk works. A lot of the parts of a BizTalk implementation rely on configuration settings. These configuration settings are typically set at development or post-deployment phases. The ESB Toolkit adds the runtime resolution capabilities required for custom, dynamic processing. For example, many global context properties are created for you when a message is constructed or received. The ESB Toolkit uses context properties to house the metadata it needs to operate on a message as it goes through its lifecycle within the ESB.

Just as BizTalk builds on the functionality of the .NET framework and WCF, the ESB Toolkit introduces building blocks layered upon BizTalk Server itself (Figure 1).

Working our way through the layers in Figure 15.5 from the top down, the upcoming sections cover these parts of the ESB Toolkit.

Itineraries

The lifecycle of a message inside the BizTalk ESB environment is controlled by itineraries. At the simplest level, an itinerary can be thought of as the series of processing steps that are applied to a message.

For example:

Itineraries can be considered a form of microflow, meaning that they can be used as a lightweight form of service composition. The notion of an itinerary (when classified as a microflow) is not to contain a great deal of business, compensation, and complex branching logic. For that, we have established service composition and orchestration technologies. The goal and appropriate use of an itinerary is to represent a simple series of steps that do not necessarily constitute an entire business process.

The metadata envelope (Microsoft.Practices.ESB.Itinerary.Schemas.System_Properties) contains properties that are related to flow control of messages through the use of itineraries, as shown in Table 1.

The ServiceName and ServiceState will typically be used to define a subscription. For example, a subscription may be based on logic such as: “Send me all messages that are of type Order that have a ServiceName of OrderProcess and a ServiceState of pending.”

Message publishers (either the service sending messages to the bus, or the proxy that receives the message and puts it on the bus, or the receive pipeline through which the message flows) will set these properties, and then filters on subscribers will use it to determine if a message should be picked up. With this technique, publishers and subscribers are completely autonomous. If you would like to introduce a new subscriber to an existing message flow, it can be deployed and start processing the messages without any need to modify (or even notify) the publishing service.

An itinerary can be comprised of a set of services that need to be invoked in order to process a message. These services can be activated in one of two execution contexts: in the messaging layer or in an orchestration. In either case, when the service completes its logic, it is expected to have advanced the itinerary to the next step. This means the current step is effectively removed to make the next step the current step. It is essentially a stack, with a linear flow of states.

This may seem like a sequential flow, but it doesn’t need to be. For example, nothing would prevent multiple agents from listening for the same subscription properties, which would result in a parallel branch in the message flow.

In addition, the starting itinerary may not be the same itinerary that stays with a message throughout the message’s lifecycle. One agent may perform an operation on a message and may completely replace its existing itinerary with a new itinerary. This can occur, for example, if an error condition is encountered.

Within the ESB Toolkit, a message’s entire itinerary is serialized and stored in the ItineraryHeader context property of that message. In order for BizTalk’s routing engine to have visibility and to be able to fulfill subscriptions, parts of the current itinerary step are de-normalized and kept in context properties.

Itineraries Types

There are three ways that itineraries can be specified:

• server-side– Itineraries can be stored in and retrieved from a centralized repository. There are several ways in which this can be done.

• client-side hybrid– In this scenario, the client makes some calls into the ESB infrastructure (for example, resolving an endpoint address) and then constructs an itinerary. The itinerary is then stored in SOAP headers and the message is sent in to the on-ramp.

• client-side– Here, the client constructs the itinerary that is stored in SOAP headers. The message is then sent in to the on-ramp.

It should be noted that the client-side and client-side hybrid options are generally less desirable as they put a lot of responsibility on the service consumer to specify what will happen to the message as it is being processed. While in some situations this may be acceptable, it can result in a fairly brittle messaging framework that further introduces responsibilities pertaining to updates and versioning.

The server-side option provides a single source of truth for itineraries (the repository), and relieves service consumers of any responsibility around message processing. They simply send in their messages. In addition, this approach can be used with any transport, as there is no specific reliance on SOAP headers.

The Itinerary Lifecycle

As shown in Figure 2, an itinerary is initially created using a visual design surface inside Visual Studio (although itineraries exist as XML, meaning they can also be created in code). The itinerary is then exported either as XML (possibly for migration to other environments) or to the itinerary repository (most likely SQL Server).

Figure 2. An example scenario showing an itinerary, from design-time creation to runtime utilization.

At runtime, a message is received at an ESB on-ramp and an itinerary is selected. The itinerary is then applied to the message, meaning the details of the itinerary are added to the message context properties. The message then continues on into the ESB for further processing.

For the actual selection, we use an itinerary selector pipeline (a pipeline that includes an itinerary selector pipeline component that is included with the ESB Toolkit) in the receive location. We then look up the itinerary in the itinerary repository. In the properties of that component, we need to specify a resolver string.

Typically we would do so by using one of the following resolver strings:

• STATIC:// – We hard code the value to be looked up using this static resolver string that specifies which itinerary to use. This puts in a static reference in the itinerary allowing the flow to be changed by updating the itinerary without the need to change the receive location properties.

• BRI:// – We use this resolver string to query the business rules engine. This allows us to change which itinerary is selected by changing a business rule without changes to code.

Note that resolvers are explained in the next section. The message receipt flow is as shown in Figure 3.

Figure 3. Messages exchanged via the itinerary pipeline and the use of resolvers.

Resolvers

Some parts of an ESB infrastructure require the dynamic runtime resolution of values, namely those parts that involve dynamic message routing and transformation. This type of functionality has been abstracted into a series of pre-built components called resolvers that are invoked at pre-defined points in a message’s lifecycle.

It is the responsibility of the resolver to determine a set of values needed for a runtime query and to then execute this query against an appropriate metadata store. A common example is the UDDI resolver, which is used to query a UDDI registry and return an endpoint location. However, resolvers are not just about endpoints. They can be used to resolve other artifacts, such as maps or itineraries.

Resolvers included with the ESB Toolkit for endpoint resolution include UDDI 2.0, UDDI 3.0, WS-MEX, Static, BRE, and XPath. Resolvers provided for artifact resolution include Static, Static (for itineraries), SQL Server, Composite, BRE (for maps), and BRE (for itineraries).

Resolvers represent one of several extensibility points within the ESB Toolkit. Should you require, for example, endpoint resolution to occur against a custom registry, you can create and register a new resolver.

Resolution can occur as part of the execution of an itinerary or the components can also be invoked programmatically.

Typically, resolution will occur at one of two points:

• when a message is received, if it has some properties that we know will lead to a resolution requirement (for example, the message is routed, but the endpoint address is not yet defined)

• at the last possible moment (just in time or JIT resolution), before a specified operation is performed (for example, if a message is about to be transformed but we don’t yet know the map type)

It is possible that resolution can fail when a message is first received. This may occur because a process has not yet run that would provide that information. A resolution failure at this stage is non-fatal because the operation that requires the information is not about to be performed. In addition, as is the case with long-running processes, the location of an endpoint could change from the time the process starts to the time we invoke the service. By using JIT resolution, we are not affected by changes in endpoint location during process execution.

However, if the JIT resolution of an endpoint fails, it is a fatal error because the operation in question would be the next thing that needs to occur and the failure means that we are missing a piece of information required to proceed.

In this scenario, items that require resolution are:

• endpoint addresses

• endpoint configuration information

• map types

Resolution can be performed by querying a variety of sources, for example:

• UDDI

• BizTalk Rules Engine

• XPath into the message itself

• hard coded resolution logic specified through post-deployment configuration (not in code)

• a custom assembly that implements the IResolver interface

The last option allows you to effectively extend the resolution mechanism by creating custom assemblies that perform custom actions, such as invoking a Web service or querying a database to perform the resolution.

The resolution mechanism itself is exposed as a Web service although, for performance reasons, the internal ESB components make direct calls to it. Exposure via a Web service contract allows other parties to call directly into the resolution mechanism, if required.

Adapter Providers

The itinerary components provided by the ESB Toolkit exist as .NET components. When resolution is performed by a resolver, a .NET Dictionary object is returned. However, when we go to set adapter properties that will be required for messaging, we need to work with BizTalk adapters, which cannot accept Dictionary objects. Adapter providers form a bridge, taking the appropriate values from the Dictionary object and setting the corresponding adapter properties.

The adapter providers provided by the ESB Toolkit include: WCF-BasicHttp, WCF-WsHttp, WCF-Custom, FTP, File, SMTP, and MQSeries. It is worth noting that there are actually fewer adapter providers than there are adapters. This does not mean that the other adapters cannot participate in ESB-based messaging exchanges; it simply means that to use adapters without adapter providers will require a hybrid solution (such as using a static send port subscribing to a specific ServiceName/ServiceState combination).

You do not invoke adapter providers explicitly; they are invoked as part of the resolution process. If the resolver connection string contains an EndPointConfiguration attribute, it will be populated by the resolution component with a dictionary of name/value pairs. When the resolution process ends (if this attribute is present and populated) the adapter manager will instantiate the appropriate adapter provider. The adapter provider will then iterate through the Dictionary object, setting appropriate properties in the message context that will ultimately be used by the dynamic off-ramp when the message is transmitted.

WCF-Custom and REST Services

Communication with REST services raises specific considerations, for which several architectural options exist. For example:

• From the receiving side, you can expose a REST service that uses the PUT method to submit data to the service. The service would then act as a relay by submitting data into BizTalk as a message. You could do so by interacting with the helper components or core services provided by the ESB Toolkit and directly submitting the message using the direct submit approach, or by invoking a WCF on-ramp service.

• From the sending side (invoking the REST service), you have several options. You can programmatically call a REST service from a helper class and invoke it from an expression shape in an orchestration, but this circumvents the publish-and-subscribe mechanism, and means any changes you make need to be code-level changes.

The WCF-Custom adapter provider opens the door to working with adapters that are part of the BizTalk Adapter Pack, as well as with any other WCF bindings that may be available. Specifically, this adapter provider can be helpful for enabling communication with REST services. You can use the WCF-Custom adapter provider in conjunction with the WebHttpBinding to enable communication directly with REST services. Note that when following this approach with the GET method, you will need to implement a custom behavior.