BizTalk 2009 : The Enterprise Service Bus Toolkit 2.0 - The Architecture (part 1) - Mediation Components & Resolvers

Figure 1. The architectural concepts that comprise the ESB Toolkit

Since the ESB Toolkit is built on top of BizTalk, it obviously forms the foundational layer. The other four layers consist of .NET components, web services, and prebuilt BizTalk components (such as pipelines and orchestrations) that ship exclusively as part the ESB Toolkit. It is these four layers that we will focus on in the following sections.

1. Mediation Components

Even though the Mediation Components layer falls right in the middle of the overall architecture, we are going to start by looking at it first. The reason for this is that the mediation components perform all the functions that you will want your ESB to do. They provide dynamic transformations, give you the ability to dynamically route your messages to remote services, manage any required adaptation between transport protocols, and so on.

To get started with the ESB Toolkit mediation components, you will need to understand four core components. The first two components are the generic Routing Service and the generic Transformation Service. These are often referred to as the ESB services or ESB agents. The other two mediation components are the on-ramps and off-ramps.

Before you dig into these components, you should know that although all four of them perform very different functions, there is one common design pattern to all of them. They are all designed to work with untyped messages. This may seem strange to you if you are an experienced BizTalk developer, since you likely start all of your BizTalk projects by defining very structured XSD schemas for every message you will ever receive, send, or work with. However, the ESB needs to be much more generic, reusable, and flexible than a traditional BizTalk solution. In order to ensure that these mediation components are as flexible as possible, you really should avoid locking them down to specific schemas. Every schema you deploy into the architecture can potentially reduce the reusability of the overall system. You should also remember that in a web services world, developers are already defining a very specific schema for each service they develop (in other words, the WSDL). The service they develop is already going to validate incoming messages against this schema. Why then do you need to duplicate this schema inside of BizTalk? You can simply let the end service be responsible for its own schema and its own validation.

Let's look at a scenario where you are using BizTalk to route messages between a client application and a web service and you are told that the web service's WSDL has just been changed. You realize that you will now need to modify the schema inside of BizTalk before it will be able to successfully route messages to this updated web service. This involves opening up the BizTalk project in Visual Studio, importing the updated WSDL into the project, recompiling the project, redeploying it, and finally restarting the host so that BizTalk will use the new version of the schema. You would have to do all of this before the client application could even begin to test the updated web service. Once testing is done, you would have to deploy this new schema from your development environment all the way back out to your production environment. This is a crazy amount of work to have to do just to make a simple change to a web service's WSDL. In this scenario, BizTalk would really be a bottleneck to the agility of the overall development team.

To avoid this crazy scenario, it makes much more sense to have an ESB that does not require a schema for every message that is going to flow through it. Instead, the ESB should treat every message simply as a black box. The only requirement that the ESB should have on the message is that it be well-structured XML. Following this design principle, you could avoid having to update BizTalk whenever web services needed to change their schema or add a new message type.

This is the pattern that all of the ESB mediation components follow (that is, generic, reusable components that are not tightly bound to any one schema). They have all been built to support generic XML messages so that they do not need to be modified when you introduce a new schema within your organization.

1.1. On-Ramps and Off-Ramps

Although the ESB agents are really the heart of the ESB (since they provide the key capabilities of dynamic routing and dynamic transformation), we'll talk about the concepts of on-ramps and off-ramps first. The ESB agents have been developed as BizTalk components and thus have to reside on a BizTalk server. Since your service consumers and service providers will most likely reside on different machines than BizTalk, you need to provide some way of getting information into and out of the BizTalk server so that you can use the ESB agents. In the world of the ESB Toolkit, these entry points are on-ramps, and the exit points are off-ramps. On-ramps are responsible for the following:

• Providing a common and reusable entry point for applications to use when they want to leverage the ESB agents

• Initializing several properties associated with message that the other mediation components depend on

Now, these on-ramps and off-ramps probably will not seem like all that new of a concept to you as a BizTalk developer. You are probably thinking that you could just use BizTalk's Windows Communication Foundation (WCF) Publishing Wizard to expose a BizTalk schema as a web service. You could then configure a new BizTalk receive port to wrap this service using the WCF adapter, and, voila, you would now have a mechanism that most systems could use to send a message into BizTalk and thereby access ESB agents. What's the big deal? Well, we just finished saying in the previous section that the ESB components have been built to be highly reusable and loosely coupled. This means they cannot be tied directly to a specific schema. However, in the approach we just outlined, you would have exposed a web service using a schema that was already defined within BizTalk. Using this approach, you would need to expose one web service and one receive port for each message that you would ever want your ESB to receive. Imagine if your ESB needed to route messages to hundreds of different services. You would need hundreds of different web services and ports just to get the messages into BizTalk for processing. Every time you added a new service within your organization, a developer would need to build a new web service and receive port. You would quickly end up with a bloated and slow-moving environment.

When working in an SOA world, this traditional approach to exposing a typed web service is just not practical. You need a different approach that does not require you to deploy all of these distinct web services and receive ports.

What you need is a generic web service that would be able to accept any valid XML. This means that it will not define a detailed schema for the incoming data but rather will accept any and all XML as long as it is well formed.

Using this approach, the ESB will be able to easily accept new messages as needed without a developer having to make any change to the BizTalk environment. This also means that BizTalk administrators will have an easier job since they will have fewer ports to configure and manage.

Note that on-ramps do not represent a new component or feature within BizTalk. They are still comprosed of web services, ports, adapters, and pipelines that developers have always used. We simply use the term on-ramp to refer to a generic and reusable entry point into the ESB. In a later section, you will look at the actual BizTalk components that comprise these on-ramps.

Now, in a utopian and purely theoretical world, we can talk about the possibility of having just a single receive port to accept all of our messages. However, in reality, this is not going to be possible. In the most basic scenario, you are going to need to support at least two web services: one that handles one-way messages and one that provides two-way request-response messages. You also have to remember that as ubiquitous as web services are, they will never be able to handle all of your messaging needs. Many systems will need to access the ESB using MQ Series, FTP, flat files, or any number of other transport mechanisms. Therefore, inreality, ESB will need multiple on-ramps. However, as long as your on-ramps are capable of accepting generic messages without requiring a set schema, you can still dramatically reduce the number of ports that your BizTalk environment will need to support.

1.2. ESB Services

The generic Routing Service and generic Transformation Service represent the core functions that you will want to leverage right out of the box.

The generic Transformation Service does just what its name implies. It allows you to run any map that has been deployed into the BizTalk environment against your message. What makes this different from traditional BizTalk projects is that this service does not require you to specify, at design time, the name of the exact map that you want to run. Traditionally, a BizTalk developer has used a map by adding a map shape to their orchestration. Or they have specified the name of the map directly in the configuration properties for a send or receive port. The problem with both of these approaches is that only one map will ever be able to run. This is not a very dynamic or reusable solution and certainly does not fit into the overall goal of your ESB. To change this, the generic Transformation Service does not require a developer to lock in the exact name of a map at design time. Instead, it can dynamically run any map that exists within BizTalk, removing the need to specify the map name prior to runtime. This is accomplished by relying on the BizTalk APIs; doing this allows you to execute a map using code. By using these APIs, you can build a more intelligent and reusable service that is able to specify the name of the map as a parameter. This allows you to set up a single service that can be reused to handle any transformation that you need. Inside the ESB Toolkit, the Transformation Service is formally named Microsoft.Practices.ESB.Services.Transform.

Based on its name, you might think that it would be pretty easy to figure out what the generic Routing Service does. You might assume that this service handles the routing of message to remote services. However, what this service does is not quite as straightforward as that.

You have to remember that the actual transmission of a message is handled by off-ramps. What the generic Routing Service does is prepare a message so that it contains all the information required by an off-ramp. Off-ramps rely on dynamic ports to handle the actual transmission of messages. These dynamic ports work by reading a series of context properties that are attached to the message. The job of the generic Routing Service is to assign these context properties to the message so that the off-ramp can function. To do this, the generic Routing Service service must do the following:

Conceptually, you present the ESB services as single isolated services within the ESB. However, they are actually implemented in two different places. Both of these services exist as messaging-level services and orchestration-level services. Messaging-level services are implemented within BizTalk pipelines. Orchestration-level services are implemented as orchestrations. Both messaging and orchestration services perform exactly the same function for you. They simply give you an option of where you want the service to execute. Messaging-level services have much better performance than orchestration services. However, orchestration services can be chained together with other orchestrations. This allows you to build your own custom business processes within orchestrations while still leveraging the ESB services.

So, to summarize, the mediation components in the ESB Toolkit are composed of on-ramps, off-ramps, and the ESB services. Figure 2 illustrates how complex a BizTalk project can get if you use traditional BizTalk practices to integrate services.

Figure 2. A complex solution that can result from traditional BizTalk design patterns

Figure 3 illustrates the much more generic and elegant pattern that the ESB Toolkit implements through the use of its mediation components.

Figure 3. The dramatically simplified architecture using the ESB pattern

2. Resolvers

When ESB components execute, they need to be provided with some data on how they should run. For example, the mapping service needs to know which map it should execute. Likewise, the Routing Service needs to know how it should route a particular message. Off-ramps need to know what BizTalk adapter they should use.

To build truly generic and reusable mediation components, you need to avoid hard-coding configuration parameters into these components. In an SOA, these parameters might be stored in a number of different locations (all outside of BizTalk). Service endpoint information might be stored in a centralized UDDI server or some other kind of service repository. It might be stored in a database, a rules engine, or any number of different locations. Therefore, the ESB needs to be able to interact with all of these diverse data stores at runtime in order to retrieve runtime parameters.

This is where resolvers enter the picture. The resolver mechanism can be used to retrieve runtime data from a number of different data sources using a number of different transport mechanisms. The ESB Toolkit ships with resolvers that have been prebuilt to retrieve configuration data from a specific source. An example of these prebuilt resolvers is the UDDI resolver. Many people want to use a UDDI server to store information about where their services are currently located. The UDDI resolver allows the ESB to call out the UDDI server and inquire about where a current service is currently located. The ESB can then use the information returned from the UDDI server to route the message to the service's current endpoint.

2.1. The Resolver Mechanism

As you begin to examine the resolver mechanism, you will see that it is a fairly elegant solution that is quite flexible and very extensible. As mediation components execute, they call the Resolver Manager and provide it with a resolver connection string. This connection string specifies the resolver that you want to use, and it also includes parameters that the resolver will need to use when trying to retrieve your data. The ESB Toolkit contains a number of different resolvers that all implement the IResolveProvider interface. Since they all implement the same interface, the Resolver Manager can instantiate any of them at any time and know that they will behave the same way. Each resolver has been built to know how to retrieve information from one specific location while still implementing the same interface as every other resolver. Figure 4 shows this Resolver Manager and outlines the different resolvers that the ESB Toolkit 2.0 provides.

Figure 4. The resolver mechanism

When the generic Routing Service executes, you might want it to route your message to a service that is defined in a UDDI 2 server. To do this, you would need to provide the generic Routing Service with the following resolver connection string:

UDDI:\\ServerUrl=http://localhost/uddi;ServiceKey=;ServiceName=MySampleService;
ServiceProvider=MyServiceProvider.

					  

The UDDI part of the string that proceeds the : \\ is referred to as the moniker. It is this moniker that the Resolver Manager will use to determine which resolver you want to use. The remaining part of the string consists of the parameters that the UDDI resolver will need to have in order to connect to the UDDI server and retrieve information for the desired service.

2.2. The Out-of-the-Box Resolvers

The UDDI 2, UDDI 3, and WS-Metadata Exchange resolvers can be used to retrieve service information from service registries. The Business Rules resolver can be used to invoke a policy in the BizTalk Business Rule Engine. The XPath resolver can be used to retrieve information that is stored in the body of the message itself. The Static resolver is used when you want to hard-code configuration information into a mediation policy (you will learn about these shortly). The LDAP resolver is useful if you have a message that needs to be routed to a human via e-mail since you can use it to query an LDAP directory to find their current e-mail address.

The resolver framework has been designed to be incredibly reusable and extensible. Each resolver returns the data that it collects using a .NET dictionary object. This object is not tightly bound, which means you can add or remove data to/from your external data store without having to modify the resolver. To help you understand this concept, Figure 5 shows a screenshot from a UDDI server.

Figure 5. Services registered in a UDDI server that the ESB will use

In Figure 5, you can see that a number of services are registered within the Microsoft.Practices.ESB provider.You can also see that the Activate Service has nine bindings defined, whereas the OrderFileServiceWBindings service has only five bindings defined. The UDDI resolver is set up so that it will return whatever binding information it finds, no matter what the structure is of those bindings. This means that it can be used to retrieve information for both of these services. Whatever data it does find is returned in a .NET dictionary object that uses a key/value pair configuration. The resolver is not going to validate the structure of the data it retrieves; it simply retrieves it. This makes the resolver mechanism very flexible.

If you are wondering where you define these resolver connection strings, be patient just a bit longer.