BizTalk Server 2009 Operations : Scalability and High Availability

• New Hyper-V virtualization support

• Improved failover clustering

1. Windows Server 2008 64-bit Support

64-bit native execution allows BizTalk Server 2009 to take advantage of the benefits of the x64 Windows platform such as a larger memory address space, support for larger RAM, and faster I/O. The white paper "Benefits of Microsoft Windows x64 Editions" located at the following link covers the benefits of the Windows x64 platform in detail: www.microsoft.com/windowsserver2003/techinfo/overview/x64benefits.mspx.

We highly recommend implementing BizTalk on Windows x64 editions to take advantage of these benefits where appropriate. With x64, even 32-bit Windows applications can benefit from running on x64 Windows; however, note that BizTalk Server 2009 host instances can run x64 natively.

1.1. 64-bit Caveats

There are some caveats to BizTalk's 64-bit support, however. Some of the native BizTalk features are available only as 32-bit and must be run in a 32-bit host on a 64-bit Windows Server. Additionally, BizTalk Server Standard does not support 64-bit execution. The product will install fine on a 64-bit Windows Server but will be limited to 32-bit process execution. The product documentation provides good coverage on the native 64-bit support for each particular feature of the product. A few highlights are as follows:

• ExplorerOM: Supported only in 32-bit mode. If you have custom code that calls ExplorerOM, it has to be run in a 32-bit process.

• EDI AS/2: The EDI AS/2 components must be run in a 32-bit host process.

2. Hyper-V Support

The formal support of Hyper-V virtualization technology represents a huge step forward for the BizTalk platform. Starting in 2008, Microsoft announced a comprehensive list of server-based products that are officially supported for use within Hyper-V virtual OSs. Fortunately, BizTalk was one of them.

While it is unlikely that many organizations will run BizTalk applications that have a strict performance requirement or where throughput and latency may be a concern, Hyper-V does simplify the creation of virtual images for test, integration, and development environments. Often organizations are reluctant to spend significant hardware budget for functional, acceptance, and integration environments. Hyper-V now allows for these separate environments to be created with no additional hardware costs.

3. Scalability

Even after tuning an application, bottlenecks in performance can develop, especially if load increases, which may require upgrading or adding hardware to the existing BizTalk solution architecture. Thankfully, BizTalk Server 2009 provides options to easily scale up or scale out the solution architecture.

Depending on where a bottleneck exists, it might be necessary to scale out or scale up either the BizTalk tier or the SQL Server tier. This is why it is extremely important to monitor a BizTalk solution to help identify trends in hardware resource utilization such as memory, CPU, or disk before the problem occurs. System Center Operation Manager (SCOM) 2007 with the SCOM 2007 management pack for BizTalk Server 2009 can greatly assist with this task.

The white paper "BizTalk Server 2004 Performance Characteristics" is a great document to review in order to get a feel for how application design can affect resource utilization/performance. 

In addition to the details on adapter performance, this white paper provides detail on how to determine Maximum Sustainable Throughput (MST), which is a critical step for understanding the performance characteristics of any BizTalk application.

The next subsections cover when to choose an option and the steps involved.

3.1. Scaling Out the BizTalk Tier

Scaling out the BizTalk tier is a good option when BizTalk Server is the bottleneck in terms of high CPU, memory, or heavy disk I/O and adding servers makes economic sense and can solve the issue. High CPU can result from intensive pipeline processing, message maps converting between complex schemas, or large/complex orchestrations. High memory or disk I/O can occur under high load and can generally be addressed through scale-out, but not always. See the "Scaling Up the BizTalk Tier" subsection later for more information. In many cases, scaling out by adding another BizTalk Server is more cost effective than replacing an existing machine with a more powerful one as long as it can solve the issue.

Scaling out the BizTalk tier does not help when the Messagebox database is the bottleneck, which again highlights how important it is to monitor resource utilization on both the BizTalk tier and the SQL tier. Another scenario of when scaling out the BizTalk tier may not help is when an adapter is a bottleneck. For example, if the FTP adapter is the bottleneck, adding more BizTalk servers does not help if the limit exists on the other end of the FTP communication.

Scaling out the BizTalk tier is achieved through managing BizTalk hosts and host instances. BizTalk hosts are logical groupings where artifacts such as receive locations, send ports, and orchestrations execute. Hosts are physically implemented as host instances, which are .NET Windows Services. A typical way to organize a BizTalk application is to create a receive host, a send host, a tracking host, and an orchestration host. This type of organization can isolate processing so that if a receive problem is occurring, troubleshooting and debugging can be focused on the receive host instance(s), and it is a good idea even when there is just a single BizTalk server.

The way scale-out works is that when a receive adapter (or a send port or orchestration) is configured to run in a host such as a receive host, the adapter physically runs on every BizTalk server where an instance of the receive host is deployed. So, to scale out the host where the receive adapter executes, a new BizTalk server is configured to join the BizTalk Group, and a host instance of the receive host is deployed to the new server. Deploying a new instance of a host takes a few clicks in the BizTalk Server 2009 Administration Console. For information on managing hosts and host instances, please refer to the BizTalk Server 2009 documentation.

Most adapters will scale out as the number of number of BizTalk Servers increases. There are exceptions, however, such as the FTP and POP3 adapters. These adapters are not able to run multiple receive endpoints because of limitations in the underlying protocols (you can't have more than one BizTalk server receiving a file via FTP, for example). In these cases, you will need to introduce Windows clustering to cluster the FTP Receive adapter for purely failover and recovery reasons. There are no performance-based reasons to do this. The same situation may also occur with third-party adapters. Check with the vendor of the adapter for how they implement scale-up and scale-out within their products.

3.2. Scaling Out the SQL Tier

Scale-out of the SQL tier is primarily focused on the Messagebox database. The Messagebox database is where BizTalk application processing occurs from a database perspective. The first Messagebox database is created during initial BizTalk configuration and is the master Messagebox database. The master Messagebox contains the master subscription information and routes messages to the appropriate Messagebox database if there is more than one Messagebox.

When it is determined that scaling out the Messagebox database is required, Microsoft recommends dedicating the master Messagebox to do routing only by selecting "Disable new message publication for the master Messagebox database" in the BizTalk Server 2009 Administration Console, and then having the other Messagebox databases perform application processing. Microsoft also recommends going from one Messagebox to three Messagebox databases to benefit from scaling out the SQL tier and to overcome the additional processing overhead that occurs when there is more than one Messagebox database. For more information on scaling out the Messagebox, please refer to the BizTalk Server 2009 documentation.

3.2.1. Scaling the Messagebox

BizTalk Server provides the ability to scale the Messagebox into multiple Messageboxes for load balancing and performance reasons. Creating multiple Messageboxes has no benefit for disaster recovery scenarios and is useful only for spreading load across multiple databases and/or database servers.

The process of creating a multi-Messagebox installation is quite architecturally simple. One Messagebox will always act as the "master Messagebox," and it will have several child Messageboxes. The job of the master Messagebox is to do subscription matching and routing, and it stores all subscription information used by the message agent to match and route messages to the appropriate child Messagebox. If the subscription is an activation subscription, the message is routed to a child Messagebox in a round-robin fashion. If the subscription is an instance subscription, the message is routed to the child Messagebox that holds the dehydrated instance orchestration or the send port, which is waiting for a response from an adapter (e.g., SOAP or HTTP).

Adding multiple Messageboxes adds a fair degree of complexity to a BizTalk solution and should be done only in the cases of extreme load and to distribute instance subscriptions across multiple databases and potentially multiple SQL Servers. Depending on the type of application and the type of work it is doing, implementing a multi-Messagebox architecture will not help performance and may decrease overall throughput.

3.3. Scaling Up the BizTalk Tier

Scaling up the BizTalk tier is a good option when BizTalk Server is the bottleneck, if it can solve the issue and is a good alternative economically. Scaling up the BizTalk tier makes sense in the following scenarios:

• Large message transforms

• Large number of messages per Interchange

• High memory utilization by some BTS components such as pipelines or adapters

• A transport-related issue, such as EDI

Scaling up includes adding CPUs, adding memory or faster disk I/O to an existing BizTalk server, or replacing an existing BizTalk server with a more powerful machine. If you are using a 64-bit OS and BizTalk Server Enterprise, you can more easily scale up your available memory because of the OS's ability to address larger processes. This will help if you are constantly bumping up against throttling within the Message Agent because of memory pressure.

Next, we cover scale-up of the SQL tier.

3.4. Scaling Up the SQL Tier

Scaling up the database tier avoids the overhead of adding Messagebox databases that results from message routing and can be a good option if it makes economic sense and addresses the issue. In general, scale-up should be considered before scale-out for the SQL tier.

Scaling up the database tier can include adding CPUs, memory, or a faster disk as well as replacing the existing server with a more powerful server. One scenario that scale-up cannot address is when there is SQL lock contention on the SQL tier. In this case, scaling out of the SQL tier should be considered.

4. High Availability

Most BizTalk Server solutions are mission critical to a company and require high availability to meet business requirements. Therefore, BizTalk Server 2009 provides features for high availability as well as takes advantage of high-availability capabilities in the underlying operating system such as clustering to provide a robust architecture.

Both BizTalk and SQL Server must be highly available in order for the solution to function, and we cover this in the next two subsections. In addition, the master secret server must also be highly available for a BizTalk solution to function. Clustering the master secret server is covered in the following text as well.

4.1. BizTalk Tier High Availability

As with scalability, BizTalk hosts are the key feature to provide high availability for the BizTalk tier. As mentioned previously, BizTalk artifacts such as receive locations, send ports, and orchestrations are configured to run in a logical host but actually execute in host instance Windows Services that can be deployed to multiple BizTalk servers in the BizTalk Group.

If there are two BizTalk servers with an instance of a particular host but one server becomes unavailable, processing will continue on the other BizTalk server with the instance of the host, providing processing redundancy or availability.

BizTalk Server 2009 introduces Windows clustering for host instances. Host clustering is necessary for integrated BizTalk adapters that should not be run in multiple host instances simultaneously such as the FTP receive handler or potentially the POP3 receive handler. The FTP does not have file locking, so if there were two FTP host instances running, both could potentially receive the same file, resulting in duplicate processing. Therefore, only one FTP receive host instance should be configured, which is why it should be clustered because it is a single instance. For POP3, the scenario is similar to FTP's. If the POP3 server allows multiple simultaneous connections to the same mailbox such that duplicate messages may be received, the POP3 adapter should be run as a single host instance and should therefore be clustered for high availability. Another scenario where host clustering is necessary for high availability is when a single host instance is required to maintain ordered delivery for the MSMQ or the WebSphere MQ-integrated adapters. For these adapters, the single host instance can be clustered to provide high availability. For more information on implementing high availability for the BizTalk tier, please refer to the BizTalk Server 2009 documentation.

The next subsection covers SQL tier high availability.

4.2. SQL Tier High Availability

The BizTalk Group exists in the SQL tier in SQL Server database instances, which can be clustered for high availability. From a BizTalk standpoint, it is transparent whether the SQL Server instance is clustered or not, so SQL tier high availability is a matter of implementing SQL Server in a highly available manner by configuring the BizTalk Group databases on a SQL Server instance that is clustered. For more information on SQL Server high availability, please refer to the SQL Server documentation installed as part of the product or available online here:Next, we cover how to implement the master secret server for high availability.

4.3. Master Secret Server High Availability

If the master secret server becomes unavailable, all runtime operations already running will continue to run, but SSO servers will not be able to encrypt new credentials. Since there can be only one master secret server in the SSO system, Microsoft recommends clustering the master secret server, and the BizTalk Server 2009 documentation covers the steps in detail.

Clustering the master secret consists of installing Enterprise Single Sign-On on a Windows cluster. Quite often, customers choose to cluster the master secret server on the same Windows cluster that hosts the BizTalk Group databases to avoid the cost of having a separate Windows cluster just for Enterprise SSO. Where you install Enterprise SSO depends on the business requirements of the solution as well as economic considerations. Just ensure that the master secret is installed in a Windows cluster to ensure high availability whether on a standalone Windows cluster or in the same Windows cluster where the BizTalk Group SQL Server database instances are clustered.

NOTE

You must not cluster the master secret server on any machine that is running BizTalk Server. To use clustering, Enterprise SSO must be clustered on the SQL Server cluster or a separate Windows Server cluster.