A typical BizTalk solution includes adapters, pipelines, maps, message schemas, orchestrations, and ports. Figure 1
shows a logical illustration of a BizTalk solution detailing the
different components that might be involved in it and dividing them into
their logical containers, inbound Receive Adapters and receive
pipelines, processing orchestrations and business rules, as well as
outbound send pipelines and Send Adapters. The illustration highlights
the constant reliance on the Messagebox and a typical message flow
within the solution. The receive locations and send locations are
implied as the Inbound and Outbound
containers. The maps and filters are applied to the implied ports. To
properly tune the environment and the solution, you need to identify the
different components used by your solution.
1. ASP.NET, SOAP, and HTTP
To tune HTTP, as well as SOAP
Send and Receive Adapters and WCF adapters, you have to fiddle around
with ASP.NET web.config and machine.config files as well as
BTSNTSvc.exe.config and the registry keys for the BTSNTSvc.exe service
instance that is hosting the adapter.
1.1. Receive Adapter Tuning
Tuning the SOAP and WCF
receive host is mostly about tuning web services, IIS, and the ASP. NET
worker process. You can use LoadGen or the Microsoft Web Application
Stress (WAS) Tool to stress test the isolated host and gather performance metrics. Table 1
shows a set of performance metrics to monitor while load testing the
web services. Web services use ASP. NET thread pooling to process
requests. To ensure that your web services use the thread pool most
effectively, consider the following guidelines (Meier, 2004):,
Set the maximum thread-pool thresholds to reduce contention:
Set maxIOThreads and maxWorkerThreads in machine.config to 100. The max-IOThreads
setting controls the maximum number of I/O threads in the common
language runtime (CLR) thread pool. This number is then automatically
multiplied by the number of available CPUs. The recommendation is to set
this to 100. The maxWorkerThreads
setting controls the maximum number of worker threads in the CLR thread
pool. This number is then automatically multiplied by the number of
available CPUs. The recommendation is to set this to 100.
Set minFreeThreads
in machine.config to 88 × the number of CPUs. The worker process uses
this setting to queue up all the incoming requests if the number of
available threads in the thread pool falls below the value for this
setting. This setting effectively limits the number of concurrently
executing requests to max-WorkerThreads – minFreeThreads.
The recommendation is to set this to 88 times the number of CPUs. This
limits the number of concurrent requests to 12 (assuming maxWorkerThreads is 100).
Set minLocalRequestFreeThreads to 76 × the number of CPUs.
This worker process uses this setting to queue up requests from
localhost (where a web application calls a web service on the same
server) if the number of available threads in the thread pool falls
below this number. This setting is similar to minFreeThreads, but it applies only to requests that use localhost. The recommendation is to set this to 76 times the number of CPUs.
Set the minimum thread-pool thresholds to handle load bursts. Set minIOThreads and minWorkerThreads to prep the thread pool for incoming large loads instead of creating new threads as requests come in to fill up the pool.
Table 1. Basic Counters to Monitor for ASP.NET (Northrup)
| Object | Counter | Description |
|---|
| Processor | % CPU Utilization | The
overall measure of total processor utilization on a web server. The
processor is the most common bottleneck on ASP.
NET web servers. If this counter peaks near 100% while the web server is
under load, you should add the % Processor Time counter for the Process
object to isolate which process is bogging down the server. |
| Process | % Processor Time | This
counter provides similar information to the % CPU Utilization counter
but identifies which specific process is using the most CPU time. To be
certain you gather all the information you need, you should select the
All Instances radio button in the Add Counters dialog box when adding
this counter. If the aspnet_wp process is consuming most of the
processor, it is a good indication that rendering ASP.NET pages is the
bottleneck. If the inetinfo process is to blame, IIS itself is the cause
of the problem. These conditions can be remedied by upgrading the web
server's processor, adding multiple processors, or adding more web
servers. If your ASP.NET application is database-driven and you run a
Microsoft SQL Server on the same system, you will very likely discover
that the process named sqlservr is causing the CPU bottleneck. The best
remedy for this situation is to move the SQL Server software to another
physical server. Alternatively, upgrading the processor or adding more
processors will help. |
| ASP.NET Applications | Requests/Sec | This
counter measures the current rate of incoming ASP. NET requests and is a
useful way to measure the peak capacity of your web application while
under load. The counter will report on only the number of requests for
files with extensions configured in IIS to be passed to ASP.NET—most
commonly, .aspx and .asmx files. To view the total number of requests,
including requests for images, add the Get Requests/Sec counter from the
Web Service object instead. |
| ASP.NET Applications | Sessions Active | This
counter measures the current number of active ASP.NET sessions. A
session is created by an ASP.NET application when a new user makes the
first request. The session lives until 1) the application explicitly
abandons it when the user logs out, or 2) no requests are received from
the user for the period of the session timeout. By default, ASP.NET
sessions timeout after 20 minutes. This setting can be adjusted by
modifying the timeout attribute of the sessionState element in the web.config or machine.config files. |
| ASP.NET | Requests Queued | Requests
are queued when the time required to render a page is greater than the
time between incoming client requests. In normal web traffic, request
rates are very erratic, and queuing might occur for a few seconds during
a busy moment. This will cause page load times to increase temporarily,
but the queue is quickly eliminated during the next quiet moment.
Traffic generated by a load-testing tool such as WAS might not have the
same erratic patterns and might cause the ASP.NET Requests Queued
counter to climb before it would do so under real traffic conditions. To
simulate these random peaks and valleys in web traffic, enable the Use
Random Delay check box on the script settings page of WAS. If this
counter still increases with this setting enabled, the server is
currently above its peak capacity, and a bottleneck should be identified
and resolved before continuing testing. By default, ASP.NET is
configured to queue a maximum of 100 requests. This limit is defined by
the appRequestQueueLimit attribute of the httpRunTime element of the web.config or machine.config files. |
| ASP.NET | Requests Rejected | After
the ASP.NET request queue is full, new requests are rejected. This
process is generally a good way for ASP.NET to behave under extremely
heavy load because it is better to return an error to the user
immediately and remove the request from the web server's queue than to
force users to wait for their browser to timeout. Monitoring this
counter gives you a running total of the requests received while the
queue length was at the maximum. |
1.2. Send Adapter Tuning
The default number of
maximum concurrent outgoing SOAP connections is two. This can cause
outgoing SOAP requests to timeout and create a bottleneck in scenarios
where a high volume of outgoing SOAP requests is expected. If not
treated, this problem can cause a buildup of messages in the Messagebox
and entries in the spool table resulting in overall performance
degradation. It may also cause the host instances running the SOAP Send
Adapter to recycle because eventually, if the load is high enough and
the time taken for the called web method to complete is long enough, all
threads in that host instance's thread pool will be exhausted, and the
host instance will be unable to service new requests as it runs out of
threads. Increasing the batch size in this scenario will only make the
situation worse. Decreasing it might eliminate the errors, but it will
not improve the performance of the solution.
To rectify this problem, you
need to increase the maximum number of connections for the host
instances hosting the SOAP Send Adapter. The recommended value is 25
connections × the number of CPUs on the server. This is done by adding a maxconnection
element to the ConnectionManagement node in the BTSNTSvc.exe.config
file on each of the BizTalk Servers in the BizTalk Server Group. This
file holds the common configuration for the BizTalk host services. If
the solution needs to call web services with performance issues that
cannot handle more than a specific number of concurrent transactions,
the same setting can be used to specify a low number of maximum
concurrent connections for that specific web service and ensure that you
do not overdrive that service to failure. Per the config file fragment
that follows, the maxconnection key could be defined multiple times with different name
attributes identifying different endpoints, thus allowing for the
customization of different maximum values for the concurrent number of
connections for different endpoints. To specify the default value used
for all web service endpoints not specifically called out by name, use *
as the name attribute for the default key value.
<configuration>
...
<system.net>
<connectionManagement>
<add name = "www.MyLowThroughputWebService.com" maxconnection = "4" />
<add name = "*" maxconnection = "50" />
</connectionManagement>
</system.net>
</configuration>
Web services located on the same
computer as your BizTalk solution share the same thread pool with web
services exposed by the BizTalk solution if running within the same
application pool. Therefore, the client-facing web services and the web
service being called through the SOAP Send Adapter share the same
threads and other related resources, such as CPU for request processing.
Calling a local web service also means that your request travels
through the entire processing pipeline and incurs overhead, including
serialization, thread switching, request queuing, and deserialization.
In addition, the maxconnection
attribute of machine.config has no effect on the connection limit for
making calls to local web services. Therefore, local web services always
tend to give preference to the requests that come from the local
computer over requests that come from other machines. This degrades the
throughput of the web service for remote clients (Meier, 2004). If the
local web services are not making calls to any external systems and
their web method processing time is considerably low, package them into a
.NET library and call them from within your orchestrations. If those
web services are calling external systems or take a considerable amount
of processing time, move them off the BizTalk Server Group servers.
By default, the .NET
thread pool used by BizTalk host instances is 100 threads per CPU. To
configure the maximum number of threads allocated by the thread pool,
set the MaxWorkerThreadsPerProcessor DWORD registry key under software\Microsoft\BizTalk Server\3.0\Administration.
1.3. HTP-Specific Tuning
Several configuration
and tuning parameters are accessible for the HTTP adapter through
registry key entries and through the modification of the
BTSNTSvc.exe.config file that is located in the root BizTalk
installation directory. Table 2
describes the registry settings that affect the performance of the HTTP
and WCF adapter. Note that by default there are no HTTP adapter keys in
the registry, so the HTTP adapter uses the default settings. To change
the default settings, you need to create the following registry keys
under the following locations in the registry:
DisableChunkEncoding, RequestQueueSize, and HttpReceiveThreadsPerCpu must be defined in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc.3.0\HttpReceive.
HttpOutTimeoutInterval, HttpOutInflightSize, and HttpOutCompleteSize must be defined in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc{$HostName}.
Table 2. HTTP Adapter Settings (Microsoft, "BizTalk Server 2009 Documentation," 2006)
| Key Name | Type | Default Value | Description |
|---|
| DisableChunkEncoding | DWORD | 0 | Regulates whether or not the HTTP Receive Adapter uses chunked encoding when sending responses back to the client.
Set to a nonzero value to turn off chunked encoding for HTTP Receive Adapter responses.
Minimum value: 0
Maximum value: Any nonzero value |
| RequestQueueSize | DWORD | 256 | Defines the number of concurrent requests that the HTTP Receive Adapter processes at one time.
Minimum value: 10
Maximum value: 2048 |
| HttpReceive ThreadsPerCpu | DWORD | 2 | Defines the number of threads per CPU that are allocated to the HTTP Receive Adapter.
Minimum value: 1
Maximum value: 10 |
| HttpOutTimeoutInterval | DWORD | 2,000 | Defines the interval in seconds that the HTTP Send Adapter will wait before timing out.
Minimum value: 500
Maximum value: 10,000,000 |
| HttpOutInflightSize | DWORD | 100 | This is the maximum number of concurrent HTTP requests that a BizTalk Server HTTP Send Adapter instance will handle.
The recommended value for latency is between three to five times that of the maxconnection configuration file entry.
Minimum value: 1
Maximum value: 1024 |
| HttpOutCompleteSize | DWORD | 5 | This
is the size of the batch of messages that is returned from the HTTP
Send Adapter. If the buffer is not full and there are outstanding
responses, the adapter will wait for 1 second until it commits the
batch. For low-latency scenarios, this should be set to 1, which will
allow the adapter to send response messages immediately to the
Messagebox for processing. This will have the greatest effect during
times of low-throughput activity with varied response times from backend
systems.
Minimum value: 1
Maximum value: 1024 |
The number of concurrent
connections that the HTTP or WCF adapter opens for a particular
destination server is configured by modifying the maxconnection entry in
the BTSNTSvc.exe.config file that is located in the root BizTalk
installation directory.
|
This property will be applied
to both the HTTP and SOAP adapters if they send messages to the same
destination HTTP server. By default the maximum connections for all URIs
is 20.
|
|
This configuration file entry replaces the functionality of the HttpOutMaxConnection
registry key that was used in BizTalk 2004. If you have upgraded from
BizTalk Server 2004 to BizTalk Server 2009 and you were using this
registry key, you will need to apply this configuration file entry
instead (Microsoft, 2006).
2. CLR Tuning
Bottlenecks caused
by contention for resources, misuse of threads, inefficient resource
cleanup, or resource leaks can be rectified by tuning the CLR thread
pool or memory thresholds. The use of memory thresholds will be
discussed later in the "Throttling" section.
In situations with low CPU
utilization or the CPU is fully saturated and yet the solution is not
meeting the required throughput, increasing the maximum number of
threads in the .NET thread pool by modifying the maxIOThreads and maxWorkerThreads
registry keys might improve performance. Tuning the maximum number of
threads in the thread pool down might come in handy if the CPU
utilization is pretty high, while the solution's overall throughput is
still lower than expected. This could be because the system is spending
more time context-switching between threads than processing.
If the solution is expected
to handle load bursts, prepping the engine to maintain a minimum number
of threads active to avoid the overhead of resources and thread
allocation when those bursts occur is a good idea. This is done by
setting the minIOThreads and minWorkerThreads
registry keys to ensure that a minimum number of threads are always
allocated in the thread pool. A value of the expected load during a
spike + 10% is usually the recommended value for the minIOThreads and minWorkerThreads settings.
To modify the hosted CLR
.NET thread pool for a particular BizTalk host, you have to create the
following registry keys and set their values for that particular host:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc{$HostName}\CLR Hosting\MaxWorkerThreads (REG_DWORD)
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc {$Host-Name}\CLR Hosting\MaxIOThreads (REG_DWORD)
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc {$Host-Name}\CLR Hosting\MinWorkerThreads (REG_DWORD)
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc {$Host-Name}\CLR Hosting\MinIOThreads (REG_DWORD)
