BizTalk 2010 Recipes : Messaging and Pipelines - Creating Flat File Send and Receive Pipelines

3/31/2011 9:46:32 PM

1. Problem

You are integrating two systems that can communicate only via flat file messages and must provide the mechanisms for receiving and sending the messages within BizTalk Server.

2. Solution

Create a flat file receive and send pipeline, as follows (these steps assume that a flat file schema has already been created):

Open a project that contains a flat file schema.
Right-click the project, and select Add => New Item.
In the Add New Item dialog box, select Receive Pipeline from the list of templates. Give a descriptive name to your new pipeline, as shown in Figure 1, and then click Add.

Figure 1. Adding a new receive pipeline to a project
From the BizTalk Pipeline Components section of the toolbox, drag and drop a Flat File Disassembler component into the Disassemble stage of the receive pipeline.
With the Flat File Disassembler component highlighted, set the Document schema property in the Properties window to the flat file schema (CustomerSalesOrder in the example), as shown in Figure 2.

Figure 2. Configuring a receive pipeline for a flat file schema

NOTE

Setting the Document schema property is required when using the Flat File Disassembler component. All other properties are optional.
Save the receive pipeline.
Now create the Send Pipeline. Right-click the project, and select Add => New Item.
In the Add New Item dialog box, choose Send Pipeline from the list of templates. Give a descriptive name to your new pipeline, as shown in Figure 3, and then click Add.

Figure 3. Adding a new send pipeline to a project
From the Pipeline Components section of the toolbox, drag and drop a Flat file assembler component into the Assemble stage of the send pipeline.
With the Flat File Assembler component highlighted, set the Document schema property in the Properties window to the flat file schema, as shown in Figure 4.
NOTE

If the Document schema property is not specified, runtime schema discovery will be attempted. During schema discovery, BizTalk Server attempts to determine the correct flat file schema to assemble the message with, based on the namespace and root node of the message.

Figure 4. Configuring a send pipeline
Save the send pipeline.
Build and deploy the solution.

Once the flat file schema and pipelines (CustomerSalesOrder, CustFlatFileReceive, and CustFlatFileSend in this example) have been deployed, BizTalk Server can use them in message processing.

3. How It Works

Many systems in use today were designed and engineered prior to the advent of XML. To communicate with these systems, it is often necessary to use flat file formats such as positional, delimited, or a combination of the two. In our example, a simple positional flat file message is parsed into XML as BizTalk Server receives the document. BizTalk Server then assembles the XML back into its correct flat file format when sending the message to the appropriate subscribers.

More specifically, the custom receive pipeline created in the example (CustFlatFileReceive) is used in a receive location that retrieves inbound flat file messages. This receive pipeline disassembles the flat file message into an XML document by leveraging the out-of-the-box Flat File Disassembler pipeline component. By setting the Document schema property on the Flat File Disassembler component, BizTalk Server is instructed to apply the correct schema to inbound documents, thereby producing the appropriate XML representation of the original flat file.

Figure 5 represents one possible instance of the inbound flat file document.

Figure 5. Example instance of the flat file

When passed through the FlatFileReceive pipeline created in the solution (which can be implemented via a receive location), BizTalk Server disassembles the flat file into the following XML message, based on the details defined within the CustomerSalesOrder.xsd schema (see Figure 6).

Figure 6. Example instance of the XML

Since the CustFlatFileSend pipeline (which can be implemented via a send port) is configured to assemble instances of the CustomerSalesOrder.xsd message type back into flat file format, the original flat file document would be output again .

As well as the required Document schema property, you can configure other optional properties for the Flat File Disassembler component, as shown in Table 1.

Table 1. Flat File Disassembler Optional Properties
Property	Description
Header schema	Defines the schema to apply to any header included in the flat file message.
Preserve header	Specifies if the flat file header should be stored in the message context (properties). The message context contains metadata about the message and is attached to the message throughout its lifetime in BizTalk Server.
Recoverable interchange processing	Specifies if the flat file disassembler should attempt to recover from errors occurring during message processing.
Trailer schema	Defines the schema to apply to any trailer included in the flat file message.
Validate Document Structure	Specifies if the configured schemas (document, header, and trailer schemas) should be used to validate their respective message parts (document, header, and trailer, respectively).

The custom send pipeline created is then used in a send port to deliver flat file messages to subscribers. This send pipeline assembles the XML messages back into flat file format by using the Flat File Assembler pipeline component provided with BizTalk Server. By setting the Document schema property on the Flat File Assembler component, BizTalk Server is instructed to apply the correct schema to outbound documents, thereby producing the appropriate flat file representations of the original XML document.

Along with the required Document schema property, you can set the optional properties listed in Table 2 for the Flat File Assembler component.

Table 2. Flat File Assembler Optional Properties
Property	Description
Header schema	Defines the schema to apply to the header portion of the flat file message. If a header schema is defined on an instance message via the XMLNORM. HeaderSpecName context property, it will override this property setting.
Preserve byte order mark	Specifies if a byte order mark (BOM) should be added to messages passing through the pipeline.
Target charset	Specifies the target character set, which is used as the outbound message is encoded.
Trailer schema	Defines the schema to apply to the trailer portion of the flat file message.

You can configure encoding of outbound messages in a number of different locations, and you should understand the logic BizTalk Server uses to determine which to use if there are discrepancies. Table 3 lists the methods that can be used to specify message encoding, ordered to reflect their precedence. If none of the methods listed in Table 3 is used to configure message encoding, UTF-8 is used.

Table 3. Methods for Encoding Outbound Messages
Method	Description
Message– XMLNORM.TargetCharset	This message context property takes precedence over all other encoding configurations and will be used if its value is set.
Component– Target Charset	This Flat File Assembler pipeline component property will be used if its value is set and none of the higher precedence (XMLNORM.TargetCharset) values are set.
Schema– CodePage	This schema property will be used if its value is set and none of the higher precedence values (Target Charset or XMLNORM.TargetCharset) are set.
Message– XMLNORM.SourceCharset	This message context property will be used if its value is set and none of the higher precedence values (CodePage, Target Charset, or XMLNORM.TargetCharset) are set.

You can also configure flat files through their message headers. In addition to configuring the header schema via the Flat File Disassembler or Flat File Assembler pipeline components, you can use the XMLNORM.HeaderSpecName message context property.

One common challenge encountered when receiving flat file messages from source systems is determining which schema to apply to different messages. This is particularly important when multiple message types (for example, customer and order message types) are received from the same location. Since flat file messages are not inherently self-describing, as XML messages are, it can be difficult to determine which flat file message schema to apply to the inbound messages. In these scenarios, you can use a solution similar to the Schema Resolver Component SDK example that comes with BizTalk Server. This example shows how to implement a custom Flat File Disassembler pipeline component to dynamically associate message types to inbound instances based on an identifier contained in each instance message.

Finally, to assist in testing the assembling and disassembling of flat file messages, the FFAsm.exe and FFDasm.exe tools are provided with BizTalk Server. These tools allow you to run the flat file assembler and disassembler directly against instance messages. These tools can be particularly useful and time-saving when working with large or complex flat file schemas.