Exchange Server 2010 : Managing Logging (part 2) - Managing Agent Logging & Managing Exchange Store Logging

3. Managing Agent Logging

You can use agent logs to record actions performed on a message by specified anti-spam agents that you have installed and configured on an Edge Transport or Hub Transport sever. The following agents can write information to the agent log:

• Connection and content filter agents

• Edge rules agent

• Recipient and sender filter agents

• Sender identity (ID) agent

Information written to the agent log depends on the agent, the SMTP event, and the action performed on the message. By default, agent logging is enabled on Hub Transport servers or Edge Transport servers. The following are default values:

• The path where the agent logs are stored; this is C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\AgentLog.

• The maximum size for the individual agent log files. This is 10 MB.

• The maximum size for the directory that contains the agent log files. This is 250 MB.

• The maximum age for the agent log files. This is 30 days.

Exchange Server 2010 uses circular logging to limit the agent logs based on file size and file age and to help limit the hard disk space used by the log files. If you need to keep agent log files longer than allowed by file age or directory size values, you should create a scheduled task that periodically moves agent log files to a different location for archiving.

3.1. Transport Agents

Transport agents act on messages at specific access points in the SMTP command sequence that transports messages through a Hub Transport server or Edge Transport server. These access points are called SMTP events. Each agent has a priority value, but SMTP events must always occur in a specific order, and agent priority depends on the SMTP event. If two agents act on a message during the same SMTP event, the agent that has the highest priority acts on the message first.

Table 1 lists SMTP events and the agents that write information to the agent log for each SMTP event.

The anti-spam agent logs on a Transport server are located in the folder C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\AgentLog. If no agent logging has occurred on the server, this folder will not exist. The naming convention for these files is AGENTLOGyyyymmdd-nnnn.log. The placeholder yyyymmdd is the UTC date when the log file was created. The variables yyyy = year, mm = month, and dd = day. The variable nnnn is an instance number that starts at the value of 1 for each day.

Information is written to the log file until the file size reaches 10 MB, at which point a new log file is opened with an incremented instance number. This process is repeated throughout the day. Circular logging deletes the oldest log files when the agent log directory reaches 250 MB or when a log file is 30 days old.

Agent log files are text files that contain data in CSV format. Each agent log file has a header that contains the same information that was previously described for protocol log files earlier in this lesson. Specifically, the headers are #Software, #Version, #Log-Type, #Date, and #Fields.

The agent log stores each agent transaction on a single line in the log. The information stored on each line is organized by fields. These fields are separated by commas. Typically, the field name is sufficiently descriptive to determine the type of information it contains, such as Timestamp, SessionId, MessageId, Recipient, and so on. Some of the fields may be blank, and the type of information stored in the field may change based on the agent or the action performed on the message by the agent.

3.3. Searching Agent Logs

You can use commands based on the Get-AgentLog EMS cmdlet on an Edge or Hub Transport server to access the information in an agent log. For example, the following command returns a report containing statistics collected between 09:00 (9:00 AM), October 4, 2010, and 18:00 (6:00 PM), November 5, 2010:

Get-AgentLog -StartDate "10/04/2010 9:00:00 AM" -EndDate "11/05/2010 6:00:00 PM"

					  

If the C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\AgentLog folder does not exist on the server, this command returns an error. You can also use the Get-AntiSpamFilteringReport.ps1 script to search the agent logs and obtain anti-spam filtering statistics. The script displays statistics for each agent, taking one of the following values as a mandatory parameter:

• Connections

• Commands

• MessageRejected

• Messagesdeleted

• Messagesquarantined

By default, agent logging is enabled on a Hub Transport or an Edge Transport server. You can enable or disable agent logging by modifying the EdgeTransport.exe.config file located in the folder C:\Program Files\Microsoft\Exchange Server\V14\Bin. By default, only the AgentLogEnabled key is present in the EdgeTransport.exe.config file.

To configure the maximum file size for individual logs, the maximum age of the agent log files, and the maximum size for the agent log directory, you need to add the AgentLogMaxFileSize, AgentLogMaxAge, and AgentLogMaxDirectorySize keys. You do this in the practice session later in this lesson. The files EdgeTransport.exe.config and MSExchangeTransport.exe are used by the Microsoft Exchange Transport service. You need to restart this service before any configuration changes you make take effect.

Note:

CONFIGURABLE AGENT LOGGING SETTINGS

Because the AgentLogMaxFileSize, AgentLogMaxAge, and AgentLogMaxDirectorySize keys do not exist by default in the EdgeTransport.exe.config file, some documentation states that the only configurable option for agent logging is to disable or enable it. However, if you add the keys to the file, as described in the practice session later in this lesson, you can then configure the maximum file size for individual logs, the maximum age of the agent log files, and the maximum size for the agent log directory.

Remember that agent logging assists in spam filtering and is configured by amending a configuration file rather than by using EMS commands.

4. Managing Exchange Store Logging

The Exchange store provides a single repository for managing several types of information and is the core data storage repository for Exchange Server 2010. It contains mailbox databases and public folder databases that can reside on a single server or can be distributed across multiple servers.

Exchange Server 2010 stores data in a specialized set of data files, such as Exchange database (EDB) files, transaction logging (LOG) files, and checkpoint (CHK) files. Exchange Server 2010 writes operations such as creating or modifying a message to a LOG file for a specific database. Committed transactions are later written to an EDB file in the database itself. All completed and in-progress transactions are logged, so data integrity is maintained in case of a service interruption. The databases share a single set of transaction logs that are named using consecutive numbers (for example, E0000000001.log and E0000000002.log).

Checkpoint files store information that indicates when a transaction is successfully saved to the database files on the hard disk. Exchange Server 2010 uses checkpoint files to allow an instance of the Extensible Storage Engine (ESE) to automatically replay log files into an inconsistent database when recovering from a service interruption, starting with the next unwritten transaction. The checkpoint files are placed in the same log location as the log files.

4.1. Transaction Logging

Exchange store logging, or transaction logging, is an ESE recovery mechanism designed to restore an Exchange database to a consistent state after any sudden stoppage. The logging mechanism is also used when restoring online backups.  Before changes are made to an Exchange database file, Exchange writes the changes to a transaction log file. After a change has been logged, it can be written to the database file. Changes are securely written to a transaction log file and can be made available if required.

Changes are not lost when a database suddenly stops. When the database restarts, Exchange scans the log files and reconstructs and applies any changes not yet written to the database file. This process is called replaying log files. Exchange can determine whether any operation in a log file has already been applied to the database, whether it needs to be applied to the database, or whether it does not belong to the database.

Exchange uses a series of log files, each exactly 1 MB in size. When a log file is full, Exchange closes it and renames it with a sequential number. The first log that is filled ends with the name Enn00000001.log, where the variable nn refers to a two-digit number known as the base name or log prefix.

Log files for each database are distinguished by file names with sequentially numbered prefixes (for example, E00, E01, E02, or E03). The log file currently open for a database is named Enn.log—it does not have a sequence number until it has been filled and closed. The checkpoint file (Enn.chk) tracks how far Exchange has progressed in writing logged information to the database files. Each log stream has a checkpoint file, and all the databases share a single log stream. Thus, a single log file often contains operations for multiple databases.

Note:

LOG FILE NUMBERING

Log files are numbered in a hexadecimal manner, so the log file after E0000000009.log is E000000000A.log, not E0000000010.log.

You can examine the header of a log file by using the Eseutil utility. For example, the following command redirects header information for the log file E000000000B.log into the text file C:\logheader.txt:

Eseutil /ml E000000000B.log > C:\Logheader.txt

The information in a log file header tells you the base name of the file and whether it is the current log file. If the file is current, the header information tells you what its sequence number will be when it is filled and closed. The Checkpoint value in the header tells you how far into the log file the checkpoint is. If the checkpoint file has been destroyed, the Checkpoint value reads NOT AVAILABLE. In this event, Exchange can still recover and replay log files appropriately. To do so, it scans all the log files, beginning with the oldest file available. Exchange ignores data that has already been applied to the database and works sequentially through the logs until it encounters data that needs to be applied.

Figure 4 shows some lines of log file header information that has been redirected into a text file. You know that this is the current log file because the log file name does not have a sequence number. The 1Generation value shows that when the log is filled and closed, its sequence number will be A, corresponding to the decimal value 10. The base name is e01, and therefore the final log file name will be E010000000A.log.

When an Exchange database shuts down normally, all outstanding data is written to the database files. After normal (clean) shutdown, the database file set is considered to be consistent, and Exchange detaches it from its log stream. This means that the database files are now completely up to date. The transaction logs are not required to start consistent database files.

Note:

ONLINE BACKUP AND RESTORE

In addition to allowing Exchange to recover reliably from an unexpected database stop, transaction logging is also essential to making and restoring online backups. 

4.2. Circular Logging

You can configure Exchange Server 2010 to save disk space by enabling circular logging, although Microsoft does not recommend this as a best practice. Circular logging allows Exchange to overwrite transaction log files after the data that the log files contain has been committed to the database. However, if circular logging is enabled, you can recover data only up to the last full backup.

In standard transaction logging, each database transaction is written to a log file and then to the database. When a log file reaches 1 MB in size, it is renamed, and a new log file is created. Over time, this results in a set of log files. If Exchange stops unexpectedly, you can recover the transactions by replaying the data from these log files into the database. Circular logging overwrites and reuses the first log file after the data it contains has been written to the database.

Circular logging is disabled by default on Exchange databases. By enabling it, you reduce drive storage space requirements. However, without a complete set of transaction log files, you cannot recover any data more recent than the last full backup.