Recovering from a Disaster in an Exchange Server 2007 Environment : Recovering from Database Corruption

A couple of methods can be used to repair a corrupt Exchange database, or at least restore the database and extract good information from the database. Key to the successful recovery of as much information as possible is using the right tool. In many cases, administrators jump right into using the ESEUTIL /p repair command; instead of repairing the Exchange database to 100% condition, the utility finds a corrupt section of the database and deletes all information from that portion of the database on. So, although the Exchange system becomes 100% clean, the utility deleted 20%–30% of the data that was in the database to get the database to a clean state. The ESEUTIL /P command is the task of last resort: Other tools work around corrupt database areas and allow the administrator to recover as much of the data as possible.

Exchange 2007 makes this process easier for the new administrator by introducing the Database Recovery Management tool that will analyze the failed databases for the administrator and choose what tools to run against them.

Using the Database Recovery Management Tool

Exchange 2007 brings several new tools to make life easier for the Exchange administrator. The new Database Recovery Management tool automates many of the processes of evaluating corrupted databases and repairing them. To utilize this tool with a corrupted database, perform the following steps:

1.	Open EMC.
2.	Expand tools.
3.	Choose Database Recovery Management.
4.	Let the tool check for updates.
5.	Choose the Go to Welcome screen.
6.	Type in a label for this activity.
7.	Enter server and domain controller names, and click Next.
8.	If the connectivity tests are acceptable, click Gather Database Information from AD.
9.	Click Troubleshoot Database Mount Problem.
10.	Select the storage group you want to troubleshoot, and click Next.
11.	Select the database you are troubleshooting, and click Analyze Selected Databases.
12.	View the information given.
13.	Click Go Back to Task Center.
14.	Click Repair Database Wizard.
15.	Select the storage group, and click Next.
16.	Select the database and click Next.
17.	View the repair results as shown in Figure 1. Figure 1. Viewing the repair results.
18.	Click Go Back to Task Center.
19.	Close the Troubleshooting Assistant.

Flat-File Copying the Exchange Databases

One of the best techniques Exchange experts use when working to recover corruption in a database is to make a flat-file copy of the Exchange databases. A flat-file copy is merely an exact copy of the Exchange databases copied to another portion of the server hard drive or to another server. To do a manual copy of the databases, do the following:

Moving Mailboxes to Another Server in the Site

One way of extracting mail from a corrupt database is to move the mailbox or mailboxes to a different server in the site. Instead of trying to run utilities to fix the corruption in the database, which can take several hours (or even days, depending on the size of the database and the amount of corruption that needs to be fixed), an administrator can set up another server in the Exchange site and move the mailboxes to a new server.

Moving mailboxes grabs all of the mail, calendar, contacts, and other mailbox information from one server and moves the information to a new server. As the information is written to the new server, the information is automatically defragmented and corruption is not migrated. In addition, mailboxes can be moved from one server to another without ever having to bring down the production server. A mailbox user must be logged off Outlook and must not be accessing Exchange before the mailbox can be moved. However, if mailboxes are moved when individuals are out of the office, at lunch, or on weekends, the mailboxes can be moved without users ever knowing that their information was moved from one system to another.

The two caveats to moving mailboxes are these: Corrupt mailboxes will not move, and user Outlook profiles will be changed. For Outlook profiles, because a user’s Outlook profiles point to a specific server, when a mailbox is moved from one server to another, the user’s profile also needs to point to the new server. Fortunately, with Exchange and Outlook, when a user’s mailbox is moved, Outlook tries to access the mailbox on the original server, and the server notifies Outlook that the mailbox has been moved to a new Exchange server. The user’s Outlook profile automatically changes to associate the profile to the new server where the user’s mailbox resides. So, as long as the old server remains operational and the user attempts to access email from the old server, the profiles will be automatically changed the next time the user tries to access email. Typically, within 1 to 2 weeks after moving mailboxes from one server to another, the user profiles are all automatically changed.

As for corrupt mailboxes, unfortunately, Exchange typically does not move a corrupt mailbox. So, if a user’s mailbox has been corrupted, the mailbox will remain on the old server. However, if 80%–90% of the user mailboxes can be moved to a new server, the administrators are trying to recover only a handful of mailboxes instead of all mailboxes on a server. This could mean far less downtime for all users who had mail on the server and could limit the exposure of data loss to a limited number of users. It will also result in much faster results when running the database recovery tools as the database will be much smaller.

To move mailboxes between servers in a site, do the following:

Running the ISINTEG and ESEUTIL Utilities

When a database is determined to be corrupt, usually an administrator is directed to run the built-in utilities on Exchange to run maintenance on the databases. The utilities are the ISINTEG (“eye-ess-in-tehg”) and ESEUTIL (“ee-ess-ee-u-tihl”). However, depending on the condition of the database, a very corrupt database can take several hours to run, only to result in the loss of data. Some administrators are incorrectly told to never run the utilities because they will always result in loss of data. It’s typically just a lack of knowledge of how the utilities work that leads to misunderstanding the potential results of the databases.

As noted in the previous two sections, there might be better options for recovering information from a corrupt database. Instead of trying to fix a known corrupt database, simply migrating the information off a server or extracting information from corrupt databases is frequently a better fix. However, if the determination is to run the utilities, a few things should be noted:

• The ISINTEG utility is a high-level utility that checks the consistency of the database, validating the branches of the database that handle data, data directory tables, attachment objects, and the like. Fixing the database table makes way for a more intensive data integrity check of the database.

• The ESEUTIL utility is a low-level utility that checks the data within the database. ESEUTIL does not differentiate between a corrupt section of the database and how that section impacts mailboxes or messages. So, when a complete repair is performed using ESEUTIL, entire mailboxes can be deleted or all attachments for the entire database can be eliminated to fix the corruption. This is why running ESEUTIL to repair a database is a function of last resort.

• To run ISINTEG on a database takes around 1 hour for every 10GB being scanned for a moderately corrupt database. The repairs are done relatively quickly, and the database is ready for more extensive scanning.

• Running ESEUTIL on a database takes anywhere from 1 hour for every 10GB to up to 1 hour per 1GB, depending on the level of repair being performed. It is not unreasonable to see a relatively corrupt 30-GB database take more than 24 hours to complete the repair.

• ISINTEG and ESEUTIL can be performed only offline, meaning that the Exchange server is offline during the process. Users cannot access their mailboxes during the ISINTEG and ESEUTIL processes. Thus, if it takes 20 to 40 hours of downtime to complete the repair of a database, the Move Mailbox method that can be run without bringing servers offline is frequently a more palatable solution.

• However, if run on a regular basis, the ISINTEG and ESEUTIL utilities can clean up an Exchange database before serious corruption occurs. Administrators who get scared off performing maintenance because of the potential threat of losing data could actually minimize their chance of data corruption if the utilities are run regularly. 

The common parameters used for the ISINTEG and ESEUTIL utilities are as follows. For regular maintenance such as checking the database structure’s integrity and performing defragmentation of the database, the following commands should be run:

isinteg –s SERVERNAME –test allfoldertests
eseutil /d priv1.edb

When run against an Exchange server, the ISINTEG utility produces a summary similar to the one in Figure 2.

Figure 2. Results from an ISINTEG utility run.

When a database needs to be repaired, eseutil /p priv1.edb can be run. Beware: The /p repair command is a brute-force repair and deletes sections of the database to make the integrity of the database clean. A message provides an additional warning about ESEUTIL, as shown in Figure 3. When running the /p command in ESEUTIL, entire sections of the database might be deleted to repair and recover the state of the database.