Troubleshooting Veeam Application-Aware Backups

Overview

In order to get transactionally consistent backups Veeam is capable of using several methods of application-aware guest OS processing. In some cases, issues with application-aware processing can cause backups to fail. These issues can be caused by issues with Microsoft Volume Shadow Copy (VSS), SQL log processing and/or truncation, Hyper-V integration services, or VMware tools.

Microsoft VSS

VSS Writer Failures

Failed to prepare guest for hot backup. 
Error: VSSControl: -1 Backup job failed. 
Cannot create a shadow copy of the volumes containing writer's data.

Sometimes Veeam can fail with errors similar to the message above. These are typically caused by failed VSS writers on the Guest OS.

To check for any failed VSS writers run the following command:

vssadmin list writers

In most cases, a reboot will resolve failed writers. Although in certain scenarios the writers can fail again the next time the backup runs. An alternative to a reboot is to restart the services associated with the failed VSS writer. A list of these services can be found here: KB2041: Windows VSS Writers and Corresponding Service Names (veeam.com)

VSS Storage Related Failures

VSSControl: -2147467259 Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. VSS asynchronous operation is not completed. Operation: [Shadow copies commit]. Code: [0x8004231f].

Unable to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Crash consistent). Details: Unknown status of async operation
The shadow copy provider had an unexpected error while trying to process the specified operation.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks.

Volume Shadow Copy Service error: Unexpected error calling routine Cannot find anymore diff area candidates for volume 

VSS requires free storage on the guest OS for space that can be used for storing shadow copies. The first thing to check is that there is enough free space on the volumes being backed up. Usually this is around 10% free space, but will depend on the rate of change and total size of the volumes.

If there is enough free space on the volumes, the next thing to check is that the “shadow storage” is configured. To find this open command prompt as administrator and run the following command:

vssadmin list shadowstorage

Make sure that each volume included in the backups is listed with storage configured. If any of the volumes are missing add shadow storage for them with the following command:

vssadmin add shadowstorage /For=C: /On=C: /MaxSize=20%
OR
vssadmin add shadowstorage /For=C: /On=C: /MaxSize=UNBOUNDED

If any of the volumes with configured shadow storage are configured for less than 10-20%, they can be increased with the following commands:

vssadmin resize shadowstorage /For=C: /On=C: /MaxSize=20%
OR
vssadmin resize shadowstorage /For=C: /On=C: /MaxSize=UNBOUNDED

To store shadow copies on a different volume, change the /On=C: flag to the other drive. (For example if the C: drive was close to full but the D: drive had significant free space, the flag would be /On=D:)

VSS Provider Failures

VSS_E_UNEXPECTED_PROVIDER_ERROR

Sometimes VSS providers can be left over from previous solutions that are no longer in place. In some cases, Veeam will run into problems initiating vss shadow copies when these providers are present. In most cases the fix for these issues is to remove the bad VSS provider.

Run the following command to check what providers are installed on the system:

vssadmin list providers

This will output a list of the providers, take note of the GUID of non-Microsoft providers or any from unused solutions. Next, open regedit to the following key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\

Match the previous GUIDs with the keys in the registry. Right click the keys and export them as a backup. Then delete the matching keys.

Next, restart the Volume Shadow Copy service (sometimes the machine must be rebooted for the changes to go into effect.)

VSS Timeout

VSSControl: Failed to prepare guest for freeze, wait timeout
VSSControl: Failed to freeze guest, wait timeout

Veeam has a knowledge base article with information on how to troubleshoot these types of warnings: KB1377: VSS wait timeout (veeam.com)

Multiple VSS Applications Conflict

Failed to create snapshot. Error code -2147212522. ‘Backup job failed.
Error VSS error: VSS_E_SNAPSHOT_SET_IN_PROGRESS. Code:0x80042316

Both of these warnings indicate that there are multiple applications attempting to use the same writers at the same time. It is recommended that you adjusting the time that you run your backup to not interfere with other applications that utilize the same VSS writers.

Veeam has a knowledge base article with information on how to troubleshoot these types of warnings:
KB1784: 0x80042316 or Error: VSSControl: -2147212522

VMware Tools

Guest processing skipped (check VMware Tools status).

This warning message occurs when Veeam tries to perform application aware processing or guest indexing, but it cannot communicate with the Guest OS of the VM being backed up. By default, Veeam will try to communicate over the network from the Guest interaction proxy specified in the backup job. But if that fails, Veeam will try to communicate through VMware tools.

This message is indicating that Vmware tools is either not installed or not running.

Please make sure that VMware tools is installed and running on the affected VM. If it is installed, but not running, you can typically restart the service from Windows Services. If it fails to start, reviewing the Windows Application logs can provide info on what is causing the service to fail to start.

Microsoft SQL

Failed to truncate Microsoft SQL Server transaction logs. Details: Failed to process 'TruncateSQLLog' command.

Failed to truncate SQL server transaction logs for instances:  SQLINSTANCE. See guest helper log.

Unable to truncate SQL server transaction logs. Details: Failed to truncate SQL server transaction logs for instances: MSSQLSERVER. Login failed for user 'DOMAIN\Bkpadmin'.  

Veeam has a knowledge base article with information on how to troubleshoot these types of warnings: KB2027: Job reports warning “Failed to truncate transaction logs for SQL instances: Possible reasons: lack of permissions, or transaction log corruption.” (veeam.com)

Hyper-V Integration

Error Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception)

Error: Hyper-V Integration Service is not accessible through the network on source host

Veeam has a knowledge base article with information on how to troubleshoot these types of warnings: KB1855: Hyper-V Guest processing skipped (check guest OS VSS state and integration components version) (veeam.com)

There is also a great blog post from Veeam on the process to install and setup the Hyper-V integration services: Hyper-V installation and configuration step-by-step (veeam.com)