Warning

Just recently, right after upgrading to Veeam 9.5, we ran into an error with one of our customers that would show up whenever backups started to run and when we tried to rescan the Veeam repository. The errors messages were: Warning Failed to synchronize Backup Repository Details: Failed to synchronize 2 backups Failed to synchronize 2 backups Warning Failed to import backup path E:|PATH|TO|BACKUP|FILES|BackupJobName.vbm Details: Incorrect file system item link received: BackupJobName Based on the errors it looked like there were issues with the database entries for the backup job mentioned in the error. As a troubleshooting step we tried removing the backup files from the GUI under Backup & Replication>Backups by right clicking on the backup and selecting ‘Remove from Configuration.’ However, this ended up giving us the same error in a popup dialogue: Incorrect file system item link received: BackupJobName After opening a ticket with Veeam they informed us that this is a known issue caused by unexpected backup entries in the VeeamBackup SQL database. Namely, the issue backup jobs were listed in the database with a zero entry for the job ID or repository ID causing Veeam to not be able to locate the backup files. Because this involves the Veeam SQL database we are going to be making changes to the database so it’s best to back it up before the next steps. Here’s a knowledge base article from Veeam that shows the suggested methods to backing up the database. To determine whether there are any errant backup entries in the SQL database run the following query: SELECT TOP 1000 [id] ,[job_id] ,[job_name] ,[repository_id] FROM [VeeamBackup].[dbo].[Backup.Model.Backups] Under ‘repository_id’ you should see one or more of the backup jobs showing ‘00000000-0000-0000-0000-000000000000’ or ‘NULL’ as the id for job or repository. Any entries with this issue will need to be removed from the database in order to resolve the error. It’s best practice to save a backup of the SQL database before making any changes. If you’re unsure of how to back up the SQL database, follow Veeam’s knowledge base article.  After backing up the SQL database run the following query for each job with ‘00000000-0000-0000-0000-000000000000’ as the repository_id: EXEC [dbo].[Backup.Model.DeleteBackupChildEntities] ‘REPLACE_WITH_ID’ EXEC [dbo].[Backup.Model.DeleteBackup] ‘REPLACE_WITH_ID’ Replacing the ‘job_id’ with that of any backups with the unexpected ‘repository_id’ found with the previous query. After that issues with the local backup server was resolved. However, we were still seeing errors when trying to connect the backup server to our cloud connect repository to do backup copies. We were still getting the following errors: Warning Failed to synchronize Backup Repository Details: Failed to synchronize 2 backups Failed to synchronize 2 backups Warning Failed to import backup path E:|PATH|TO|BACKUP|FILES|BackupJobName.vbm Details: Incorrect file system item link received: BackupJobName In order to resolve this we had to remove the entries for the problem jobs from the cloud connect server database. For those using a cloud connect service provider you will have to have them make changes to the SQL database. We had 2 VMs that were giving the ‘Incorrect file system item link received: JobName” error. So we had to remove any entries for those jobs from the SQL db. We ran the following query to get the Job ID of both jobs mentioned in the errors: SELECT TOP 1000 [id] ,[job_id] ,[job_name] FROM [VeeamBackup].[dbo].[Backup.Model.Backups] Then we ran the same delete query as before using the new job_id’s: EXEC [dbo].[Backup.Model.DeleteBackupChildEntities] ‘REPLACE_WITH_ID’ EXEC [dbo].[Backup.Model.DeleteBackup] ‘REPLACE_WITH_ID’ After those entries were deleted we were able to rescan the repository. Lastly, once we rescanned our cloud repository and imported the existing backups we started getting the following error message: Warning Failed to import backup path E:|PATH|TO|BACKUP|FILES|BackupJobName.vbm Details: Path E:|PATH|TO|BACKUP|FILES|BackupJobName2016-11-21T220000.vib is absolute and cannot be added to other path This error indicates that there is a problem where the backup chain isn’t accurately resolving the location of the GFS restore points. In order to resolve this we had to manually remove the Imported backups from the cloud connect server by going to Backup & Replication>Backups and selecting the backup job and choosing ‘Remove from Configuration’ making sure to check ‘Include archived full backups.’ After the backups had been removed from the cloud connect repository we were able to manually rescan the repository on the local backup server and the backup files were imported again successfully. Update:  After deleting these affected rows using the commands above you may get the following error message: Unable to map tenant cache ID ‘ID-GOES-HERE’ to service provider side ID ‘ID-GOES-HERE’ because it is already mapped to ID ‘ID-GOES-HERE’ If you do see this error a solution can be found in this Veeam KB article. Essentially, the data for these deleted rows still exists in the Veeam DB cache table and needs to be removed. In order to do this run the following query on the VeeamBackup database: delete from [cachedobjectsidmapping] This will delete the DB cache table. In order to re-populate it you will need to rescan the service provider and the Cloud Repository.