Contact Us

Author name: Managecast Technologies

General Cloud Backup

VSS Troubleshooting Guide

/

A VSS (Volume Shadow Copy Service) is a Windows feature that enables users to create backup copies or snapshots of data, even while it’s in use. When VSS malfunctions, it can lead to data loss or corruption. Here’s a troubleshooting guide to help you fix common VSS issues: By following these steps, you should be able to troubleshoot most VSS issues. However, if the problem persists, Managecast may be able to help. Contact us today at sales@managecast.com or 513-735-6868.

Linux, Veeam

File level recovery of OES/NSS Volumes – Veeam Agent for Linux (VAL) 

/

It was observed that when attempting to use Veeam B&R to deploy and manage a Veeam Agent for Linux backup job, that Veeam does not display any data within those volumes when attempting to perform a file-level recovery of that backup data. Furthermore, Veeam does NOT back up any of the data on those volumes.   To effectively back up and restore the data on an OES/NSS volume, we have found the following process to be successful. While you will not be able to utilize the Veeam Explorer/File Level Recovery wizard to perform restorations of those files, the veeamconfig commands make it quick and easy to perform file-level recovery of data on OES/NSS volumes.   Build the Backup Job with the Veeam Backup & Replication console  Job Mode  Select Server type here, and agent managed.   It is critical that we build the job in Managed by agent mode; if we select the job to be managed by the backup server, we will be unable to mount the backup later in the restoration process due to file permission restrictions.  Name the job to your desire, and select the appropriate machines to be backed up. To better understand creating protection groups for your VAL servers, please refer to Veeam’s documentation here — Creating Protection Groups – Veeam Agent Management Guide. Backup Mode For backup mode, we will select file level backup with the option checked to backup directly from the live file system. Objects For objects, this is where you will select the NSS volumes in which you desire to be backed up. For this demonstration, we will specify two directories that we need to be protected. Target For the target location for the backup, we are using the Veeam Backup & Replication server in which we are using to build the job. This is to your desire, though for centralized backups, we recommend keeping all backups on your main backup repository. Guest Processing For guest processing, we want to enable Application Aware Image Processing. This will utilize the credentials configured in the protection group for the agent’s installation. After the backup has processed, we can proceed to the next section, where we will restore files from the CLI of the machine in which we are protecting. Restoring Files and Folders on Linux protected by the Veeam Agent for Linux (VAL) The following text was excerpted from the Veeam Agent for Linux 5.0 official documentation, which can be viewed here — Restoring Files and Folders with Command Line Interface – Veeam Agent for Linux User Guide If some files and folders on your computer get lost or corrupted, you can restore them from backups. For file-level restore, you can use backups of any type: When you perform file-level restore, Veeam Agent publishes the backup content directly into the computer file system. You can browse to files and folders in the backup, restore files and folders to their initial location, copy files and folders to a new location or simply target applications to restored files and work with them as usual. With the Veeam Agent command line interface, you can restore files and folders in a more flexible way than with the use of the File Level Restore wizard. In particular, you can specify a directory in which Veeam Agent should mount the backup file content for file-level restore. You can also mount several backups to different directories to work with files and folders restored from different backups simultaneously. Before You Begin Before you begin the file-level restore process, check the following prerequisites: Step 1. Locate Backup To view a list of backups created by Veeam Agent, use the following command: In the list of backups, Veeam Agent displays information about all Veeam Agent for Linux backups stored in all backup repositories. If Veeam Agent is connected to a Veeam backup server, all Veeam Agent for Linux backups that are kept on Veeam backup repositories managed by this server also appear in this list. The –all parameter is optional. If you do not use this option, Veeam Agent displays information about backups created by the current Veeam Agent computer only. For security reasons, if you work with Veeam Agent connected to a Veeam backup server as a member of a protection group for pre-installed Veeam Agents, the veeamconfig backup list –all command will display backups created only by the current Veeam Agent computer with the current connection settings. For each backup, Veeam Agent displays the following information: Parameter Description Job name Host name of the computer on which the backup job was configured and name of the job by which the backup was created. Backup ID ID of the backup. Repository Name of the backup repository in which the backup was created. Imported backups are marked as Imported in the Repository column. Created at Date and time of the backup creation. For example: Step 2. Explore Backup Content For file-level restore, you can use backups of any type: To view detailed information about specific backup, use the following command: where: <backup_id> — ID of the backup for which you want to view detailed information. For a volume-level backup, Veeam Agent for Linux displays the following information: Parameter Description Machine name Host name of the computer on which the backup job was configured and name of the job by which the backup was created. Name Name of the volume in the backup. Device Path to the block device that represents the volume FS UUID File system ID Offset Position of the volume on the computer disk Size Size of the volume in the backup For a file-level backup, Veeam Agent for Linux displays the following information: Parameter Description Machine name Host name of the machine on which the backup job is configured and the name of the job. Backed up Backup scope for the file-level backup job For example: Step 3: Mount Backup To mount a backup for file-level restore, use the following command: where: <backup_id> — ID of the backup that you want to

General Cloud Backup

World Backup Day is March 31st

/

World Backup Day is coming up quickly this year. WBD 2023 takes place on March 31st, the day before April Fool’s Day, for a good reason: Don’t be a fool backup your data! What is World Backup Day? World Backup Day is the day to bring attention to whether your data, most likely your most critical and vital business asset, is well protected. World Backup Day originated in 2005 as World Backup month, but was changed to World Backup Day in 2011 on March 31st as a reference to April Fool’s Day. Why Should You Backup Your Data? If you had to choose between all of the money in the bank or all of the data that your organization has, what would you choose? In some cases, choosing all the money would make sense, but in most cases, data is what helps the company stay going and keep making money for years to come. Backing up your data is important for many reasons, including enabling your organization to go on as usual in the event of a disaster. Disasters can happen anywhere: from a critical server failing, to natural disasters like flooding and fires, to man-made disasters such as theft, sabotage, and chemical spills. Additionally, ransomware attacks are on the rise globally and are evolving to be more complex. Data backup is also important in an event like a pandemic, which we are currently experiencing. In this case, suffering an additional disaster such as fire, floods, tornadoes or ransomware could cripple an entire business. Supply chains are stressed, getting replacement equipment may be difficult and more problems would arise. Plus, many workers are going remote since the start of 2020, and you need to be sure their data is protected. Now more than ever, we have seen why a good disaster recovery and business continuity plan is imperative. What Can I Do to Protect My Data? We encourage you to think about all of the data you need to protect, and whether or not you are meeting best practices. Are you following the simple 3-2-1 rule of backup, which states you should have 3 copies of your data, on 2 different media with at least one offsite? In recognition of World Backup Day, please consider how protected your data really is. We are here to help improve your data protection methods.  Feel free to contact us here, send me an email at ngolden@managecast.com, or give me a call at 513-735-6868 Ext. 4002 to request more information. Join us in taking the pledge here: http://www.worldbackupday.com/en/

Veeam

Troubleshooting Veeam Application-Aware Backups

/

Overview In order to get transactionally consistent backups Veeam is capable of using several methods of application-aware guest OS processing. In some cases, issues with application-aware processing can cause backups to fail. These issues can be caused by issues with Microsoft Volume Shadow Copy (VSS), SQL log processing and/or truncation, Hyper-V integration services, or VMware tools. Microsoft VSS VSS Writer Failures Sometimes Veeam can fail with errors similar to the message above. These are typically caused by failed VSS writers on the Guest OS. To check for any failed VSS writers run the following command: In most cases, a reboot will resolve failed writers. Although in certain scenarios the writers can fail again the next time the backup runs. An alternative to a reboot is to restart the services associated with the failed VSS writer. A list of these services can be found here: KB2041: Windows VSS Writers and Corresponding Service Names (veeam.com) VSS Storage Related Failures VSS requires free storage on the guest OS for space that can be used for storing shadow copies. The first thing to check is that there is enough free space on the volumes being backed up. Usually this is around 10% free space, but will depend on the rate of change and total size of the volumes. If there is enough free space on the volumes, the next thing to check is that the “shadow storage” is configured. To find this open command prompt as administrator and run the following command: Make sure that each volume included in the backups is listed with storage configured. If any of the volumes are missing add shadow storage for them with the following command: If any of the volumes with configured shadow storage are configured for less than 10-20%, they can be increased with the following commands: To store shadow copies on a different volume, change the /On=C: flag to the other drive. (For example if the C: drive was close to full but the D: drive had significant free space, the flag would be /On=D:) VSS Provider Failures Sometimes VSS providers can be left over from previous solutions that are no longer in place. In some cases, Veeam will run into problems initiating vss shadow copies when these providers are present. In most cases the fix for these issues is to remove the bad VSS provider. Run the following command to check what providers are installed on the system: This will output a list of the providers, take note of the GUID of non-Microsoft providers or any from unused solutions. Next, open regedit to the following key: Match the previous GUIDs with the keys in the registry. Right click the keys and export them as a backup. Then delete the matching keys. Next, restart the Volume Shadow Copy service (sometimes the machine must be rebooted for the changes to go into effect.) VSS Timeout Veeam has a knowledge base article with information on how to troubleshoot these types of warnings: KB1377: VSS wait timeout (veeam.com) Multiple VSS Applications Conflict Failed to create snapshot. Error code -2147212522. ‘Backup job failed.Error VSS error: VSS_E_SNAPSHOT_SET_IN_PROGRESS. Code:0x80042316 Both of these warnings indicate that there are multiple applications attempting to use the same writers at the same time. It is recommended that you adjusting the time that you run your backup to not interfere with other applications that utilize the same VSS writers. Veeam has a knowledge base article with information on how to troubleshoot these types of warnings:KB1784: 0x80042316 or Error: VSSControl: -2147212522 VMware Tools This warning message occurs when Veeam tries to perform application aware processing or guest indexing, but it cannot communicate with the Guest OS of the VM being backed up. By default, Veeam will try to communicate over the network from the Guest interaction proxy specified in the backup job. But if that fails, Veeam will try to communicate through VMware tools. This message is indicating that Vmware tools is either not installed or not running. Please make sure that VMware tools is installed and running on the affected VM. If it is installed, but not running, you can typically restart the service from Windows Services. If it fails to start, reviewing the Windows Application logs can provide info on what is causing the service to fail to start. Microsoft SQL Veeam has a knowledge base article with information on how to troubleshoot these types of warnings: KB2027: Job reports warning “Failed to truncate transaction logs for SQL instances: Possible reasons: lack of permissions, or transaction log corruption.” (veeam.com) Hyper-V Integration Error Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception) Error: Hyper-V Integration Service is not accessible through the network on source host Veeam has a knowledge base article with information on how to troubleshoot these types of warnings: KB1855: Hyper-V Guest processing skipped (check guest OS VSS state and integration components version) (veeam.com) There is also a great blog post from Veeam on the process to install and setup the Hyper-V integration services: Hyper-V installation and configuration step-by-step (veeam.com)

Veeam

Veeam Backup for Salesforce – Now Available!

/

Veeam Backup for Salesforce provides complete access and control of Salesforce data and metadata, as well as providing powerful, rapid-recovery capabilities for IT departments and Salesforce administrators. Veeam recently announced the launch of its newest product offering – Veeam Backup for Salesforce. Salesforce is one of the most utilized CRMs in the world because of its advanced features, but many don’t realize the importance of backing up Salesforce data. Do I need to back up Salesforce? Some users think Salesforce is responsible for backing up the data. Still, it clearly states, “It is important for Salesforce customers to develop a routine data backup strategy as part of their overall data management and security model.” We often talk about the Microsoft shared responsibility model, and Salesforce follows the same rule. Even though Salesforce manages the infrastructure, this doesn’t replace YOUR responsibility to protect Salesforce from data loss and data corruption. Key Capabilities Veeam Backup for Salesforce eliminates the risk of losing your Salesforce data and metadata — so that your business is reliably protected, giving you complete control. Key capabilities include: Salesforce native: Purpose-built to create backups and restore Salesforce data and metadata Fast and flexible recovery: Restore Salesforce records, hierarchies, fields, files and metadata Secure data: Run a backup environment anywhere, on-premises or in the cloud (AWS, Microsoft Azure, etc.) Custom scheduling: Set granular backup schedules and retention settings at the object level Simplified management: Manage several Salesforce instances from one console Incremental changes: With incremental sync and flexibility scheduling, you can back up your Salesforce data and metadata almost continuously Simple and easy-to-use UI: Run backup policies and restore jobs in minutes See and compare: See versions of records and metadata and quickly compare them with production Restore hierarchy: Granularly restore linked objects to any record, including parent/child records Ready to start? Getting started is simple! Veeam provides a licensing structure that aligns with your Salesforce usage. From now until December 31, 2022, Veeam has special pricing on backup for Salesforce: 50 users or less – FREE! Up to 300 users – $2,000 USD for the first year Unlimited users – $10,000 USD for the first year At Managecast, we fully manage and monitor your backups so that you can focus on more strategic initiatives. Interested in learning how Managecast can help your business with its cloud backup and disaster recovery solutions? Fill out this form for more information!

Veeam

Error: Cannot proceed with the job existing backup meta file is not synchronized with the DB [56]

/

We recently ran into an issue where a Veeam Cloud Connect replication job started to fail with the error: Restore point is corrupted We noticed that the source for the replication job was set as the offsite backup copy job, but the source backup copy job was also failing with the following error: Looking into the error message, this can happen when the backup server is rebooted or loses connection during the offsite backup, and it causes things to go out of sync. It is possible this can be resolved by just rescanning the repository and re-enabling the backup copy job. We tried this, but the rescan on the repository came back with an error “XYZ Cloud repository: Failed to synchronize 1 backups.” After a bit more investigation, we determined that the VBM file in the customers’ repository had a size of 0 MB. This indicated that the metadata file had been corrupted and needed to be recreated. Veeam can typically recreate the VBM file as long as the job and remaining backup files are intact. In this case, the solution for this issue was to: Disable the backup copy job. From the Cloud Connect provider side, access the customers’ backup repository. Rename the backup metadata file to “XYZ.vbm.bak” Then rescan the repository from the customer’s backup server. Re-enable the backup copy job. This time, the rescan completed successfully. The backup copy and replication jobs were re-enabled and the next run for each completed successfully.

Office 365 Backup

Do You Really Need to Backup Microsoft 365 Data?

/

I often educate customers on the 3-2-1 rule of backup which is an industry-standard minimum data protection level. The 3-2-1 rule of backup states you should always have: 3 copies of your data (including the original) Stored on 2 different media With 1 copy stored offsite Does Microsoft 365 by itself meet the 3-2-1 rule backup for customers? Do they provide backup at all? First, what Microsoft provides to customers natively does not meet the 3-2-1 rule of backup. Microsoft clearly states the responsibility is yours. Simply Google “Microsoft shared responsibility model” and click the first link. You will note that at the top of the list of those responsibilities always retained by the customer is “information and data”.   Companies are increasingly moving from on-premises file and email servers to Microsoft 365 and other cloud services, but backup and data protection in the cloud is too often a secondary thought.   With the rise of ransomware, other threat actors, and natural disasters, the need for data protection has never been greater.  You should protect your Microsoft 365 data!  For help and assistance with protecting Microsoft 365 data feel free to contact us at sales@managecast.com

Veeam

Error: Item with the same key has already been added.

/

This is an error that occurs when an instant VM recovery has started. Then, the backup copy job runs while the instant VM recovery is running and has not migrated to production. The original VM is deleted and replaced by the one that is in the instant VM recovery mode. The local job can pick up on this change and backup the new VM that is running on the backup storage. However, the copy job begins to have a bug that causes the whole job to fail. Even if you remove the data from the offsite location and remove the VM from the local job, the job continues to fail. The error also will not list out the VM that is causing the issue, but we have only seen this error occur after one of the VMs in the job has gone through an instant VM recovery. The first thing that needs to be done is to migrate the new VM from the backup storage to production. The new VM should already be in the local job, and it can be left in there. The issue is fixing it on the offsite location. Doing a rescan on the service provider or removing the data for that VM does not work. For this to work, the backups also need to be set to Per VM backup chain. If they are not set to Per VM Backups, then this solution will not work. Next, you need to remove the Per VM chain of the VM that is causing the issue. It cannot be removed from the Veeam Console or else it will display this error: The VM needs to be removed from the service provider side in the repository. Once this is deleted the issue will still be there. Even though all the past data is gone, it will still show the same error for the whole job, and none of the other VMs in the job can be backed up. The final step is taking all the data from the job and moving it outside of the tenant directory on the service provider side. It needs to be moved outside of the tenant directory and rescanned from the tenant side in Veeam. After the rescan, the job data should no longer show up under the backups section. You will then want to move that job data back into the tenant directory from the service provider side in the same way that you moved it out. Next, you will want to do another rescan on the tenant side in Veeam. The job should show back up in the backups section. You will want to ensure that the Copy Job that was getting this error is mapped to the job correctly. After that has been completed, the job should be able to transfer data for the other VMs in the job. I have seen where the VM that was causing the issue fails after this but the other VMs have been able to run. My solution to that was to make a separate local job and offsite job just for that VM and get all new data for it.

Managecast News

Managecast Named Veeam VCSP Competency Partner

/

Managecast Technologies, Inc., an award-winning Veeam Cloud Service Provider (VCSP), was named a Veeam VCSP Competency Partner earlier this month. This program designates a service provider that offers the highest level of qualification for providing Veeam-as-a-service solutions. Becoming a VCSP Competency Approved Partner involves thorough testing and approval processes and ensures partners follow strict business, technical and documentation requirements. Managecast Technologies, Inc. is a proud Veeam Gold Partner and has been part of the partner program since 2016. Managecast was also presented with the prestigious Veeam Innovation Award, which celebrates creating new and unique ways to add value to Veeam technology, in 2020 at VeeamON. “Not all Veeam service providers are the same, and some focus more on the providing instead of on the service, like we do,” says Nathan Golden, Owner of Managecast. “Managecast is proud to be recognized for our commitment, focus and expertise we provide for Veeam solutions.” Managecast has met the competency requirements for the following services: BaaS for Microsoft 365 Microsoft provides a wide array of powerful services within Microsoft 365, but a comprehensive backup solution is not one of them. Customers who don’t back up their Microsoft 365 data are at risk of accidental deletion, security threats and retention policy gaps. By achieving the Backup as a Service (BaaS) for Microsoft 365 competency, Managecast has been identified as an expert in providing an out-of-the-box BaaS solution that’s specifically built to protect your critical Microsoft 365 data. Off-site backup We make it easy to extend your data protection solutions with whatever storage option makes sense for your recovery needs, whether it’s tape, object storage or sending backups to a service provider’s off-site cloud infrastructure. Managecast offers a ready-made off-site backup solution to help you get started. DRaaS The need for a disaster recovery (DR) solution cannot be understated, particularly with the rise in frequency of ransomware attacks. Managecast delivers a best-in-breed DR solution that enables you to focus your resources on the important projects to grow and innovate your business, spending less time on risk mitigation. —- About ManagecastManagecast Technologies is a data protection business founded in February of 2000. Passionate about providing world-class data protection solutions, we offer a broad spectrum of services around disaster recovery and business continuity solutions. We combine professional management and disaster recovery planning services with the ability to deliver complete offsite disaster recovery solutions for virtually any platform and application. Our data centers are located in Cincinnati, Ohio, and Detroit, Michigan and are HIPAA, SSAE, SOX, and PCI compliant/certified. To learn more, visit www.managecast.com About VeeamVeeam is the leader in Backup solutions that deliver Cloud Data Management. Veeam’s Availability Platform is the most complete backup solution for helping customers on the journey to achieving success in the 5 Stages of Cloud Data Management. To learn more, visit www.veeam.com ContactNathan Golden, Presidentngolden@managecast.com(513) 735-6868 x 4002

General Cloud Backup

What are you using for offsite backups? – Spiceworks Discussion

/

While scrolling through Spiceworks (an online community where IT professionals can collaborate and seek advice from one another), we found a post requesting options for offsite backups. We see questions like this (below) frequently and wanted to provide a thoughtful and helpful response. If you’re looking for offsite backup options, please take our response into consideration. Nathan Golden – CloudBackupGuy https://community.spiceworks.com/people/nathan-managecast Hey there C.J.  I don’t think you are going to find anything that absolutely prevents bad stuff from being included on the backups as 0-day attacks are often not detected until it’s too late. I do know Sophos makes a great end-point protection product and combined with their firewall offers pretty good defense, but it is not perfect. Nothing is. However, it is wise to do everything possible to keep from being infected such as regularly training your users not to click on bad things (knowb4.com), as well as advanced firewalls and end-point protection, etc.  Backup is the last line of defense! As others mentioned, with Veeam it is possible to run the restore through an AV engine. The idea is that once you’ve found you’ve been compromised you will hopefully find a way to detect the bad stuff. Maybe you need to wait for an updated AV signature, but the idea is once the AV has been updated you can run your restore through it and not restore infected files. Hopefully, only a few files had bad stuff and hopefully, you are restoring to a point in time before the malware has executed and done its thing. The other issue that some are eluding to is how do you protect your past backups from being corrupted/deleted by the malware. A lot of folks will talk about “air gap”, meaning you physically separate your media from the network, which is a great way to protect your data from being compromised by ransomware – though realize air gap does nothing to stop you from backing up malware. However, I see a lot of people do the air gap but then don’t get data offsite frequently. So while negating one threat (ransomware), they are actually introducing additional risk.  Personally, I think anything that relies on human intervention (like rotating media, etc) is prone to error and failure. Our solution is to use a Veeam certified immutable object-store. Yes, it’s still connected to the network, but protected by a different set of credentials and is a solution that is designed to provide unchanging read-only data, even by the network administrator. This maintains full automation and what I call a “virtual airgap”. This is on top of the protections offered by Veeam Insider Protection, etc., so you are really minimizing any risks. As you said there are thousands of companies out there offering these types of services. My suggestion is to ask these questions when choosing an offsite solution: 1) Is the company 100% dedicated to backup and DR, or is it 1 of many different offerings?   2) Is the company a utility-type provider doing their best to keep their humans from interacting with you?  They may provide a service, but little to no expertise – so know what you are doing. 3) Does the company offer Veeam (or other) certified immutable backup capabilities? 4) If you needed your data quickly, how fast can you get it? Is it download only or will they quick ship your data?  Do they offer failover/DR services? Hope this was helpful in your search.

Scroll to Top