On January 31, 2017 Gitlab experienced a massive database outage. They suffered hours of downtime, which they have detailed here. The transparency in that link is breathtaking for a couple reasons. First, any company willing to details their failures and challenges in such public terms gets my kudos. It's not easy to put yourself on display, warts and all. Second, there are so many self inflicted wounds and unforced errors in the story. This line is particularly noteworthy
So in other words, out of five backup/replication techniques deployed none are working reliably or set up in the first place.
I've worked in IT Operations for ten years and can recall many times in my past where I could have been Gitlab's Team-member-1. When the pressure is on it is EXTREMELY easy to make a fatal error. Even when I did cause problems that could (or should) have been a Resume Generating Event I knew that I had a solid, tested recovery strategy to rely on, powered by Veeam.
Veeam has been the market leader in virtualization backups for over ten years. Many of the standard "must have" recovery features we expect in a backup solution were introduced or perfected by Veeam. They even include the ability to create a "lab" to validate recoverability of backups and to provide a method to test upgrades, updates, or application changes against actual copies of production machines. For all of the great features of Veeam's virtual backup tools there has always been a gap in the solution. I have yet to work at an organization that had achieved 100% virtualization. Having a solution that rocks for virtual environments is amazing but it's also frustrating to need to maintain a second system for bare metal machines.
At Tech Field Day 13, Veeam demonstrated an agent based approach to fill this gap. With Veeam Agent for Linux and Veeam Agent for Windows, you can get the same management and recovery options you're accustomed to for VMs on physical machines. Not only is it designed to work on bare metal servers but also workstations and cloud instances can be protected. A struggle I experienced in my early days using public cloud was a lack of viable backup options. Most cloud workloads are ephemeral but there are some that need to be backed up so this would fit the bill.
Veeam has provided a free Endpoint Backup program for years. In fact, it is their most popular free product. There are some key feature differences between the different license levels of the agent based solution. Clint Wyckoff (@ClintWyckoff) wrote a fantastic article here that can detail everything much better than I can. I will show their handy chart to list the differences.
There is one really neat entries on this chart that I want to mention quickly. There is a line item called "Endpoint protection for mobile users". This feature is a lot cooler than the name would lead you to believe. If your laptop is being protected with Veeam Agent for Windows but you are away during your scheduled backup time, the backup can still run and cache on your machine. When you get back to a connection that can resolve the Veeam Backup Repository it will silently sync the backup. This is a great option to make sure your road warriors' data is still protected. This works great with the Backup directly to Veeam Cloud Connect feature. This allows the backup to go to a Veeam service partner or your HQ with the Cloud Connect feature configured. With the paid version, a backup could run on the plane and sync up when you get to a strong wifi connection. That's pretty cool.
I've chosen to feature just one of the new products Veeam discussed during the Tech Field Day 13 presentation. There are many more management and recovery tools displayed in the session itself. You can watch the full video on the Tech Field Day site if you're interested in seeing everything they showed. As of this article the Veeam Agent for Windows product is in beta and can be downloaded at their site. I am currently running it and plan to write up those experiences as well for a future article. I've explored many of the recovery options and I am extremely impressed.