Not having to think about backups is awesome and what would be even more awesome is not having to design a solid backup plan for your environment. But that’s not the name of the game. In complex IT environments it’s difficult to come up with a backup solution, however it’s necessary if you want to be able to sleep peacefully at night. Here’s some tips on how to design a solid backup plan:
In some areas of systems administration a test and tune approach can work quite well. For instance, if you’re tuning a web application it is relatively easy to repeat a process of making some changes. Then observing the results until you have identified a problem or achieved acceptable performance. A backup scheme on the other hand needs to be fully working from the start, unless you thoroughly enjoy when something goes horribly wrong in the middle of the night. Don’t let yourself realize you don’t have all of your data due to an oversight you made when originally designing your backup plan up. Design your backup plan properly in the first place!
Being able to quickly restore your server from a complete meltdown is nice and most backup products make it pretty easy to do this, but successfully restoring all of the different kinds of data that may live on a server is even more important and a bit more tricky. One of the best examples of this is backing up the database. Any kind of file level or even block level backup solution is not going to get a good backup of your database, since when the database is running all of the data is not going to be consistent on the disk. This can put you in a common situation where even though the database files were being backed up, if you ever need to do a restore you may find that the database was not backed up cleanly and is corrupt after it has been restored.
There are many different ways to back up a database “properly” and often the correct solution is one that is specific to your database characteristics and use patterns. There are lots of other types of data in modern environments that need similar care when being backed up to ensure that they can be reliably restored, such as mail servers, file servers, or caches. Questioning the methods used to backup such services and testing your ability to restore from them is very important if you want to be sure that you have backups you can count on.
Another often overlooked backup element is retention. Most likely if something goes wrong you will want to restore from your most recent backup. However there are plenty of situations where a change may go unnoticed for a period of time and you will want to recover files from some time ago. It is important to be thorough when defining a retention policy as it is easy to make mistakes that won’t be noticed until it’s too late. For instance, it’s often a good idea to combine differential or incremental backups with full backups, since they can save a lot of space. However, since differential and incremental backups store only the changes since the full backup they are based on, they require the full backup to be present to do a complete restore. Not carefully checking retention between these two backup types can result in a situation where you have plenty of backups but are unable to recover all of your files. These types of backup mistakes are completely silent, you won’t realize you were doing anything wrong until it’s too late and you’ve lost important data.
To conclude my post here’s 3 tips you should always consider when designing a pristine backup plan:
- Just because you’re backing it up doesn’t mean the data is usable. Consider application specific backups for important data.
- Determine what data needs to be backed up and how long it should be kept for. For instance you may want to back up your database data more frequently, but not keep it as long.
- Have a recovery plan for going from a restore back to production.
