Consistent backup with Linux Logical Volume Manager (LVM) snapshots

by on June 1, 2006 · 13 comments· LAST UPDATED June 11, 2009

in , ,

LVM is an implementation of a logical volume manager for the Linux kernel. The biggest advantage is that LVM provides the ability to make a snapshot of any logical volume.

In a production environment many users access the same file (file is open) or database. Suppose you start backup process when a file is open, you will not get correct or updated copy of file i.e. inconsistent backup (see accurate definition of inconsistent backup).

Read-only partition - to avoid inconsistent backup

You need to mount partition as read only, so that no one can make changes to file and make a backup:
# umount /home
# mount -o ro /home
# tar -cvf /dev/st0 /home
# umount /home
# mount -o rw /home

As you see, the draw back is service remains unavailable during backup time to all end users. If you are using database then shutdown database server and make a backup.

Logical Volume Manager snapshot to avoid inconsistent backup

This solution will only work if you have created the partition with LVM. A snapshot volume is a special type of volume that presents all the data that was in the volume at the time the snapshot was created. This means you can back up that volume without having to worry about data being changed while the backup is going on, and you don't have to take the database volume offline while the backup is taking place.

# lvcreate -L1000M -s -n dbbackup /dev/ops/databases

Output:

lvcreate -- WARNING: the snapshot must be disabled if it gets full
lvcreate -- INFO: using default snapshot chunk size of 64 KB for "/dev/ops/dbbackup"
lvcreate -- doing automatic backup of "ops"
lvcreate -- logical volume "/dev/ops/dbbackup" successfully created

Create a mount-point and mount the volume:
# mkdir /mnt/ops/dbbackup
# mount /dev/ops/dbbackup /mnt/ops/dbbackup

Output:

mount: block device /dev/ops/dbbackup is write-protected, mounting read-only

Do the backup
# tar -cf /dev/st0 /mnt/ops/dbbackup

Now remove it:
# umount /mnt/ops/dbbackup
# lvremove /dev/ops/databases

Please note that LVM snapshots cannot be used with non-LVM filesystems i.e. you need LVM partitions. You can also use third party commercial proprietary (see below for discussion) or GPL backup solutions/software.

MySQL Backup: Using LVM File System Snapshot

Login to your MySQL server:
# mysql -u root -p
At mysql prompt type the following command to closes all open tables and locks all tables for all databases with a read lock until you explicitly release the lock by executing UNLOCK TABLES. This is very convenient way to get backups if you have a file system such as Veritas or Linux LVM or FreeBD UFS that can take snapshots in time.
mysql> flush tables with read lock;
mysql> flush logs;
mysql> quit;

Now type the following command (assuming that your MySQL DB is on /dev/vg01/mysql):
# lvcreate --snapshot –-size=1000M --name=backup /dev/vg01/mysql
Again, login to mysql:
# mysql -u root -p
Type the following to release the lock:
mysql> unlock tables;
mysql> quit;

Now, move backup to tape or other server:
# mkdir -p /mnt/mysql
# mount -o ro /dev/vg01/backup /mnt/mysql
# cd /mnt/mysql
# tar czvf mysql.$(date +"%m-%d%-%Y).tar.gz mysql
# umount /mnt/tmp
# lvremove -f /dev/vg01/backup

If you are using a Veritas file system, you can make a backup like this (quoting from the official MySQL documentation):

Login to mysql and lock the tables:
mysql> FLUSH TABLES WITH READ LOCK;
mysql> quit;

Type the following at a shell prompt
# mount vxfs snapshot
Now UNLOCK TABLES:
mysql> UNLOCK TABLES;
mysql> quit;

Copy files from the snapshot and unmount the snapshot.

Further reading:

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 13 comments… read them below or add one }

1 J.B. Nicholson-Owens June 3, 2006 at 4:02 am

By the way, “commercial or GPL” backup programs are not opposites. You don’t mean commercial here, you mean non-GPL’d backup programs or possibly you mean proprietary versus free software backup programs.

Many business redistribute GPL-covered software as part of business activity, hence GPL-covered software is commercial. Since GPL-covered software is commercial software, GPL-covered software is not the opposite of commercial software, GPL-covered software is a proper subset of commercial software.

See http://www.gnu.org/philosophy/words-to-avoid.html#Commercial and http://www.gnu.org/philosophy/selling.html for more on this common confusion of terms.

Reply

2 nixCraft June 3, 2006 at 10:20 am

J.B. Nicholson-Owens,

Thanks for pointing out this issue.

Appreciate your post.

Reply

3 Gordon Messmer June 5, 2006 at 7:48 pm

It should probably be pointed out that you still need to take the database offline while you make the snapshot. Snapshots don’t eliminate the need for downtime to get consistent backups, they just make it very short (a few seconds, instead of the full backup time).

I’ve worked with a number of people who read articles like this one, and came away with that misconception.

Reply

4 Naan Yaar June 5, 2006 at 10:53 pm

To echo what Gordon Messmer says above, LVM snapshots will not obviate the need to make applications that use the filesystem quiescent before making the snapshot. Otherwise, there would be no guarantee that the application that writes to the filesystem has consistent, workable data that it can recover from at the instant in which the snapshot was created.

Reply

5 durian June 6, 2006 at 7:43 am
6 Anand Gupta August 21, 2007 at 7:15 pm

I have heard about lvm2 snapshots not being stable, and lot of people loosing data over it. What are your thoughts on that ?

Reply

7 Thomas January 3, 2008 at 5:50 pm

From my own testing and according to other’s people articles you need to FLUSH TABLES WITH READ LOCK; on MySQL prior to making the snapshot, and release the lock right after the snapshot command have completed.

Note that normally, MySQL can recover from any interruption as long as its fsync() calls secure the data (That means the storage subsystems must not cache writes unless the memory is backed up by a battery, this is usually called WRITE-TROUGH caching on RAID subsystems). That was part of my testing too and I had nodes of a busy MySQL database of a few hundread GBs being power-cycled hundreads of times without a glitch. A single power-cycle on the same database with a WRITE-BACK cache destroyed the database (obviously a testing copy).

This is a limitation of LVM and I experienced issues with both LVM1 and LVM2, both RO and RW (LVM2 only) snapshots. Entreprise-grade hardware raids with snapshot features shouldn’t suffer from this issue.

Reply

8 Brad April 13, 2008 at 8:05 am

I realize this is a very old post and so it’s possible my comment will never be seen… but I figure it’s worth a try.

I am using this technique to back up a mysql database that is currently relatively small 1.5gb on disk. Everything is working great… except once I have the snapshot mounted, and I try to do something like TAR the results for a backup… the tar is taking about 25 minutes the FIRST TIME… immediately after the snapshot was created. Any subsequent attempts to TAR will happen in about 45seconds.

Is this expected behavior? I am guesing it has something to do with how the snapshot is actually filling in the disk image on the first read… or something, and so the TAR is actually causing a create.

I’ve tried mounting read only to see if that speeds things up… but it doesn’t seem to.

Thoughts?

Reply

9 Vasiliy May 15, 2009 at 4:49 pm

Dear All,

You may want to notice, that there is a mistake in your instructions. The portion where you tell the users to remove the snapshot, not the actual logical volume…

Thanks

Reply

10 Chris February 10, 2010 at 6:03 am

The steps in this article won’t lock your tables. The example shows you locking the tables and then quitting the mysql utility. I’m not sure if this used to work differently (I doubt it), but in MySQL 5, quitting the mysql utility drops the connection and unlocks the tables. This is easy to test for yourself by trying to write to them.
Instead, you can get a correct snapshot like this:
mysql> FLUSH TABLES WITH READ LOCK;
mysql> system lvcreate –snapshot –-size=1000M –name=backup /dev/vg01/mysql;
mysql> UNLOCK TABLES;
mysql> quit

Also, the official documentation (http://dev.mysql.com/doc/refman/5.1/en/backup-methods.html) doesn’t mention any need for the FLUSH LOGS statement before taking your snapshot. Searching around the net I am finding some people saying to use it and others not, and nobody explains their reasoning…

Reply

11 JW November 14, 2012 at 1:36 am

FLUSH LOGS would be relevant if binary logging is enabled (in case Point-In-Time-Recovery is desired/required).

Reply

12 Mike July 6, 2010 at 10:53 pm

Hi, did not read all the comments, but I did find an error.

You can not “quit;” after doing the “flush tables with read lock;” The second you quit “unlock_tables” is called and thus not holding the lock while you make the lvm snapshot.

That is all.

Reply

13 Kaicheng Zhang August 10, 2010 at 9:42 am

The article helps me a lot.
But I think the
lvremove /dev/ops/databases
should be
lvremove /dev/ops/dbbackup
since we need to cleanup the snapshot and let the origin volume still work.

Reply

Leave a Comment

Tagged as: , , , , , , , , , , , , , , , ,

Previous post:

Next post: