≡ Menu

data center

Linux: Local / Remote Backup For Large Files

You can easily backup large files using Linux backup software combination - tar, split and md5sum program. From the article:

I have a directory containing some files for a virtual computer. The files that hold the data for the virtual computer's hard discs are not only big, they are also "sparse" files, which means that they use only enough disc space for the data that was actually written to the file. For example, the virtual computer may have a 30GB drive of which 2GB has been used. Even though it uses only 2GB on my hard disc, a program that reads the file may see it as a 30GB file. This type of file can be tricky to back up because, when you copy it, you can end up with a 30GB file, or it might simply fail to copy, depending on the type of file system used on the backup storage.

=> Backing up Large Files

Linux Data Center Power Consumption Less Than Windows

According to new independent tests Red Hat Linux pulls as much as 12% less power than Windows 2008 on identical hardware. It means Linux is better for reducing data center power consumption or electrical power consumption. From the report:

Our tests point to Linux as the winner of the green flag by margins that topped out at 12%. We ran multiple power consumption tests using Windows Server 2008 Enterprise Edition, Red Hat's Enterprise Linux (RHEL) 5.1 and SUSE Enterprise Linux 10 SP1 on four, popular 1U server machines, one each from Dell and IBM and two from HP. The results showed that while Windows Server 2008 drew slightly less power in a few test cases when it had its maximum power saving settings turned on, it was RHEL that did the best job of keeping the power draw in check across the board.

Read full research report about computer power consumption online:

Now the million dollar question, how important is green computing / green computers to you? Would you make switch to save the power?

Amazon.com – Http/1.1 Service Unavailable – Down?

I was just looking to order few books and I got

Http/1.1 Service Unavailable

Anyone else finding Amazon unavailable? If anything can go wrong, it will. I'm sure they are losing big money every minute they are down.

How To Avoid Sudden Outburst Of Backup Shell Script / Program Disk I/O

A sudden outburst of violent disk I/O activity can bring down your email or web server. Usually, a web / mysql or mail server serving millions and millions pages per months are prone to this kind of problem. Backup activity can increase current system load. To avoid this kind of sudden outburst problem, run your script with scheduling class and priority. Linux comes with various utilities to manage this kind of madness.
[click to continue…]

Google Data Center Information

CNet has published an interesting information about Google data center and estimates that they have 2,00,000 servers spanned across 36 data centers across the globe. From the article:

On the other hand, Dean seemingly thinks clusters of 1,800 servers are pretty routine, if not exactly ho-hum. And the software company runs on top of that hardware, enabling a sub-half-second response to an ordinary Google search query that involves 700 to 1,000 servers, is another matter altogether.

Google doesn't reveal exactly how many servers it has, but I'd estimate it's easily in the hundreds of thousands. It puts 40 servers in each rack, Dean said, and by one reckoning, Google has 36 data centers across the globe. With 150 racks per data center, that would mean Google has more than 200,000 servers, and I'd guess it's far beyond that and growing every day.

(Fig.01: Google data center [credit:cnet news])

I'm well aware of HA and clustering technologies but this is massive setup with tons and tons of systems. Google uses distributed storage system and other in house developed tools.

Sounds like a great place to work :)

=> Google spotlights data center inner workings

Consistent backup with Linux Logical Volume Manager (LVM) snapshots

LVM is an implementation of a logical volume manager for the Linux kernel. The biggest advantage is that LVM provides the ability to make a snapshot of any logical volume.

In a production environment many users access the same file (file is open) or database. Suppose you start backup process when a file is open, you will not get correct or updated copy of file i.e. inconsistent backup (see accurate definition of inconsistent backup).

Read-only partition - to avoid inconsistent backup

You need to mount partition as read only, so that no one can make changes to file and make a backup:
# umount /home
# mount -o ro /home
# tar -cvf /dev/st0 /home
# umount /home
# mount -o rw /home

As you see, the draw back is service remains unavailable during backup time to all end users. If you are using database then shutdown database server and make a backup.

Logical Volume Manager snapshot to avoid inconsistent backup

This solution will only work if you have created the partition with LVM. A snapshot volume is a special type of volume that presents all the data that was in the volume at the time the snapshot was created. This means you can back up that volume without having to worry about data being changed while the backup is going on, and you don't have to take the database volume offline while the backup is taking place.

# lvcreate -L1000M -s -n dbbackup /dev/ops/databases


lvcreate -- WARNING: the snapshot must be disabled if it gets full
lvcreate -- INFO: using default snapshot chunk size of 64 KB for "/dev/ops/dbbackup"
lvcreate -- doing automatic backup of "ops"
lvcreate -- logical volume "/dev/ops/dbbackup" successfully created

Create a mount-point and mount the volume:
# mkdir /mnt/ops/dbbackup
# mount /dev/ops/dbbackup /mnt/ops/dbbackup


mount: block device /dev/ops/dbbackup is write-protected, mounting read-only

Do the backup
# tar -cf /dev/st0 /mnt/ops/dbbackup

Now remove it:
# umount /mnt/ops/dbbackup
# lvremove /dev/ops/databases

Please note that LVM snapshots cannot be used with non-LVM filesystems i.e. you need LVM partitions. You can also use third party commercial proprietary (see below for discussion) or GPL backup solutions/software.

MySQL Backup: Using LVM File System Snapshot

Login to your MySQL server:
# mysql -u root -p
At mysql prompt type the following command to closes all open tables and locks all tables for all databases with a read lock until you explicitly release the lock by executing UNLOCK TABLES. This is very convenient way to get backups if you have a file system such as Veritas or Linux LVM or FreeBD UFS that can take snapshots in time.
mysql> flush tables with read lock;
mysql> flush logs;
mysql> quit;

Now type the following command (assuming that your MySQL DB is on /dev/vg01/mysql):
# lvcreate --snapshot –-size=1000M --name=backup /dev/vg01/mysql
Again, login to mysql:
# mysql -u root -p
Type the following to release the lock:
mysql> unlock tables;
mysql> quit;

Now, move backup to tape or other server:
# mkdir -p /mnt/mysql
# mount -o ro /dev/vg01/backup /mnt/mysql
# cd /mnt/mysql
# tar czvf mysql.$(date +"%m-%d%-%Y).tar.gz mysql
# umount /mnt/tmp
# lvremove -f /dev/vg01/backup

If you are using a Veritas file system, you can make a backup like this (quoting from the official MySQL documentation):

Login to mysql and lock the tables:
mysql> quit;

Type the following at a shell prompt
# mount vxfs snapshot
mysql> quit;

Copy files from the snapshot and unmount the snapshot.

Further reading: