How big a tar file that Linux can handle?

by on November 16, 2005 · 2 comments· LAST UPDATED November 16, 2005

in

If you are working in an enterprise computing facility or an Internet Data center (IDC), then I do not have to explain the significance of backup. tar is traditional command used in custom backup scripts, but wait how big a tar that Linux can handle? The Latest version of GNU tar supports large files. However support is, depend upon the following three factors:

  1. Your operating system and kernel version - If you are using latest distribution (created after 2004+) or Linux kernel version 2.4.0+ then you doesn’t have to worry about OS/Kernel
  2. tar version - Latest version of tar supports whatever supported by your Kernel
  3. Large File system (LFS) support – Both ext2 and ext3 filesystem has file size & parition size limitation factors. This table should clear your picture i.e. both file system and command should support large file system (LFS). Support is done via Linux kernel and the GNU C library i.e. glibc.

In short, if you are using LFS then you should able to create file up to 2 TB with tar and all other Linux commands.

How to verify my tar has large file support?
Do not trust blindly on me, you need to verify above three factors yourself. Use procedure outlined below to verify that large file support is available.

(1) Your operating system- Kernel version 2.4.0 or onwards requires to support LFS.

$ uname -mrs
Linux 2.6.14 i686

(2) Version of tar – Old version tar does not support LFS get latest version from GUN website.

$ tar --version
tar (GNU tar) 1.15.1 

Please note that archive format plays big role on all filesize and other limitation. If you use posix Archive format defined by POSIX.1-2001 specification then it does not impose any restrictions on file sizes or filename lengths.

(3) Large File system support

(a) Make sure you have glibc version 2.2.x. Here is command to find out version of glibc

$ ls -l /lib/libc.so.6

lrwxrwxrwx 1 root root 13 2005-09-25 18:22 /lib/libc.so.6 -> libc-2.3.5.so

(b) Make sure tar command include 64 bit support
Use strings command to find out if tar is using 64 API to create files.

$ strings $(which tar) | grep 64

readdir64
creat64
__xstat64
__lxstat64
lseek64
__fxstat64
fopen64
ftruncate64
Archive signed base-64 string %s is out of %s range
Archive contains obsolescent base-64 headers

To support large file sizes tar command include setting the FILE_OFFSET_BITS macro to allow files larger than 2 GB on Linux and using uint64_t variables to specify and track file sizes larger than 4 GB on Linux and Solaris. For example fopen64() and other call support LFS.

Online References:

Offline References:

  • Read man pages with 'man tar' command
  • Read GNU tar manual with 'info tar' command
TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 2 comments… read them below or add one }

1 jk November 18, 2005 at 12:22 am

nice post, it really help newbie like me. Thanks a lot :)

Reply

2 thiyagi December 13, 2010 at 6:54 pm

thanks, that was really helpful..

Reply

Leave a Comment

Previous post:

Next post: