How big a tar file that Linux can handle?

Posted by Vivek Gite [Last updated: November 16, 2005]

If you are working in an enterprise computing facility or an Internet Data center (IDC), then I do not have to explain the significance of backup. tar is traditional command used in custom backup scripts, but wait how big a tar that Linux can handle? The Latest version of GNU tar supports large files. However support is, depend upon the following three factors:

Your operating system and kernel version - If you are using latest distribution (created after 2004+) or Linux kernel version 2.4.0+ then you doesn’t have to worry about OS/Kernel
tar version - Latest version of tar supports whatever supported by your Kernel
Large File system (LFS) support – Both ext2 and ext3 filesystem has file size & parition size limitation factors. This table should clear your picture i.e. both file system and command should support large file system (LFS). Support is done via Linux kernel and the GNU C library i.e. glibc.

In short, if you are using LFS then you should able to create file up to 2 TB with tar and all other Linux commands.

How to verify my tar has large file support?
Do not trust blindly on me, you need to verify above three factors yourself. Use procedure outlined below to verify that large file support is available.

(1) Your operating system- Kernel version 2.4.0 or onwards requires to support LFS.

$ uname -mrs

Linux 2.6.14 i686

(2) Version of tar – Old version tar does not support LFS get latest version from GUN website.

$ tar --version

tar (GNU tar) 1.15.1

Please note that archive format plays big role on all filesize and other limitation. If you use posix Archive format defined by POSIX.1-2001 specification then it does not impose any restrictions on file sizes or filename lengths.

(3) Large File system support

(a) Make sure you have glibc version 2.2.x. Here is command to find out version of glibc

$ ls -l /lib/libc.so.6

lrwxrwxrwx 1 root root 13 2005-09-25 18:22 /lib/libc.so.6 -> libc-2.3.5.so

(b) Make sure tar command include 64 bit support
Use strings command to find out if tar is using 64 API to create files.

$ strings $(which tar) | grep 64

readdir64
creat64
__xstat64
__lxstat64
lseek64
__fxstat64
fopen64
ftruncate64
Archive signed base-64 string %s is out of %s range
Archive contains obsolescent base-64 headers

To support large file sizes tar command include setting the FILE_OFFSET_BITS macro to allow files larger than 2 GB on Linux and using uint64_t variables to specify and track file sizes larger than 4 GB on Linux and Solaris. For example fopen64() and other call support LFS.

Online References:

Offline References: