Truncate Large Text File in UNIX / Linux

by on February 20, 2010 · 19 comments· LAST UPDATED October 2, 2012

in

How do I truncate or shrink large text file under UNIX / Linux operating systems?

There are various tools and methods to truncate

Tutorial details
DifficultyEasy (rss)
Root privilegesNo
RequirementsNone
Estimated completion timeN/A
large text files under UNIX / Linux operating systems as follows.

Options #1: Shell Output Redirction

Your can truncate text file and make the size to zero using redirection:

> {filename}
ls -l largefile.txt
> largefile.txt
ls -l largefile.txt

Please note that largefile.txt file is created if it doesn't exist. And largefile.txt file is overwritten if it exists.

Option #2: truncate Command

Use the truncate command to shrink or extend the size of each FILE to the specified size:

truncate -s 0 {filename.txt}
ls -lh filename.txt
truncate -s 0 filename.txt
ls -lh filename.txt

The -s option is used to set SIZE to zero. See truncate command man page for more details:
man truncate

Option #3: logrotate Utility

logrotate command is designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large. See how to use logrotate command to rotates, compresses, and mails system logs stored in /var/log and other locations under UNIX / Linux oses.

Option #4: /dev/null

The null device /dev/null act as the black hole that discards all data written to it under Unix like operating system. You can use it as follows (hat tip to Philippe Petrinko):

 
cp /dev/null largefile.txt
 

OR

 
cat /dev/null > largefile.txt
 
TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 19 comments… read them below or add one }

1 Nilesh February 20, 2010 at 1:20 pm

Isn’t this more of a useless thing ? You lose all the data in the first two methods.

Reply

2 nixCraft February 20, 2010 at 2:07 pm

@Nilesh,

That is the point, you want to truncate file size. For example, some stupid app crate a log file due to some networking issue. Next you fixed the issue and you don’t want the data. So just truncate it.

Reply

3 jaysunn February 20, 2010 at 2:32 pm

I vote for Option#3. Can’t live without my logrotate.conf.

Jaysunn

Reply

4 Philippe Petrinko February 21, 2010 at 12:13 pm

Option #4:

cp /dev/null toobig

Option #5:

cat /dev/null > toobig

Reply

5 Philippe Petrinko February 21, 2010 at 12:13 pm

Option #4:

cp /dev/null toobig

Option #5:

cat /dev/null > toobig

Reply

6 tiptop February 21, 2010 at 8:54 pm

What does it mean to “rotate” log files?

Reply

7 Philippe Petrinko February 21, 2010 at 10:51 pm
8 Shoaibi February 23, 2010 at 12:49 pm

“Please note that largefile.txt file is created if it doesn’t exist. And largefile.txt file is overwritten if it exits.”

Last word should be “exists” :)

Reply

9 nixCraft October 2, 2012 at 9:53 am

Thanks for the heads up!

Reply

10 Dave February 24, 2010 at 4:09 pm

Truncating brings up a good time to remind users about sparse files. Before you try to truncate a file, make sure the file isn’t a sparse file. With sparse files, checking the size with ls -l will NOT give you the true file size. In fact, the file may look many times larger than it really is.

Always check the file with the command du to check the size of the file if there is any chance you may be dealing with a sparse file or if you aren’t sure.

Reply

11 Philippe Petrinko February 24, 2010 at 5:45 pm

@Dave: Yes, good to know/remember.

Nevertheless, [ls] can show blocks with [s / --size] option:


dd if=/dev/zero of=sparse bs=1 count=1 seek=1024k # let's create a sparse-file
ls -ls sparse # show me your blocks
8 -rw-r--r-- 1 user1 user1 1048577 feb 24 18:38 sparse

Reply

12 Jay July 31, 2011 at 12:40 am

mentioning the ‘truncate’ command (as opposed to the library call), makes this Linux only (or other Unix-like without its own file utils, relying on GNU).

Reply

13 Rama Akella September 27, 2012 at 7:14 am

Please execute the below command in shell

> filename

it will truncate the file..

Thanks,,
Rama

Reply

14 Philippe Petrinko September 27, 2012 at 8:49 pm

BTW Vivek, when will you add my options, as they are functional and relevant?

And yes, please modify “exits” into “exists” as Shoaibi pointed out, don’t you think?

Reply

15 nixCraft October 2, 2012 at 9:54 am

Done. The faq has been updated.

I appreciate your feedback :)

Reply

16 Vex October 22, 2012 at 11:51 pm

Rama Akella points out the useless use of cat . The end goal is that you don’t want to change the inode of the file you wish to truncate to zero bytes. It’s functionally equivalent to run `> file` to accomplish this.

http://partmaps.org/era/unix/award.html#cat

The truncate option that truncates the file in place saves me from the problem of having to dd the file to a temp file and write it back as I’m dealing with files in the hundreds of gigabytes. My outdated install of Gentoo doesn’t provide truncate in coreutils. I instead found the source from a FreeBSD machine and compiled it manually, not knowing at the time that the newer coreutils includes it.

Locate the truncate.c file in the FreeBSD world source tree. I found it here:
/usr/src/usr.bin/truncate/truncate.c
Copy this to your *nix system and compile it
gcc -o truncate truncate.c
Test it out in case the C libraries are too divergent before you use it.

For Gentoo I created an ebuild for those who are not able to update coreutils. I offer absolutely no warranties of my work or the files that I am using. I am redistributing the code without modification.

Reply

17 Arun Bagul November 27, 2012 at 3:29 am

Good Article and simple topic.

Reply

18 roger February 28, 2013 at 6:17 pm

Is there a yum package I’m missing?

bash: truncate: command not found

Reply

19 Vex March 1, 2013 at 7:52 am

Roger, I made a post about this issue, http://www.devunmounted.com/2012/10/22/truncate-your-proble/ , I hope it helps you with your problem. The jest is that the truncate ships with a newer glibc, but if you don’t have a newer glibc you can either compile truncate.c from the newer glibc or from a libc bsd source. I believe my write up on it should cover both. Good luck.

Reply

Leave a Comment

Tagged as: , , , , , , , , , , , , , , , , , , , , , ,

Previous Faq:

Next Faq: