Linux/UNIX: Argument list too long error for rm, cp, mv commands in a shell error

last updated in Categories , , ,

I‘m a new Linux user. We have a backup server and most our developers sftp or ftp to upload files. I just need to backup of last four weeks. If I try to list or remove directories I get an error cannot remove directory. I’m getting “Argument list too long” message. How do I solve this problem when executing commands in the bash/ksh/csh/tcsh/zsh shell?

Each command under Linux and UNIX-like systems accepts a parameter commonly known as command arguments (or args). For example, the command ‘cd /etc’ has one command line arguments, namely, /etc. Some command can accept more than two arguments and act on supplied args. For example, consider the following cp command:
cp /etc/file1 /etc/file2 /etc/file3 /mnt/pen[donotprint][/donotprint]

ADVERTISEMENTS


The cp command has total four command line arguments. The shell can hold a maximum of 131072 bytes for command line arguments. If you try to pass more than that number you will greeted with an error that read as follows:

Argument list too long

Remember each character requires one byte of storage.

Finding out maximum length of the command line argument

The maximum length of the command line argument is limited to *128k* within the Linux kernel. To find your limits, type:

getconf ARG_MAX

Sample outputs:

2621440

The limit affects the execve() kernel function, which is utilized by all other exec() functions (execl, execlp, execle, etc).

Option #1: How to get rid of “Argument list too long error” problem while using rm, ls, cp, mv or any other shell commands

The best way to deal with this problem is combination of the find and xargs commands. The syntax is:

find /dir/to/search -type f -name 'file-pattern' -print0 | xargs -0 -n 10 command-name
find /dir/to/search -type f -iname "*.EXT" -print0 | xargs -0 command-name
find /dir/to/search -type f -iname "*.EXT" -command

Just list files when you get “Argument list too long” error

Use any one of the following syntax to list only files

find /dir/to/search -type f -iname "*.EXT" -ls
find /nas01/data/ -type f -iname "*.doc" -ls

OR the following portable syntax:

find /nas01/data/ -name '*.doc' -exec ls {} +
## GNU/Find ##
find /nas01/data/ -type f -name '*.doc' -exec ls -l {} +

To avoid recursion use the following GNU find option:

find /nas01/data/ -name '*.doc' -maxdepth 1 -exec ls -l {} +

Just delete files when “Argument list too long” error

The syntax is as follows:

# GNU/find and BSD/find may not work on older *nix
find /dir/to/search -type f -iname "*.EXT" -delete
find ~/Downloads/ -type f -iname "*.avi" -delete
find ~/Projects/ -type f -iname "*.pdf" -delete
## use xargs and rm command ##
find ~/Downloads/ -name '*.avi' -print0 | xargs -0 -n 10 rm -f

More portable syntax:

find ~/Projects/ -name '*.pdf' -exec rm -vf {} +

To avoid recursion use the following GNU find option:

## GNU find and may work with bsd find too ##
find ~/Projects/ -name '*.pdf' -maxdepth 1 -exec rm -f {} +
find ~/Downloads/ -type f -iname '*.avi' -print0 | xargs -0 -n 10 rm -f

You can also use GNU parallel command:

## man parallel for more ##
find ~/Downloads/ -type f -iname '*.avi' -print0 | parallel -X -0 rm -vf

Where,

  • -name "*.pdf" : Search pattern.
  • -type f : Only match files.
  • -print 0 : Avoid nasty surprises when dealing with weird file names with blank spaces, ‘, “, \, tab, and newline characters. The -print0 primary in conjunction with xargs -0 as an effective alternative to dealing with weird filenames.
  • -maxdepth 1 : Descend at most one directory levels below the command line arguments i.e avoid recursion.
  • -exec rm -f {} + OR -exec ls {} + : Run rm or ls commands on as many as files possible.

Option #2: Iterate with the shell to avoid arg list too long error

The syntax is as follows and it is very slow and may not work at all:

for f in *; do command "$i"; done     
 
## delete all pdf files ##
for f in *.pdf; do /bin/rm -rf "$f"; done
 
############################################################################
### Delete all *.mp4 and *.avi files using pure Bash solution using arrays
### Based on http://mywiki.wooledge.org/BashFAQ/095 
############################################################################
 files=($HOME/Downloads/*.avi $HOME/Downloads/*.mp4)
 for ((i=0; i<${#files[*]}; i+=100)); do
   /bin/rm -vf "${files[@]:i:100}" 
 done

Command summary to to avoid the limit in a shell

## deals with weird filenames too ##
find /dir/to/search/ -name "file-pattern-search" -exec command {} + 
 
## find and xargs based solution ##
find /dir/to/search/ -name "file-pattern-search" -print0 | xargs -0 command
 
## For gnu parallel fans ##
find /dir/to/search/ -name 'file-pattern-search' -print0 | parallel -X -0 command
 
## older and not recommended solution; but kept here for the historical reasons only ##
for i in *; do command "$i"; done

See man pages for more info: mv(1)

UP NEXT

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin, DevOps engineer, and a trainer for the Linux operating system/Unix shell scripting. Get the latest tutorials on SysAdmin, Linux/Unix and open source topics via RSS/XML feed or weekly email newsletter.


ADVERTISEMENTS

14 comment

  7. -bash: /bin/mv: Argument list too long
    I am Getting th eavove error message while moving file from one dir to other.please help me to solve this problem.I hv root permission too.

    Reply

    1. ViVEk GITe, not agree with you. The wilcards in any shell enviroments are replaced before execute the comand line. For example:

      $ touch a1 a2 a3 a4 a5 a6
      $ ls a*
      a1 a2 a3 a4 a5 a6

      Will work fine, because the ‘a*’ is replaced before executing the ls command but the parameters list is not long enough. But, creating for example a extend list of files (10000 empty files), the ‘ls’ command will show you how the parameters list is too long even when you write ‘a*’ as the only parameter!!
      See:

      $ i=1
      $ while [ $i -le 10000 ]; do
      > touch a$i
      > i=$((i+1))
      > done

      $ ls | wc -l
      10000
      $ ls a*
      ksh: /usr/bin/ls: 0403-027 The parameter list is too long.

      That is because the wilcard is replaced before executing ls + arguments command.
      In that case I agree with Tom solution… ;)

      Reply

  9. Thank you for this!

    Btw, your iterating example is missing a y

    (which for very lazy copypasta addicts like myself is dangerous)

    Reply

  10. @Vivek

    Right, this topic should be modified, at least for two reasons:

    1. Using wildcards does not solve anything. It is just false.

    2. The more appropriate way, is to follow Unix philosophy, that is, do one thing and pass it to another program: => use xargs !

    https://en.wikipedia.org/wiki/Unix_philosophy

    Please modifiy this old 2007 topic that is just outdated and does not show how good you are at *nix! You are much better than that! (At least you are now ;-) )

    — Philippe

    Reply

      1. Whaoo!

        Congratulations for your quick reaction and full rewriting of this topic.

        Just to be thorough, I would clearly advice __not__ to use the [-exec] option of find, but use find to pipe to xargs in any situation.

        There are deep and strong reasons why, again, it is more appropriate to not use [-exec], let [find] do one thing, and then pipe to [xargs].

        I could elaborate, and share this with you if you want, feel free to e-mail me in private if you need to.

        Many thanks again for your professional work here, helping us using *nix !

        — Philippe

        Reply

  11. The path you entered, is too long. Enter a shorter path
    File Name could not be found. Check the spelling of the filename,
    and verify that the file location is correct.

    Reply

Leave a Comment

Tagged as: Tags , , , ,