How to keep a detailed audit trail of what's being done on your Linux systems

Posted on in Categories Linux, RedHat/Fedora Linux, Security, Suse Linux, Ubuntu Linux, UNIX last updated November 14, 2006

Intrusions can take place from both authorized (insiders) and unauthorized (outsiders) users. My personal experience shows that unhappy user can damage the system, especially when they have a shell access. Some users are little smart and removes history file (such as ~/.bash_history) but you can monitor all user executed commands.

It is recommended that you log user activity using process accounting. Process accounting allows you to view every command executed by a user including CPU and memory time. With process accounting sys admin always find out which command executed at what time 🙂

The psacct package contains several utilities for monitoring process activities, including ac, lastcomm, accton and sa.

  • The ac command displays statistics about how long users have been logged on.
  • The lastcomm command displays information about previous executed commands.
  • The accton command turns process accounting on or off.
  • The sa command summarizes information about previously executed commmands.

Task: Install psacct or acct package

Use up2date command if you are using RHEL ver 4.0 or less
# up2date psacct
Use yum command if you are using CentOS/Fedora Linux / RHEL 5:
# yum install psacct
Use apt-get command if you are using Ubuntu / Debian Linux:
$ sudo apt-get install acct OR # apt-get install acct

Task: Start psacct/acct service

By default service is started on Ubuntu / Debian Linux by creating /var/account/pacct file. But under Red Hat /Fedora Core/Cent OS you need to start psacct service manually. Type the following two commands to create /var/account/pacct file and start services:
# chkconfig psacct on
# /etc/init.d/psacct start

If you are using Suse Linux, the name of service is acct. Type the following commands:
# chkconfig acct on
# /etc/init.d/acct start

Now let us see how to utilize these utilities to monitor user commands and time.

Task: Display statistics about users’ connect time

ac command prints out a report of connect time in hours based on the logins/logouts. A total is also printed out. If you type ac without any argument it will display total connect time:
$ acOutput:

total       95.08

Display totals for each day rather than just one big total at the end:
$ ac -dOutput:

Nov  1  total        8.65
Nov  2  total        5.70
Nov  3  total       13.43
Nov  4  total        6.24
Nov  5  total       10.70
Nov  6  total        6.70
Nov  7  total       10.30
Nov 12  total        3.42
Nov 13  total        4.55
Today   total        0.52

Display time totals for each user in addition to the usual everything-lumped-into-one value:
$ ac -pOutput:

        vivek                             87.49
        root                                 7.63
        total       95.11

Task: find out information about previously executed user commands

Use lastcomm command which print out information about previously executed commands. You can search command using usernames, tty names, or by command names itself.

Display command executed by vivek user:
$ lastcomm vivekOutput:

userhelper        S   X vivek  pts/0      0.00 secs Mon Nov 13 23:58
userhelper        S     vivek  pts/0      0.00 secs Mon Nov 13 23:45
rpmq                    vivek  pts/0      0.01 secs Mon Nov 13 23:45
rpmq                    vivek  pts/0      0.00 secs Mon Nov 13 23:45
rpmq                    vivek  pts/0      0.01 secs Mon Nov 13 23:45
gcc                     vivek  pts/0      0.00 secs Mon Nov 13 23:45
which                   vivek  pts/0      0.00 secs Mon Nov 13 23:44
bash               F    vivek  pts/0      0.00 secs Mon Nov 13 23:44
ls                      vivek  pts/0      0.00 secs Mon Nov 13 23:43
rm                      vivek  pts/0      0.00 secs Mon Nov 13 23:43
vi                      vivek  pts/0      0.00 secs Mon Nov 13 23:43
ping              S     vivek  pts/0      0.00 secs Mon Nov 13 23:42
ping              S     vivek  pts/0      0.00 secs Mon Nov 13 23:42
ping              S     vivek  pts/0      0.00 secs Mon Nov 13 23:42
cat                     vivek  pts/0      0.00 secs Mon Nov 13 23:42
netstat                 vivek  pts/0      0.07 secs Mon Nov 13 23:42
su                S     vivek  pts/0      0.00 secs Mon Nov 13 23:38

For each entry the following information is printed. Take example of first output line:
userhelper S X vivek pts/0 0.00 secs Mon Nov 13 23:58

  • userhelper is command name of the process
  • S and X are flags, as recorded by the system accounting routines. Following is the meaning of each flag:
    • S — command executed by super-user
    • F — command executed after a fork but without a following exec
    • D — command terminated with the generation of a core file
    • X — command was terminated with the signal SIGTERM
  • vivek the name of the user who ran the process
  • prts/0 terminal name
  • 0.00 secs – time the process exited

Search the accounting logs by command name:
$ lastcomm rm
$ lastcomm passwd

rm                S     root     pts/0      0.00 secs Tue Nov 14 00:39
rm                S     root     pts/0      0.00 secs Tue Nov 14 00:39
rm                S     root     pts/0      0.00 secs Tue Nov 14 00:38
rm                S     root     pts/0      0.00 secs Tue Nov 14 00:38
rm                S     root     pts/0      0.00 secs Tue Nov 14 00:36
rm                S     root     pts/0      0.00 secs Tue Nov 14 00:36
rm                S     root     pts/0      0.00 secs Tue Nov 14 00:35
rm                S     root     pts/0      0.00 secs Tue Nov 14 00:35
rm                      vivek    pts/0      0.00 secs Tue Nov 14 00:30
rm                      vivek    pts/1      0.00 secs Tue Nov 14 00:30
rm                      vivek    pts/1      0.00 secs Tue Nov 14 00:29
rm                      vivek    pts/1      0.00 secs Tue Nov 14 00:29

Search the accounting logs by terminal name pts/1
$ lastcomm pts/1

Task: summarizes accounting information

Use sa command to print summarizes information about previously executed commands. In addition, it condenses this data into a summary file named savacct which contains the number of times the command was called and the system resources used. The information can also be summarized on a per-user basis; sa will save this iinformation into a file named usracct.
# saOutput:

     579     222.81re       0.16cp     7220k
       4       0.36re       0.12cp    31156k   up2date
       8       0.02re       0.02cp    16976k   rpmq
       8       0.01re       0.01cp     2148k   netstat
      11       0.04re       0.00cp     8463k   grep
      18     100.71re       0.00cp    11111k   ***other*
       8       0.00re       0.00cp    14500k   troff
       5      12.32re       0.00cp    10696k   smtpd
       2       8.46re       0.00cp    13510k   bash
       8       9.52re       0.00cp     1018k   less

Take example of first line:
4 0.36re 0.12cp 31156k up2date

  • 0.36re “real time” in wall clock minutes
  • 0.12cp sum of system and user time in cpu minutes
  • 31156k cpu-time averaged core usage, in 1k units
  • up2date command name

Display output per-user:
# sa -uOutput:

root       0.00 cpu      595k mem accton
root       0.00 cpu    12488k mem initlog
root       0.00 cpu    12488k mem initlog
root       0.00 cpu    12482k mem touch
root       0.00 cpu    13226k mem psacct
root       0.00 cpu      595k mem consoletype
root       0.00 cpu    13192k mem psacct           *
root       0.00 cpu    13226k mem psacct
root       0.00 cpu    12492k mem chkconfig
postfix    0.02 cpu    10696k mem smtpd
vivek      0.00 cpu    19328k mem userhelper
vivek      0.00 cpu    13018k mem id
vivek      0.00 cpu    13460k mem bash             *
lighttpd   0.00 cpu    48240k mem php              *

Display the number of processes and number of CPU minutes on a per-user basis
# sa -mOutput:

                                      667     231.96re       0.17cp     7471k
root                                  544      51.61re       0.16cp     7174k
vivek                                 103      17.43re       0.01cp     8228k
postfix                                18     162.92re       0.00cp     7529k
lighttpd                                2       0.00re       0.00cp    48536k

Task: Find out who is eating CPU

By looking at re, k, cp/cpu (see above for output explanation) time you can find out suspicious activity or the name of user/command who is eating up all CPU. An increase in CPU/memory usage (command) is indication of problem.

Please note that above commands and packages also available on other UNIX like oses such as Sun Solaris and *BSD oses.

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin and a trainer for the Linux operating system/Unix shell scripting. He has worked with global clients and in various industries, including IT, education, defense and space research, and the nonprofit sector. Follow him on Twitter, Facebook, Google+.

63 comment

  1. I think there’s an error at
    Display output per-user:

    Display the number of processes and number of CPU minutes on a per-user basis

    shouldn’t be sa -u and sa -m ?

  2. So why is Solaris a “Unix like” OS

    It is certified as a version of Unix. Late 1990s Sun replaced the BSD-derived SunOS 4 with a version of UNIX System V Release 4 (SVR4),

    When I say UNIX like means other all oses such as OpenBSD, AIX, Linux, FreeBSD etc.

  3. Hi,

    Iam planning to take daily audit output , once after taking data of audit report i need to remove old data and take next day report only with out old data.

    Plze let me know , if u need more info.


  4. Is there a way to prevent this information from being modified? I assume that somebody who roots a machine would clear/alter the accounting records. Is there a way to have this send, say, over a serial link to another machine in realtime?

  5. Hi

    This is reagrding lastcomm in Linux.
    userhelper S X vivek pts/0 0.00 secs Mon Nov 13 23:58
    userhelper S vivek pts/0 0.00 secs Mon Nov 13 23:45
    rpmq vivek pts/0 0.01 secs Mon Nov 13 23:45

    what is this 0.00 or 0.00 or 0.01 secs its CPU time or actual time.
    What is Nov 12 23:45 etc


  6. The lastcomm command will not display the path from which command was executed as well as the argument passed to the command

  7. Unfortunately it looks like it doesn’t record the command unless it completes. So if you dump the box, you won’t be able to find the command that might have cause it to crash.

  8. Hi…

    I am installing psacct on my CentOS box. lastcomm and sa commands give no output. Infact in /var/account there is only a single file created pacct which too is 0 bytes. Am i missing some other configuration.

    Note: kernel version is
    / and /usr/local are read only partitions.
    /var has read-write permissions.

    Warm Regards

  9. Don’t forget to run following two commands:
    chkconfig psacct on
    /etc/init.d/psacct start

    After that wait for some time and run a few commands. Try again. Lemee know

  10. Any idea how to get a non-truncated command to print out? Example:


    …will display as something like…


    Does the acct process store the command as typed so that there I might be able to find a way to access it in the pacct file? Also, is there any way to open this pacct data file in some sort of editor to see what all is stored in it?


    Unca Xitron

  11. I am using SLES 9. I would like to get a log when my users have changed their password. How do I do it.

    In fact I found this available with SLES 10 in the /var/log/messages file. But not finding the same in SLES 9. Please help.



  12. To clear the file which saves the pacct information run this command.

    rm -f /var/account/pacct

    We have to re-initialize the pacct command as explained in the tutorial after executing this command.


  13. Can I trace back the variables in the command, say, instead of only knowing someone ran ping, but also know which ip address she ran ping against?

  14. This only seems to track the command and not the arguments used to run the command:

    Eg. rm -rf / would only show rm which is not much of any use.

    Am I missing something?

  15. Is there any way that we can log the arguments as well using this ? Can anyone suggest an alternative way of tracking users and the commands they have run?


  16. Thanks for the usefull howto,
    as stated above, i’m also missing the ability to log the variables given to a command, this would be very usefull in my opinion.

  17. @danny, @brein and others;

    There is a package called “snoopy” it wraps “exec” calls and logs any command that has been run, it logs user, command and parameters. It does not log shell builtins like cd and ls, as they do not call exec.

  18. I have a similar configuration to KM(15) above and have the same issues and problems he reported. Is there something I am missing? Below are some things I did on this machine and the results

    [[email protected] ~]# ac dheth
    total 306.91
    [[email protected] ~]# /etc/init.d/psacct status
    Process accounting is enabled.
    [[email protected] ~]# passwd dheth
    Changing password for user dheth.
    New UNIX password:
    BAD PASSWORD: it is based on a dictionary word
    Retype new UNIX password:
    passwd: all authentication tokens updated successfully.
    [[email protected] ~]# lastcomm passwd
    [[email protected] ~]# lastcomm –help
    Usage: lastcomm [-hV] [-f file] [command] … [user] … [terminal] …
    [–file ] [–strict-match] [–print-controls]
    [–user ] [–tty ] [–command ] [–debug]
    [–version] [–help]

    The system’s default process accounting file is /var/account/pacct.
    [[email protected] ~]# lastcomm –debug|more
    [[email protected] ~]# lastcomm –version
    lastcomm: GNU Accounting Utilities (release 6.3.2)
    [[email protected] ~]# ac –version
    ac: GNU Accounting Utilities (release 6.3.2)
    [[email protected] ~]# uname -a
    Linux sandbox.****.com 2.6.9-023stab046.2-enterprise #1 SMP Mon Dec 10 15:22:33 MSK 2007 i686 i686 i386 GNU/Linux

  19. I ran
    accton /var/log/pacct

    on all the machines I have. But while on one machine “sa -m” is showing correct output,
    other machines are giving error as follows :
    sa -m
    sa: ERROR — print_stats_nicely called with num_calls == 0
    Why is this happening ? Anything wrong?

  20. Hi there. I now that this is a really old post, but I need to ask something.

    For how long the lastcomm will store commands? Lets say I’ll install a linux box today and start the psacct service. In 4 months, commands from the first days of the machile will be there? Or it only keeps something like the last 30 commands?



  21. The accounting process is easy to use but SUDO audit file can give you a better audit trail. Sudo lists everything you need to know about what has been done on the machine, including the user who became the super user and the arguments were passed to the command, timestamp, etc the downside of SUDO log file is that it is not easy to work with, at least in my experience.

    Nice tutorial



  22. I started the “psacct” service, but still “accton” is not showing any output while other commands are working like sa, lastcomm & ac.
    Where am i doing wrong?

  23. Hi,

    Im interested to know is there a way to log all keystrokes including typos & complete command parameters with this say $rm -rf /* or custom scripts such as $./foo
    My requirement is to log *EVERYTHING* the command and its associated flags and parameters.


  24. A more suitable way on modern distros (Centos 5 + ) is to install the pam_tty_audit pam module and place it into a pam stack for say, login or sshd.

    This will produce a key by key typing of what was pressed inside the session. (including, up, down backspace, etc)

    You can use ausearch to look through it.

    I.E “ausearch -ts recent -m tty -i”

  25. Hi,
    pam_tty_audit is kool, but problem is that it logs only root level stuff. Is there anyway to make it log user level tty input without patching it?


  26. Thanks Deleriux, I tried that but no luck :S. It seems im getting it wrong somewhere:

    [[email protected] ~]$ sudo vim /etc/pam.d/sshd
    [[email protected] ~]$ cat /etc/pam.d/sshd
    auth include system-auth
    account required
    account include system-auth
    password include system-auth
    session optional force revoke
    session include system-auth
    session required
    session required disable=* enable=*
    [[email protected] ~]$ date
    Wed Sep 23 11:29:16 NZST 2009
    [[email protected] ~]$ sudo su –
    [[email protected] ~]# service sshd restart
    Stopping sshd: [ OK ]
    Starting sshd: [ OK ]
    login as: fimz
    [email protected]’s password:
    Last login: Wed Sep 23 11:24:02 2009 from
    [[email protected] ~]$
    [[email protected] ~]$ ls
    [[email protected] ~]$ ls -a
    . .bashrc .gconfd .gtkrc-1.2-gnome2 .redhat
    .. Desktop .gnome .ICEauthority .Trash
    .bash_history .dmrc .gnome2 .metacity .viminfo
    .bash_logout .eggcups .gnome2_private .mozilla
    .bash_profile .gconf .gstreamer-0.10 .nautilus
    [[email protected] ~]$ ls -l
    total 8
    drwxr-xr-x 2 fimz fimz 4096 Sep 10 16:02 Desktop
    [[email protected] ~]$ mount
    /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
    proc on /proc type proc (rw)
    sysfs on /sys type sysfs (rw)
    devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    /dev/sda1 on /boot type ext3 (rw)
    tmpfs on /dev/shm type tmpfs (rw)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
    [[email protected] ~]$ date
    Wed Sep 23 11:31:05 NZST 2009

    [[email protected] ~]$ sudo grep mount /var/log/*
    /var/log/secure:Sep 23 11:31:56 localhost sudo: fimz : TTY=pts/1 ; PWD=/home/fimz ; USER=root ; COMMAND=/bin/grep mount /var/log/acpid /var/log/anaconda.log /var/log/anaconda.syslog /var/log/anaconda.xlog /var/log/audit /var/log/boot.log /var/log/boot.log.1 /var/log/boot.log.2 /var/log/btmp /var/log/conman /var/log/conman.old /var/log/cron /var/log/cron.1 /var/log/cron.2 /var/log/cups /var/log/dmesg /var/log/faillog /var/log/gdm /var/log/httpd /var/log/lastlog /var/log/mail /var/log/maillog /var/log/maillog.1 /var/log/maillog.2 /var/log/messages /var/log/messages.1 /var/log/messages.2 /var/log/news /var/log/pm /var/log/ppp /var/log/prelink /var/log/rpmpkgs /var/log/rpmpkgs.1 /var/log/rpmpkgs.2 /var/log/samba /var/log/scrollkeeper.log /var/log/secure /var/log/secure.1 /var/log/secure.2 /var/log/setroubleshoot /var/log/spooler /var/log/spooler.1 /var/log/spooler.2 /var/log/squid /var/log/tallylog /var/log/vbox /var/log/wtmp /var/log/Xorg.0.log /var/log/Xorg.0.log.old /var/log/yum.log

    I was expecting it to log to messages, but apparently its not!

    1. I’m running with pam_tty_audit and it appears it only logs root level key events, regardless of the settings in the session entry. It seems to ignore the disable= and enable= settings. The man page has the example of enabling root, but specifying a different user (ex enable=username ) still logs root only.

  27. Snoopy compiles and loads, but so far it seems to work only to log root commands, and I modified snoopy.h
    #define SNOOPY_ROOT_ONLY 0
    with both 0 or 1, makes no difference, keeps logging only root commands.

  28. I’m not totally sure yet, but support for process accounting has to be also enabled in the kernel to make all features of acct work.

  29. hi, i just installed snoopy on fedora 12. but i dont know how to view log. where snoopy`s log file located? could u explain me pls/ i read instruction file but nothing found something that help me. sry my bad english

  30. Hi,

    I gone through the entire comments but nobody clearly there any tool for tracking the user executed commands and path of the file/folder.

    Like [[email protected] pravi]# pwd
    [[email protected] pravi]# cat test
    [[email protected] pravi]#

  31. I’ve taken a project to work upon tracing of runtime activities on unix system
    into a log file. Like, to implement a program which will show the log of everything
    happened in past, including many requirements, like applications i used (with the time of access),
    kind of files/directories i opened, closed, created, deleted(with the time), etc.
    Please suggest me something to do it in a better way.

    1. rm vivek pts/0 0.00 secs Tue Nov 14 00:30
      rm vivek pts/1 0.00 secs Tue Nov 14 00:30
      rm vivek pts/1 0.00 secs Tue Nov 14 00:29
      rm vivek pts/1 0.00 secs Tue Nov 14 00:29

      vivek is remove many files. The command just showing “rm” command, So how could i know what did he remove ??????

      Please help

    2. ankit, let me know if you have done with your project. I am also doing the same project and I really need help in that.

  32. Just knowing the command (by itself) the user ran is useless. You absolutely need it’s arguments, and the parent process’s CWD. Other than that, it’s a “cute” toy, and little else.

  33. I need to implement a program in shell programming which will show me the log of everything happened in past. It should show me who has logged in and list its activities. It can include many requirements. Kind of applications he/she used (with the time of access), kind of files/directories he has opened, closed, created, deleted (with the time) and many more information. please help me for the same.

  34. I wrote a method to log all ‘bash’ commands/builtins into a text-file or a ‘syslog’ server without using a patch or a special executable tool.

    It is very easy to deploy, as it is a simple shellscript that need to be called once at the initialization of the ‘bash’.

    see the method here

    Francois Scheurer

  35. Hello,
    i want to know which ip logged into my centos server and deleted my data. Is there a command or some script you can provide me ? I’ve lost my client’s all data but unable to track ip log, which used command “rm -Rf ” to remove client’s data. Help will be highly appreciated. THanks

  36. Hi This is good, but it is not enough as context change and knowing that a file was deleted doesn’t explain why it was.
    How can I record both the command line and the screen result ?
    Cherry on the cake would be to export in real time the output to syslog so that user can’t modify it.
    That would be superbe !

  37. I have a question about how far back the ‘ac -p’ command goes. Is it for the current month? I have been using the a cronjob to process the wtmp file each month for accounting purposes but I’m looking for some alternate ways to do this. I would really like to be able to organize my individual user logins by the groups they are in but it does not seem like this information is tracked by wtmp.

    Thanks for any assistance you can offer.

  38. is there any way to analyse log without going to all log files , it is very tedious , will aurport or ausearch work or they work only with log generated by auditd ?

  39. Hi,

    My question is also simple. Is it possible to capture the arguments after rm command or kill command through lastcomm?

    like I am doing rm -rf myfolder
    rm -f myfile
    kill -9 123456789

    so I want the lastcomm should capture both the rm/kill commands & the argument.

    Thanks in advance

  40. thanks, i am using this tools these days , but i found that the information accounted by psacct will reset several days once. do you know how to change it because i want to monitor my computer for a long time. thanks again.

  41. Is there such a way to find the location where the Command is issued using ‘lastcomm’.. ?? plz help..

  42. installed and run the programm. the total log time seems to be working fine, but when I specifically search for commands from user (lastcomm ‘user’) it doesnt display anything.

    What exactly should be showing here? I believe it should at least display apt-get commands (thats how i wanted to try it). furthermore will the commands per user also be logged when it has been denied? i.e. i log in as normal user and want to apt-get upgrade.

  43. Hi,
    May I know how to find the fraction of the total audit storage capacity available in any Linux flavor?Preferably Redhat and Ubuntu.

    If any one has an idea on this, please let me know which command I can use.


Comments are closed.