Loading ...How to keep a detailed audit trail of what’s being done on your Linux systems
Intrusions can take place from both authorized (insiders) and unauthorized (outsiders) users. My personal experience shows that unhappy user can damage the system, especially when they have a shell access. Some users are little smart and removes history file (such as ~/.bash_history) but you can monitor all user executed commands.
It is recommended that you log user activity using process accounting. Process accounting allows you to view every command executed by a user including CPU and memory time. With process accounting sys admin always find out which command executed at what time 
The psacct package contains several utilities for monitoring process activities, including ac, lastcomm, accton and sa.
- The ac command displays statistics about how long users have been logged on.
- The lastcomm command displays information about previous executed commands.
- The accton command turns process accounting on or off.
- The sa command summarizes information about previously executed commmands.
Task: Install psacct or acct package
Use up2date command if you are using RHEL ver 4.0 or less
# up2date psacct
Use yum command if you are using CentOS/Fedora Linux / RHEL 5:
# yum install psacct
Use apt-get command if you are using Ubuntu / Debian Linux:
$ sudo apt-get install acct OR # apt-get install acct
Task: Start psacct/acct service
By default service is started on Ubuntu / Debian Linux by creating /var/account/pacct file. But under Red Hat /Fedora Core/Cent OS you need to start psacct service manually. Type the following two commands to create /var/account/pacct file and start services:
# chkconfig psacct on
# /etc/init.d/psacct start
If you are using Suse Linux, the name of service is acct. Type the following commands:
# chkconfig acct on
# /etc/init.d/acct start
Now let us see how to utilize these utilities to monitor user commands and time.
Task: Display statistics about users' connect time
ac command prints out a report of connect time in hours based on the logins/logouts. A total is also printed out. If you type ac without any argument it will display total connect time:
$ acOutput:
total 95.08
Display totals for each day rather than just one big total at the end:
$ ac -dOutput:
Nov 1 total 8.65 Nov 2 total 5.70 Nov 3 total 13.43 Nov 4 total 6.24 Nov 5 total 10.70 Nov 6 total 6.70 Nov 7 total 10.30 ..... .. ... Nov 12 total 3.42 Nov 13 total 4.55 Today total 0.52
Display time totals for each user in addition to the usual everything-lumped-into-one value:
$ ac -pOutput:
vivek 87.49
root 7.63
total 95.11
Task: find out information about previously executed user commands
Use lastcomm command which print out information about previously executed commands. You can search command using usernames, tty names, or by command names itself.
Display command executed by vivek user:
$ lastcomm vivekOutput:
userhelper S X vivek pts/0 0.00 secs Mon Nov 13 23:58 userhelper S vivek pts/0 0.00 secs Mon Nov 13 23:45 rpmq vivek pts/0 0.01 secs Mon Nov 13 23:45 rpmq vivek pts/0 0.00 secs Mon Nov 13 23:45 rpmq vivek pts/0 0.01 secs Mon Nov 13 23:45 gcc vivek pts/0 0.00 secs Mon Nov 13 23:45 which vivek pts/0 0.00 secs Mon Nov 13 23:44 bash F vivek pts/0 0.00 secs Mon Nov 13 23:44 ls vivek pts/0 0.00 secs Mon Nov 13 23:43 rm vivek pts/0 0.00 secs Mon Nov 13 23:43 vi vivek pts/0 0.00 secs Mon Nov 13 23:43 ping S vivek pts/0 0.00 secs Mon Nov 13 23:42 ping S vivek pts/0 0.00 secs Mon Nov 13 23:42 ping S vivek pts/0 0.00 secs Mon Nov 13 23:42 cat vivek pts/0 0.00 secs Mon Nov 13 23:42 netstat vivek pts/0 0.07 secs Mon Nov 13 23:42 su S vivek pts/0 0.00 secs Mon Nov 13 23:38
For each entry the following information is printed. Take example of first output line:
userhelper S X vivek pts/0 0.00 secs Mon Nov 13 23:58
Where,
- userhelper is command name of the process
- S and X are flags, as recorded by the system accounting routines. Following is the meaning of each flag:
- S -- command executed by super-user
- F -- command executed after a fork but without a following exec
- D -- command terminated with the generation of a core file
- X -- command was terminated with the signal SIGTERM
- vivek the name of the user who ran the process
- prts/0 terminal name
- 0.00 secs - time the process exited
Search the accounting logs by command name:
$ lastcomm rmOutput:
$ lastcomm passwd
rm S root pts/0 0.00 secs Tue Nov 14 00:39 rm S root pts/0 0.00 secs Tue Nov 14 00:39 rm S root pts/0 0.00 secs Tue Nov 14 00:38 rm S root pts/0 0.00 secs Tue Nov 14 00:38 rm S root pts/0 0.00 secs Tue Nov 14 00:36 rm S root pts/0 0.00 secs Tue Nov 14 00:36 rm S root pts/0 0.00 secs Tue Nov 14 00:35 rm S root pts/0 0.00 secs Tue Nov 14 00:35 rm vivek pts/0 0.00 secs Tue Nov 14 00:30 rm vivek pts/1 0.00 secs Tue Nov 14 00:30 rm vivek pts/1 0.00 secs Tue Nov 14 00:29 rm vivek pts/1 0.00 secs Tue Nov 14 00:29
Search the accounting logs by terminal name pts/1
$ lastcomm pts/1
Task: summarizes accounting information
Use sa command to print summarizes information about previously executed commands. In addition, it condenses this data into a summary file named savacct which contains the number of times the command was called and the system resources used. The information can also be summarized on a per-user basis; sa will save this iinformation into a file named usracct.
# saOutput:
579 222.81re 0.16cp 7220k
4 0.36re 0.12cp 31156k up2date
8 0.02re 0.02cp 16976k rpmq
8 0.01re 0.01cp 2148k netstat
11 0.04re 0.00cp 8463k grep
18 100.71re 0.00cp 11111k ***other*
8 0.00re 0.00cp 14500k troff
5 12.32re 0.00cp 10696k smtpd
2 8.46re 0.00cp 13510k bash
8 9.52re 0.00cp 1018k less
Take example of first line:
4 0.36re 0.12cp 31156k up2date
Where,
- 0.36re "real time" in wall clock minutes
- 0.12cp sum of system and user time in cpu minutes
- 31156k cpu-time averaged core usage, in 1k units
- up2date command name
Display output per-user:
# sa -uOutput:
root 0.00 cpu 595k mem accton root 0.00 cpu 12488k mem initlog root 0.00 cpu 12488k mem initlog root 0.00 cpu 12482k mem touch root 0.00 cpu 13226k mem psacct root 0.00 cpu 595k mem consoletype root 0.00 cpu 13192k mem psacct * root 0.00 cpu 13226k mem psacct root 0.00 cpu 12492k mem chkconfig postfix 0.02 cpu 10696k mem smtpd vivek 0.00 cpu 19328k mem userhelper vivek 0.00 cpu 13018k mem id vivek 0.00 cpu 13460k mem bash * lighttpd 0.00 cpu 48240k mem php *
Display the number of processes and number of CPU minutes on a per-user basis
# sa -mOutput:
667 231.96re 0.17cp 7471k root 544 51.61re 0.16cp 7174k vivek 103 17.43re 0.01cp 8228k postfix 18 162.92re 0.00cp 7529k lighttpd 2 0.00re 0.00cp 48536k
Task: Find out who is eating CPU
By looking at re, k, cp/cpu (see above for output explanation) time you can find out suspicious activity or the name of user/command who is eating up all CPU. An increase in CPU/memory usage (command) is indication of problem.
Please note that above commands and packages also available on other UNIX like oses such as Sun Solaris and *BSD oses.
Want to stay up to date with the latest Linux tips, news and announcements? Subscribe to our free e-mail newsletter or RSS feed to get all updates.
You can Email this page to a friend.
You may also be interested in other helpful articles:
- Finding out who is logged on to a Solaris system
- GnuCash for personal accounting - HOWTO
- When a user logs in what files are updated in UNIX / Linux
- Top 10 Hits and Zeitgeist of 2006 for your edification and fun
- Killing zombie process
Discussion on This Article:
Leave a Reply
We encourage your comments, and suggestions. But please stay on topic, be polite, and avoid spam. Thank you very much for stopping by our site!
Tags: accounting, accton, ac_command, apt-get_command, audit_users, lastcomm_command, monitor_user_executed_commands, rhel, sa_command, start_acct_command, start_psacct_command, sys_admin, up2date_command, users_connect_time, yum_command
It’s “sa”, not “su”…
I think there’s an error at
Display output per-user:
Display the number of processes and number of CPU minutes on a per-user basis
shouldn’t be sa -u and sa -m ?
@Andreas and GeorgeB,
Typo is fixed.
Thanks for heads up!
So why is Solaris a “Unix like” OS. Last time I looked it was based on System V.
So why is Solaris a “Unix like” OS
It is certified as a version of Unix. Late 1990s Sun replaced the BSD-derived SunOS 4 with a version of UNIX System V Release 4 (SVR4),
When I say UNIX like means other all oses such as OpenBSD, AIX, Linux, FreeBSD etc.
Hi,
Any command to flush old data like daily basis ?
Lokesh,
Sorry but I don’t get your question…
On Debian/Ubuntu systems it is in the ‘acct’ package and can be installed with ‘apt-get install acct’.
Hi,
Iam planning to take daily audit output , once after taking data of audit report i need to remove old data and take next day report only with out old data.
Plze let me know , if u need more info.
Thanx
Lok
Is there a way to prevent this information from being modified? I assume that somebody who roots a machine would clear/alter the accounting records. Is there a way to have this send, say, over a serial link to another machine in realtime?
Hi
This is reagrding lastcomm in Linux.
userhelper S X vivek pts/0 0.00 secs Mon Nov 13 23:58
userhelper S vivek pts/0 0.00 secs Mon Nov 13 23:45
rpmq vivek pts/0 0.01 secs Mon Nov 13 23:45
what is this 0.00 or 0.00 or 0.01 secs its CPU time or actual time.
What is Nov 12 23:45 etc
Regards
JP
Hello,
0.00 secs Mon Nov 13 23:58 ===> time the process exited
HTH
The lastcomm command will not display the path from which command was executed as well as the argument passed to the command
Unfortunately it looks like it doesn’t record the command unless it completes. So if you dump the box, you won’t be able to find the command that might have cause it to crash.
Hi…
I am installing psacct on my CentOS box. lastcomm and sa commands give no output. Infact in /var/account there is only a single file created pacct which too is 0 bytes. Am i missing some other configuration.
Note: kernel version is 2.6.15.1
/ and /usr/local are read only partitions.
/var has read-write permissions.
Warm Regards
KM.
Don’t forget to run following two commands:
chkconfig psacct on/etc/init.d/psacct start
After that wait for some time and run a few commands. Try again. Lemee know
Any idea how to get a non-truncated command to print out? Example:
/home/charlie/my_favorite/command/in/the/world
…will display as something like…
/home/charlie/my_fav
Does the acct process store the command as typed so that there I might be able to find a way to access it in the pacct file? Also, is there any way to open this pacct data file in some sort of editor to see what all is stored in it?
Thanks!
Unca Xitron
I am using SLES 9. I would like to get a log when my users have changed their password. How do I do it.
In fact I found this available with SLES 10 in the /var/log/messages file. But not finding the same in SLES 9. Please help.
Regards,
Soumen.
To clear the file which saves the pacct information run this command.
rm -f /var/account/pacct
We have to re-initialize the pacct command as explained in the tutorial after executing this command.
Cheers
Koushik
Can I trace back the variables in the command, say, instead of only knowing someone ran ping, but also know which ip address she ran ping against?
This only seems to track the command and not the arguments used to run the command:
Eg. rm -rf / would only show rm which is not much of any use.
Am I missing something?
Danny
Is there any way that we can log the arguments as well using this ? Can anyone suggest an alternative way of tracking users and the commands they have run?
Regards
Sajindra
Thanks for the usefull howto,
as stated above, i’m also missing the ability to log the variables given to a command, this would be very usefull in my opinion.
greets