HowTo: Create sar Graphs With kSar [ Identifying Linux Bottlenecks ]

by on December 15, 2009 · 34 comments· LAST UPDATED December 15, 2009

in , ,

The sar command collect, report, or save UNIX / Linux system activity information. It will save selected counters in the operating system to the /var/log/sa/sadd file. From the collected data, you get lots of information about your server:

  1. CPU utilization
  2. Memory paging and its utilization
  3. Network I/O, and transfer statistics
  4. Process creation activity
  5. All block devices activity
  6. Interrupts/sec etc.

sar output can be used for identifying server bottlenecks. However, analyzing information provided by sar can be difficult, so use kSar, which can take sar output and plot a nice easy to understand graph over period of time.

sysstat Package

The sar, sa1, and sa2 commands are part of sysstat package:

  1. sar : Displays the data.
  2. sa1 and sa2: Collect and store the data for later analysis. The sa2 shell script write a daily report in the /var/log/sa directory. The sa1 shell script collect and store binary data in the system activity daily data file.
  3. sadc - System activity data collector. You can configure various options by modifying sa1 and sa2 scripts. They are located at the following location:
    • /usr/lib64/sa/sa1 (64bit) or /usr/lib/sa/sa1 (32bit) - This calls sadc to log reports to/var/log/sa/sadX format.
    • /usr/lib64/sa/sa2 (64bit) or /usr/lib/sa/sa2 (32bit) - This calls sar to log reports to /var/log/sa/sarX format.

How Do I Install sar?

Type the following command:
# yum install sysstat
Sample outputs:

Loaded plugins: downloadonly, fastestmirror, priorities,
              : protectbase, security
Loading mirror speeds from cached hostfile
 * addons: mirror.cs.vt.edu
 * base: mirror.ash.fastserv.com
 * epel: serverbeach1.fedoraproject.org
 * extras: mirror.cogentco.com
 * updates: centos.mirror.nac.net
0 packages excluded due to repository protections
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package sysstat.x86_64 0:7.0.2-3.el5 set to be updated
--> Finished Dependency Resolution
Dependencies Resolved
====================================================================
 Package        Arch          Version             Repository   Size
====================================================================
Installing:
 sysstat        x86_64        7.0.2-3.el5         base        173 k
Transaction Summary
====================================================================
Install      1 Package(s)
Update       0 Package(s)
Remove       0 Package(s)
Total download size: 173 k
Is this ok [y/N]: y
Downloading Packages:
sysstat-7.0.2-3.el5.x86_64.rpm               | 173 kB     00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : sysstat                                      1/1
Installed:
  sysstat.x86_64 0:7.0.2-3.el5
Complete!

Configuration Files

Edit /etc/sysconfig/sysstat file specify how long to keep log files in days, maximum is a month:
# vi /etc/sysconfig/sysstat
Sample outputs:

# keep log for 28 days
# the default is 7
HISTORY=28

Save and close the file.

The Default Cron Job

The default cron job is located at /etc/cron.d/sysstat:
# cat /etc/cron.d/sysstat
Sample outputs:

# run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A

Tell sadc To Report Statistics For Disks

Edit /etc/cron.d/sysstat, enter:
# vi /etc/cron.d/sysstat
Update it as follows to log all disk stats (the -d option force to log stats for each block device and the -I option force report statistics for all system interrupts)
# run system activity accounting tool every 10 minutes

*/10 * * * * root /usr/lib64/sa/sa1 -I -d 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A 

Save and close the file. Turn on the service, enter:
# chkconfig sysstat on
# service sysstat start

Sample outputs:

Calling the system activity data collector (sadc): 

How Do I Use sar? How do I View Stats?

Use the sar command to display output the contents of selected cumulative activity counters in the operating system. In this example, sar is run to get real-time reporting from the command line about CPU utilization:
# sar -u 3 10
Sample outputs:

Linux 2.6.18-164.2.1.el5 (www-03.nixcraft.in) 	12/14/2009
09:49:47 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:49:50 PM       all      5.66      0.00      1.22      0.04      0.00     93.08
09:49:53 PM       all     12.29      0.00      1.93      0.04      0.00     85.74
09:49:56 PM       all      9.30      0.00      1.61      0.00      0.00     89.10
09:49:59 PM       all     10.86      0.00      1.51      0.04      0.00     87.58
09:50:02 PM       all     14.21      0.00      3.27      0.04      0.00     82.47
09:50:05 PM       all     13.98      0.00      4.04      0.04      0.00     81.93
09:50:08 PM       all      6.60      6.89      1.26      0.00      0.00     85.25
09:50:11 PM       all      7.25      0.00      1.55      0.04      0.00     91.15
09:50:14 PM       all      6.61      0.00      1.09      0.00      0.00     92.31
09:50:17 PM       all      5.71      0.00      0.96      0.00      0.00     93.33
Average:          all      9.24      0.69      1.84      0.03      0.00     88.20

Where,

  • 3 = interval
  • 10 = count

To view process creation statistics, enter:
# sar -c 3 10
To view I/O and transfer rate statistics, enter:
# sar -b 3 10
To view paging statistics, enter:
# sar -B 3 10
To view block device statistics, enter:
# sar -d 3 10
To view statistics for all interrupt statistics, enter:
# sar -I XALL 3 10
To view device specific network statistics, enter:
# sar -n DEV 3 10
# sar -n EDEV 3 10

To view CPU specific statistics, enter:
# sar -P ALL
# Only 1st CPU stats
# sar -P 1 3 10

To view queue length and load averages statistics, enter:
# sar -q 3 10
To view memory and swap space utilization statistics, enter:
# sar -r 3 10
# sar -R 3 10

To view status of inode, file and other kernel tables statistics, enter:
# sar -v 3 10
To view system switching activity statistics, enter:
# sar -w 3 10
To view swapping statistics, enter:
# sar -W 3 10
To view statistics for a given process called Apache with PID # 3256, enter:
# sar -x 3256 3 10

Say Hello To kSar

sar and sadf provides CLI based output. The output may confuse all new users / sys admin. So you need to use kSar which is a java application that graph your sar data. It also permit to export data to PDF/JPG/PNG/CSV. You can load data from three method : local file, local command execution, and remote command execution via SSH. kSar supports the sar output of the following OS:

  1. Solaris 8, 9 and 10
  2. Mac OS/X 10.4+
  3. Linux (Systat Version >= 5.0.5)
  4. AIX (4.3 & 5.3)
  5. HPUX 11.00+

Download And Install kSar

Visit the official website and grab the latest source code. Use wget to download the source code, enter:
$ wget http://ksar.atomique.net/kSar/kSar-5.0.6.zip
Use unzip command to extract files, enter:
$ unzip kSar-5.0.6.zip

How Do I Run kSar?

Make sure JAVA jdk is installed and working correctly. Type the following command to start kSar, run:
$ cd kSar-5.0.6/
$ sh run.sh

Fig.01: kSar welcome screen

Fig.01: kSar welcome screen


Next you will see main kSar window, and menus with two panels.
Fig.02: kSar - the main window

Fig.02: kSar - the main window


The left one will have a list of graphs available depending on the data kSar has parsed. The right window will show you the graph you have selected.

How Do I Generate sar Graphs Using kSar?

First, you need to grab sar command statistics. Type the following command to get stats, enter (type it on your server):
[server1 ]# LC_ALL=C sar -A > /tmp/sar.data.txt
# copy file to local desktop
[desktop ]$ scp user@server1.nixcraft.com:/tmp/sar.data.txt /tmp

Click on Data > Load data from text file > Select sar.data.txt > Open button.

Now, the graph type tree is deployed in left pane and a graph has been selected:

Fig.03: Processes for server1

Fig.03: Processes for server1

Fig.03: Disk stats (blok device) stats for server1

Fig.04: Disk stats (blok device) stats for server1

Fig.05: Memory stats for server1

Fig.05: Memory stats for server1

Zoom in and out

Using the move, you can interactively zoom onto up a part of a graph. To select a zone to zoom, click on the upper left conner and while still holding the mouse but on move to the lower-right of the zone you want to zoom. To come back to unzoomed view click and drag the mouse to any corner location except a lower-right one. You can also right click and select zoom options

Understanding kSar Graphs And sar Data

I strongly recommend reading sar and sadf command man page:
$ man sar
$ man sadf

kSar graphs can be understood from the sar command man page or by reading its official documentation.

Case Study: Identifying Linux Server CPU Bottlenecks

With sar and kSar you get detailed snapshot of memory, CPU and other subsystems. For example, if CPU utilization is more than 80% for a long period of time, a CPU bottleneck is most likely occurring. Using sar -x ALL you can find out CPU eating process. The output of mpstat command (part of sysstat package itself) will also help you understand the cpu utilization. You can easily analyzing this information with kSar.

I Found CPU Bottlenecks...

Performance tuning options for the CPU are as follows:

  1. Make sure that no unnecessary programs are running in background. Turn off unnecessary services.
  2. Use cron to schedule jobs (e.g., backup) to run at off-peak hours.
  3. Use top and ps command to find out all non-critical background jobs / services. Make sure you lower their priority using renice command.
  4. Use taskset command to set a processes's CPU affinity (offload cpu) i.e. bind processes to different CPUs. For example, run MySQL database on cpu #2 and Apache on cpu # 3.
  5. Make sure you are using latest drivers and firmware for your server.
  6. If possible add additional SMP cpus to system.
  7. Use faster CPUs for a single-threaded application (e.g. Lighttpd web server app).
  8. Use more CPUs for a multi-threaded application (e.g. MySQL database server app).
  9. Use more computer nodes and setup a load balancer for web app.

isag - Interactive System Activity Grapher (alternate tool)

The isag command graphically displays the system activity data stored in a binary data file by a previous sar run. The isag command invokes sar to extract the data to be plotted. isag has limited set of options as compare to kSar.

Fig.06: isag CPU utilization graphs

Fig.06: isag CPU utilization graphs

References:

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 34 comments… read them below or add one }

1 Bhavya December 15, 2009 at 10:00 am

I think u’ve given the wrong path to the images (screenshots of ksar), They are not being loaded, please check & update.


Bhavya

Reply

2 nixCraft December 15, 2009 at 10:31 am

@Bhavya,

They are loading fine here. I do not see any problem.

wget http://files.cyberciti.biz/uploads/tips/2009/12/memory-ksar.png
O/p

--2009-12-15 16:00:35--  http://files.cyberciti.biz/uploads/tips/2009/12/memory-ksar.png
Resolving files.cyberciti.biz... 203.190.126.132
Connecting to files.cyberciti.biz|203.190.126.132|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 100548 (98K) [image/png]
Saving to: `memory-ksar.png'
100%[====================================================================================================================>] 1,00,548    68.5K/s   in 1.4s
2009-12-15 16:00:37 (68.5 KB/s) - `memory-ksar.png' saved [100548/100548]

Reply

3 Jaysunn December 15, 2009 at 1:38 pm

I have used KSAR many times to provide quality graphs for management. What A great tutorial.

Excellent.

Jaysunn

Reply

4 Mark Stoecker December 15, 2009 at 8:21 pm

Thank you very much for the post. I enjoy reading your blog. I am a MySQL DBA and this will help me tremendously in my job. Thanks again!!

Reply

5 sims December 16, 2009 at 12:21 am

Very very very interesting. I haven’t heard of sar or ksar. Good info! Thanks!

Reply

6 nixCraft December 16, 2009 at 6:04 am

@Jaysunn,

Yes those graphs makes more sense with management folks.

@mark / sims – thanks for comment.

Reply

7 Mario Vanoni December 16, 2009 at 2:15 pm

sysstat-7.0.2-3.el5.x86_64.rpm
very old version, see
http://pagesperso-orange.fr/sebastien.godard/
Wednesday November 11th, 2009
Sysstat 9.0.6 released (stable version).
feedback only

Reply

8 nixCraft December 16, 2009 at 3:04 pm

@Mario,

Thanks for the feedback. However, RHEL / CentOS got older stable version. I already found some good stuff in latest version.

Reply

9 Tapas Mallick December 17, 2009 at 6:30 am

Hi,
“Download PDF” lisk is not working, showing “410 Gone”.
Regards,
Tapas

Reply

10 nixCraft December 17, 2009 at 5:03 pm

@Tapas,

I’ve tweaked code a little bit. Try again. This time it should work!

Reply

11 hemanth.hm December 18, 2009 at 5:32 pm
12 Liju December 19, 2009 at 4:37 am

Great info !!!!!

Really help me lots to get a portable output. Earlier I was used cacti to do this. It’s a online tool and required huge SNMP traffic to analyze. But it’s simple, easy to configure but rich in output.

Thanks,
Liju

Reply

13 Jalal Hajigholamali January 11, 2010 at 4:00 am

Hi,
Useful package, thanks a lot

Reply

14 nima0102 February 15, 2010 at 3:03 pm

Hi
First of all thanks for your good article.
I am using munin as monitoring system resources and I think you try it.

Thanks

Reply

15 Tarique February 16, 2010 at 11:01 am

Useful link!! Thanks

Reply

16 john June 22, 2010 at 3:26 pm

What if you did not generate SAR file using the locale LC_ALL=C prefix?

I have a sar output in text (without using the locale prefix), but kSar doesn’t parse it. It only knows first timestamp and that’s it.

Any options? Can I tweak the file to see the output?

Reply

17 Kester RIley January 19, 2011 at 10:03 am

Why when I did “sar -A 5 0″ yesterday did it work fine

today after a nw install on a new machine it always opens the file since midnight, and does not update every 5 seconds. Infact nothing I do can make it update “Live” it always opens from midnight to now????

Reply

18 tangosara1 September 16, 2011 at 9:12 pm

Excellent information and very nicely explained. Much appreciate all the effort in doing so. Cheers, Tangosara1

Reply

19 Jacobo September 21, 2011 at 8:12 pm

Hi, I took the liberty to create a script to get the sar statistics, please feel free to modify it if you like it

Thank you.
Jacobo.

#!/bin/bash
#
# Script to get sar statistics#
## sar_stats.sh ##
echo "Enter the number of intervals (seconds) for the sar statistics:"
read int
echo "enter the number of output lines (count) to process:"
read cnt
while :
do
clear
echo "*******************************"
echo "* Choose from the following: *"
echo "*******************************"
echo "* [1] To view process creation statistics *"
echo "* [2] To view I/O and transfer rate statistics *"
echo "* [3] To view paging statistics *"
echo "* [4] To view block device statistics *"
echo "* [5] To view statistics for all interrupt statistics *"
echo "* [6] To view device specific network statistics *"
echo "* [7] To view extended network statistics *"
echo "* [8] To view CPU specific statistics *"
echo "* [9] To view Only 1st CPU stats *"
echo "* [10] To view queue length and load averages statistics *"
echo "* [11] To view memory and swap space utilization statistics *"
echo "* [12] To view inode, file and other kernel tables statistics*"
echo "* [13] To view system switching activity statistics *"
echo "* [14] To view swapping statistics *"
echo "* [15] To view statistics for a given process ( Example: a process called Apache with PID # 3256) *"
echo "* [0] Exit *"
echo "************************"
echo -n "Enter your menu choice [0-15]: "
D=`/bin/date '+%B.%d'`
INT=$int
CNT=$cnt
read mychoice
case $mychoice in
1) echo "process creation statistics";
sar -c $INT $CNT |tee /tmp/sar_proc_stat_$D
echo "This file will be saved to /tmp/sar_pc_stat_$D"
sleep 3 ;;
2) echo "I/O and transfer rate statistics"
sar -b $INT $CNT |tee /tmp/sar_IO_TR_stat_$D
echo "This file will be saved to /tmp/sar_IO_TR_stat_$D"
sleep 3 ;;
3) echo "paging statistics"
sar -B $INT $CNT |tee /tmp/sar_pag_stat_$D
echo "This file will be saved to /tmp/sar_pag_stat_$D"
sleep 3 ;;
4) echo "block device statistics"
sar -d $INT $CNT |tee /tmp/sar_block_stat_$D
echo "This file will be saved to /tmp/sar_block_stat_$D"
sleep 3 ;;
5) echo "all interrupt statistics statistics"
sar -I XALL $INT $CNT |tee /tmp/sar_Intr_stat_$D
echo "This file will be saved to /tmp/sar_Intr_stat_$D"
sleep 3 ;;
6) echo "specific network statistics"
sar -n DEV $INT $CNT |tee /tmp/sar_Net_stat_$D
echo "This file will be saved to /tmp/sar_Net_stat_$D"
sleep 3 ;;
7) echo "Extended network statistics"
sar -n EDEV $INT $CNT |tee /tmp/sar_ENet_stat_$D
echo "This file will be saved to /tmp/sar_ENet_stat_$D"
sleep 3 ;;
8) echo "CPU specific statistics"
sar -P ALL $INT $CNT |tee /tmp/sar_CPU_stat_$D
echo "This file will be saved to /tmp/sar_CPU_stat_$D"
sleep 3 ;;
9) echo "1st CPU statistics"
sar -P 1 $INT $CNT |tee /tmp/sar_1CPU_stat_$D
echo "This file will be saved to /tmp/sar_1CPU_stat_$D"
sleep 3 ;;
10) echo "queue length and load averages statistics"
sar -q $INT $CNT |tee /tmp/sar_queue_stat_$D
echo "This file will be saved to /tmp/sar_queue_stat_$D"
sleep 3 ;;
11) echo "memory and swap space utilization statistics"
sar -r $INT $CNT |tee /tmp/sar_mem-swap_stat_$D
echo "This file will be saved to /tmp/sar_mem-swap_stat_$D"
sleep 3 ;;
12) echo "inode, file and other kernel tables statistics"
sar -v $INT $CNT |tee /tmp/sar_kern_stat_$D
echo "This file will be saved to /tmp/sar_kern_stat_$D"
sleep 3 ;;
13) echo "system switching activity statistics"
sar -w $INT $CNT |tee /tmp/sar_switch_stat_$D
echo "This file will be saved to /tmp/sar_switch_stat_$D"
sleep 3 ;;
14) echo "swapping statistics"
sar -W $INT $CNT |tee /tmp/sar_swap_stat_$D
echo "This file will be saved to /tmp/sar_swap_stat_$D"
sleep 3 ;;
15) echo "statistics for a given process (Example: PID # 3256)"
echo "Enter the proccess number you want to check:"
read PROC
sar -x $PROC $INT $CNT |tee /tmp/sar_proc_stat_$D
echo "This file will be saved to /tmp/sar_proc_stat_$D"
sleep 3 ;;
0) exit 0;;
*) echo "BAD CHOICE!!! Please select choice 1,2,3,4,5,6 ........ ";
echo "Press Enter to continue. . ." ; read ;;
esac
done

Reply

20 Peter Kerekes March 28, 2014 at 10:11 pm

WOW! COOL SCRIP! THANX! :) :)

Reply

21 Bijohn Vincent September 23, 2011 at 4:51 am

Great !!!!!!!!!!!!! Very useful article….

But there is some issues with LC_ALL=C sar -A > 1234.txt.
The -A option………….. It is throwing java exceptions.

java.lang.NumberFormatException: For input string: “kbhugfree”
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1242)
at java.lang.Float.valueOf(Float.java:405)
at java.lang.Float.(Float.java:506)
at net.atomique.ksar.Linux.Parser.parse(Parser.java:992)
at net.atomique.ksar.kSar.parse(kSar.java:750)
at net.atomique.ksar.FileRead.run(FileRead.java:62)

If I use ,
bijohn@lucid:~$ LC_ALL=C sar -H > 1234.txt
It is not generating java exceptions. But not plotting output.

Other options are Working fine :
bijohn@lucid:~$ LC_ALL=C sar -bqr > 1234.txt

Any suggestion ????

Reply

22 Peter September 13, 2012 at 12:10 pm

Hello,

To avoid java.lang.NumberFormatException set the following :

export LC_ALL=C
export S_TIME_FORMAT=ISO

Before using sar or ksar.

Peter

Reply

23 sid November 21, 2011 at 10:59 pm

One of the biggest flaws with this tool is that we need to run our servers using X session which is not a good idea.

Is there a way to get the reports through web pages using Apache or pull reports off to another server and do the same thing ?

Reply

24 Vishal Singh March 3, 2012 at 5:33 am

Great !!!!!!!!!!!!! Very useful article….
Thanks Team cyberciti

You Guys are great

Vishal

Reply

25 Edray Mashiri April 18, 2012 at 9:06 am

Hi Guys

Is there a way to generate weekly or monthly reports with this tool?

Thanks

Reply

26 Cristian Alejandro August 10, 2012 at 7:23 am

Hi!, great article. You should update the link http://sourceforge.net/projects/ksar/files/latest/download

Reply

27 Chuong August 30, 2012 at 8:18 am

Dear Vivek,

I am very interesting your post. I tried it. But i have 1 question. On the sysstat config file the system stats are kept in 28 days. So, how can i do if i want to see the history system stats (not currently). Such as, the system stats of the first day.

So thanks and Best regards
Chuong

Reply

28 Steve Ritenour October 23, 2012 at 7:56 pm

I too am very interested in doing weekly and/or monthly reports. Is that capability available in ksar? I haven’t been able to find it.

Thanks

Reply

29 Josh Feather November 12, 2012 at 8:13 pm

Nice tool. Runs in batch mode for easy daily/weekly/monthly reporting. Only flaw is that when run in batch mode, it only displays up to 18:00 – using either jpg, png or pdf. Any ideas?

Reply

30 Luis Calvo November 28, 2012 at 2:43 pm

Hi, it is a great article !

It has been very usefull in my job.

Thank’s a lot.

Reply

31 danny December 7, 2012 at 9:39 am

Guys, can somebody help me to extract pdf for CPU and Memory in PDF form, remaining graphs are just junk for me. The following line i used to create a PDF file.
java -jar /kSar.jar -input -outputPDF output.pdf.

Thanks

Reply

32 Manish January 5, 2013 at 6:00 pm

What a good article, i was aware of sar but using different tools like nmon and bonniee++ to create graphs, but ksar is really wonderful and easy to use.

thanks again
manish

Reply

33 harsha February 28, 2013 at 11:11 am

sar -x option never works.
I am currently using SUSE 11 Enterprise Linux

When I run the SAR command using “-x” options it says that option does not exist. I cant find “-x” even in man page for sar.
Can you please help ??

I really need a proper “sar ” command to monitor activity for Given process ( say Apache PID 3618) … please ????

Reply

34 samir parekh May 10, 2013 at 10:51 am

sar -x

Reply

Leave a Comment

Tagged as: , , , , , , , ,

Previous post:

Next post: