HowTo: Create sar Graphs With kSar [ Identifying Linux Bottlenecks ]

Posted on in Categories Debian Linux, fedora linux, File system, Hardware, High performance computing, Howto, Linux, Linux Scalability, Monitoring, Networking last updated December 15, 2009

The sar command collect, report, or save UNIX / Linux system activity information. It will save selected counters in the operating system to the /var/log/sa/sadd file. From the collected data, you get lots of information about your server:

  1. CPU utilization
  2. Memory paging and its utilization
  3. Network I/O, and transfer statistics
  4. Process creation activity
  5. All block devices activity
  6. Interrupts/sec etc.

sar output can be used for identifying server bottlenecks. However, analyzing information provided by sar can be difficult, so use kSar, which can take sar output and plot a nice easy to understand graph over period of time.

sysstat Package

The sar, sa1, and sa2 commands are part of sysstat package:

  1. sar : Displays the data.
  2. sa1 and sa2: Collect and store the data for later analysis. The sa2 shell script write a daily report in the /var/log/sa directory. The sa1 shell script collect and store binary data in the system activity daily data file.
  3. sadc – System activity data collector. You can configure various options by modifying sa1 and sa2 scripts. They are located at the following location:
    • /usr/lib64/sa/sa1 (64bit) or /usr/lib/sa/sa1 (32bit) – This calls sadc to log reports to/var/log/sa/sadX format.
    • /usr/lib64/sa/sa2 (64bit) or /usr/lib/sa/sa2 (32bit) – This calls sar to log reports to /var/log/sa/sarX format.

How Do I Install sar?

Type the following command:
# yum install sysstat
Sample outputs:

Loaded plugins: downloadonly, fastestmirror, priorities,
              : protectbase, security
Loading mirror speeds from cached hostfile
 * addons: mirror.cs.vt.edu
 * base: mirror.ash.fastserv.com
 * epel: serverbeach1.fedoraproject.org
 * extras: mirror.cogentco.com
 * updates: centos.mirror.nac.net
0 packages excluded due to repository protections
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package sysstat.x86_64 0:7.0.2-3.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================
 Package        Arch          Version             Repository   Size
====================================================================
Installing:
 sysstat        x86_64        7.0.2-3.el5         base        173 k

Transaction Summary
====================================================================
Install      1 Package(s)
Update       0 Package(s)
Remove       0 Package(s)

Total download size: 173 k
Is this ok [y/N]: y
Downloading Packages:
sysstat-7.0.2-3.el5.x86_64.rpm               | 173 kB     00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : sysstat                                      1/1

Installed:
  sysstat.x86_64 0:7.0.2-3.el5

Complete!

Configuration Files

Edit /etc/sysconfig/sysstat file specify how long to keep log files in days, maximum is a month:
# vi /etc/sysconfig/sysstat
Sample outputs:

# keep log for 28 days
# the default is 7
HISTORY=28

Save and close the file.

The Default Cron Job

The default cron job is located at /etc/cron.d/sysstat:
# cat /etc/cron.d/sysstat
Sample outputs:

# run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A

Tell sadc To Report Statistics For Disks

Edit /etc/cron.d/sysstat, enter:
# vi /etc/cron.d/sysstat
Update it as follows to log all disk stats (the -d option force to log stats for each block device and the -I option force report statistics for all system interrupts)
# run system activity accounting tool every 10 minutes

*/10 * * * * root /usr/lib64/sa/sa1 -I -d 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A

Save and close the file. Turn on the service, enter:
# chkconfig sysstat on
# service sysstat start
Sample outputs:

Calling the system activity data collector (sadc):

How Do I Use sar? How do I View Stats?

Use the sar command to display output the contents of selected cumulative activity counters in the operating system. In this example, sar is run to get real-time reporting from the command line about CPU utilization:
# sar -u 3 10
Sample outputs:

Linux 2.6.18-164.2.1.el5 (www-03.nixcraft.in) 	12/14/2009

09:49:47 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:49:50 PM       all      5.66      0.00      1.22      0.04      0.00     93.08
09:49:53 PM       all     12.29      0.00      1.93      0.04      0.00     85.74
09:49:56 PM       all      9.30      0.00      1.61      0.00      0.00     89.10
09:49:59 PM       all     10.86      0.00      1.51      0.04      0.00     87.58
09:50:02 PM       all     14.21      0.00      3.27      0.04      0.00     82.47
09:50:05 PM       all     13.98      0.00      4.04      0.04      0.00     81.93
09:50:08 PM       all      6.60      6.89      1.26      0.00      0.00     85.25
09:50:11 PM       all      7.25      0.00      1.55      0.04      0.00     91.15
09:50:14 PM       all      6.61      0.00      1.09      0.00      0.00     92.31
09:50:17 PM       all      5.71      0.00      0.96      0.00      0.00     93.33
Average:          all      9.24      0.69      1.84      0.03      0.00     88.20

Where,

  • 3 = interval
  • 10 = count

To view process creation statistics, enter:
# sar -c 3 10
To view I/O and transfer rate statistics, enter:
# sar -b 3 10
To view paging statistics, enter:
# sar -B 3 10
To view block device statistics, enter:
# sar -d 3 10
To view statistics for all interrupt statistics, enter:
# sar -I XALL 3 10
To view device specific network statistics, enter:
# sar -n DEV 3 10
# sar -n EDEV 3 10
To view CPU specific statistics, enter:
# sar -P ALL
# Only 1st CPU stats
# sar -P 1 3 10
To view queue length and load averages statistics, enter:
# sar -q 3 10
To view memory and swap space utilization statistics, enter:
# sar -r 3 10
# sar -R 3 10
To view status of inode, file and other kernel tables statistics, enter:
# sar -v 3 10
To view system switching activity statistics, enter:
# sar -w 3 10
To view swapping statistics, enter:
# sar -W 3 10
To view statistics for a given process called Apache with PID # 3256, enter:
# sar -x 3256 3 10

Say Hello To kSar

sar and sadf provides CLI based output. The output may confuse all new users / sys admin. So you need to use kSar which is a java application that graph your sar data. It also permit to export data to PDF/JPG/PNG/CSV. You can load data from three method : local file, local command execution, and remote command execution via SSH. kSar supports the sar output of the following OS:

  1. Solaris 8, 9 and 10
  2. Mac OS/X 10.4+
  3. Linux (Systat Version >= 5.0.5)
  4. AIX (4.3 & 5.3)
  5. HPUX 11.00+

Download And Install kSar

Visit the official website and grab the latest source code. Use wget to download the source code, enter:
$ wget http://ksar.atomique.net/kSar/kSar-5.0.6.zip
Use unzip command to extract files, enter:
$ unzip kSar-5.0.6.zip

How Do I Run kSar?

Make sure JAVA jdk is installed and working correctly. Type the following command to start kSar, run:
$ cd kSar-5.0.6/
$ sh run.sh

Fig.01: kSar welcome screen
Fig.01: kSar welcome screen

Next you will see main kSar window, and menus with two panels.
Fig.02: kSar - the main window
Fig.02: kSar - the main window

The left one will have a list of graphs available depending on the data kSar has parsed. The right window will show you the graph you have selected.

How Do I Generate sar Graphs Using kSar?

First, you need to grab sar command statistics. Type the following command to get stats, enter (type it on your server):
[server1 ]# LC_ALL=C sar -A > /tmp/sar.data.txt
# copy file to local desktop
[desktop ]$ scp [email protected]:/tmp/sar.data.txt /tmp

Click on Data > Load data from text file > Select sar.data.txt > Open button.

Now, the graph type tree is deployed in left pane and a graph has been selected:

Fig.03: Processes for server1
Fig.03: Processes for server1
Fig.03: Disk stats (blok device) stats for server1
Fig.04: Disk stats (blok device) stats for server1
Fig.05: Memory stats for server1
Fig.05: Memory stats for server1

Zoom in and out

Using the move, you can interactively zoom onto up a part of a graph. To select a zone to zoom, click on the upper left conner and while still holding the mouse but on move to the lower-right of the zone you want to zoom. To come back to unzoomed view click and drag the mouse to any corner location except a lower-right one. You can also right click and select zoom options

Understanding kSar Graphs And sar Data

I strongly recommend reading sar and sadf command man page:
$ man sar
$ man sadf
kSar graphs can be understood from the sar command man page or by reading its official documentation.

Case Study: Identifying Linux Server CPU Bottlenecks

With sar and kSar you get detailed snapshot of memory, CPU and other subsystems. For example, if CPU utilization is more than 80% for a long period of time, a CPU bottleneck is most likely occurring. Using sar -x ALL you can find out CPU eating process. The output of mpstat command (part of sysstat package itself) will also help you understand the cpu utilization. You can easily analyzing this information with kSar.

I Found CPU Bottlenecks…

Performance tuning options for the CPU are as follows:

  1. Make sure that no unnecessary programs are running in background. Turn off unnecessary services.
  2. Use cron to schedule jobs (e.g., backup) to run at off-peak hours.
  3. Use top and ps command to find out all non-critical background jobs / services. Make sure you lower their priority using renice command.
  4. Use taskset command to set a processes’s CPU affinity (offload cpu) i.e. bind processes to different CPUs. For example, run MySQL database on cpu #2 and Apache on cpu # 3.
  5. Make sure you are using latest drivers and firmware for your server.
  6. If possible add additional SMP cpus to system.
  7. Use faster CPUs for a single-threaded application (e.g. Lighttpd web server app).
  8. Use more CPUs for a multi-threaded application (e.g. MySQL database server app).
  9. Use more computer nodes and setup a load balancer for web app.

isag – Interactive System Activity Grapher (alternate tool)

The isag command graphically displays the system activity data stored in a binary data file by a previous sar run. The isag command invokes sar to extract the data to be plotted. isag has limited set of options as compare to kSar.

Fig.06: isag CPU utilization graphs
Fig.06: isag CPU utilization graphs

References:

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin and a trainer for the Linux operating system/Unix shell scripting. He has worked with global clients and in various industries, including IT, education, defense and space research, and the nonprofit sector. Follow him on Twitter, Facebook, Google+.

Share this on:

39 comment

  2. @Bhavya,

    They are loading fine here. I do not see any problem.

    wget http://files.cyberciti.biz/uploads/tips/2009/12/memory-ksar.png
    O/p

    --2009-12-15 16:00:35--  http://files.cyberciti.biz/uploads/tips/2009/12/memory-ksar.png
Resolving files.cyberciti.biz... 203.190.126.132
Connecting to files.cyberciti.biz|203.190.126.132|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 100548 (98K) [image/png]
Saving to: `memory-ksar.png'

100%[====================================================================================================================>] 1,00,548    68.5K/s   in 1.4s

2009-12-15 16:00:37 (68.5 KB/s) - `memory-ksar.png' saved [100548/100548]
    Reply Report comment

  12. Great info !!!!!

    Really help me lots to get a portable output. Earlier I was used cacti to do this. It’s a online tool and required huge SNMP traffic to analyze. But it’s simple, easy to configure but rich in output.

    Thanks,
    Liju

    Reply Report comment

  16. What if you did not generate SAR file using the locale LC_ALL=C prefix?

    I have a sar output in text (without using the locale prefix), but kSar doesn’t parse it. It only knows first timestamp and that’s it.

    Any options? Can I tweak the file to see the output?

    Reply Report comment

  17. Why when I did “sar -A 5 0” yesterday did it work fine

    today after a nw install on a new machine it always opens the file since midnight, and does not update every 5 seconds. Infact nothing I do can make it update “Live” it always opens from midnight to now????

    Reply Report comment

  19. Hi, I took the liberty to create a script to get the sar statistics, please feel free to modify it if you like it

    Thank you.
    Jacobo.

    #!/bin/bash
#
# Script to get sar statistics#
## sar_stats.sh ##
echo "Enter the number of intervals (seconds) for the sar statistics:"
read int
echo "enter the number of output lines (count) to process:"
read cnt
while :
do
clear
echo "*******************************"
echo "* Choose from the following: *"
echo "*******************************"
echo "* [1] To view process creation statistics *"
echo "* [2] To view I/O and transfer rate statistics *"
echo "* [3] To view paging statistics *"
echo "* [4] To view block device statistics *"
echo "* [5] To view statistics for all interrupt statistics *"
echo "* [6] To view device specific network statistics *"
echo "* [7] To view extended network statistics *"
echo "* [8] To view CPU specific statistics *"
echo "* [9] To view Only 1st CPU stats *"
echo "* [10] To view queue length and load averages statistics *"
echo "* [11] To view memory and swap space utilization statistics *"
echo "* [12] To view inode, file and other kernel tables statistics*"
echo "* [13] To view system switching activity statistics *"
echo "* [14] To view swapping statistics *"
echo "* [15] To view statistics for a given process ( Example: a process called Apache with PID # 3256) *"
echo "* [0] Exit *"
echo "************************"
echo -n "Enter your menu choice [0-15]: "
D=`/bin/date '+%B.%d'`
INT=$int
CNT=$cnt
read mychoice
case $mychoice in
1) echo "process creation statistics";
sar -c $INT $CNT |tee /tmp/sar_proc_stat_$D
echo "This file will be saved to /tmp/sar_pc_stat_$D"
sleep 3 ;;
2) echo "I/O and transfer rate statistics"
sar -b $INT $CNT |tee /tmp/sar_IO_TR_stat_$D
echo "This file will be saved to /tmp/sar_IO_TR_stat_$D"
sleep 3 ;;
3) echo "paging statistics"
sar -B $INT $CNT |tee /tmp/sar_pag_stat_$D
echo "This file will be saved to /tmp/sar_pag_stat_$D"
sleep 3 ;;
4) echo "block device statistics"
sar -d $INT $CNT |tee /tmp/sar_block_stat_$D
echo "This file will be saved to /tmp/sar_block_stat_$D"
sleep 3 ;;
5) echo "all interrupt statistics statistics"
sar -I XALL $INT $CNT |tee /tmp/sar_Intr_stat_$D
echo "This file will be saved to /tmp/sar_Intr_stat_$D"
sleep 3 ;;
6) echo "specific network statistics"
sar -n DEV $INT $CNT |tee /tmp/sar_Net_stat_$D
echo "This file will be saved to /tmp/sar_Net_stat_$D"
sleep 3 ;;
7) echo "Extended network statistics"
sar -n EDEV $INT $CNT |tee /tmp/sar_ENet_stat_$D
echo "This file will be saved to /tmp/sar_ENet_stat_$D"
sleep 3 ;;
8) echo "CPU specific statistics"
sar -P ALL $INT $CNT |tee /tmp/sar_CPU_stat_$D
echo "This file will be saved to /tmp/sar_CPU_stat_$D"
sleep 3 ;;
9) echo "1st CPU statistics"
sar -P 1 $INT $CNT |tee /tmp/sar_1CPU_stat_$D
echo "This file will be saved to /tmp/sar_1CPU_stat_$D"
sleep 3 ;;
10) echo "queue length and load averages statistics"
sar -q $INT $CNT |tee /tmp/sar_queue_stat_$D
echo "This file will be saved to /tmp/sar_queue_stat_$D"
sleep 3 ;;
11) echo "memory and swap space utilization statistics"
sar -r $INT $CNT |tee /tmp/sar_mem-swap_stat_$D
echo "This file will be saved to /tmp/sar_mem-swap_stat_$D"
sleep 3 ;;
12) echo "inode, file and other kernel tables statistics"
sar -v $INT $CNT |tee /tmp/sar_kern_stat_$D
echo "This file will be saved to /tmp/sar_kern_stat_$D"
sleep 3 ;;
13) echo "system switching activity statistics"
sar -w $INT $CNT |tee /tmp/sar_switch_stat_$D
echo "This file will be saved to /tmp/sar_switch_stat_$D"
sleep 3 ;;
14) echo "swapping statistics"
sar -W $INT $CNT |tee /tmp/sar_swap_stat_$D
echo "This file will be saved to /tmp/sar_swap_stat_$D"
sleep 3 ;;
15) echo "statistics for a given process (Example: PID # 3256)"
echo "Enter the proccess number you want to check:"
read PROC
sar -x $PROC $INT $CNT |tee /tmp/sar_proc_stat_$D
echo "This file will be saved to /tmp/sar_proc_stat_$D"
sleep 3 ;;
0) exit 0;;
*) echo "BAD CHOICE!!! Please select choice 1,2,3,4,5,6 ........ ";
echo "Press Enter to continue. . ." ; read ;;
esac
done
    Reply Report comment

  20. Great !!!!!!!!!!!!! Very useful article….

    But there is some issues with LC_ALL=C sar -A > 1234.txt.
    The -A option………….. It is throwing java exceptions.

    java.lang.NumberFormatException: For input string: “kbhugfree”
    at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1242)
    at java.lang.Float.valueOf(Float.java:405)
    at java.lang.Float.(Float.java:506)
    at net.atomique.ksar.Linux.Parser.parse(Parser.java:992)
    at net.atomique.ksar.kSar.parse(kSar.java:750)
    at net.atomique.ksar.FileRead.run(FileRead.java:62)

    If I use ,
    [email protected]:~$ LC_ALL=C sar -H > 1234.txt
    It is not generating java exceptions. But not plotting output.

    Other options are Working fine :
    [email protected]:~$ LC_ALL=C sar -bqr > 1234.txt

    Any suggestion ????

    Reply Report comment

  21. One of the biggest flaws with this tool is that we need to run our servers using X session which is not a good idea.

    Is there a way to get the reports through web pages using Apache or pull reports off to another server and do the same thing ?

    Reply Report comment

  25. Dear Vivek,

    I am very interesting your post. I tried it. But i have 1 question. On the sysstat config file the system stats are kept in 28 days. So, how can i do if i want to see the history system stats (not currently). Such as, the system stats of the first day.

    So thanks and Best regards
    Chuong

    Reply Report comment

  27. Nice tool. Runs in batch mode for easy daily/weekly/monthly reporting. Only flaw is that when run in batch mode, it only displays up to 18:00 – using either jpg, png or pdf. Any ideas?

    Reply Report comment

  29. Guys, can somebody help me to extract pdf for CPU and Memory in PDF form, remaining graphs are just junk for me. The following line i used to create a PDF file.
    java -jar /kSar.jar -input -outputPDF output.pdf.

    Thanks

    Reply Report comment

    1. Hi Danny,

      I use ksar version 5.0.6, where you’ll get an option to export the report to PDF, JPG, PNG, CSV, & XLS format. Once you select PDF option and the location to save it, it’ll give you choice to import all data or selected options (tick the check box you want i.e. CPU & memory etc).

      Reply Report comment

  31. sar -x option never works.
    I am currently using SUSE 11 Enterprise Linux

    When I run the SAR command using “-x” options it says that option does not exist. I cant find “-x” even in man page for sar.
    Can you please help ??

    I really need a proper “sar ” command to monitor activity for Given process ( say Apache PID 3618) … please ????

    Reply Report comment

  32. In the crontab entry, you should not be limiting the interval to 1 second. Sar uses the same system resources no matter how long the interval is. It reads kernel values, sleeps, reads the values again and records/prints the difference value. 1 second, 10 seconds, 1200 seconds are the same as far as sar’s resource usage. 99.99% of sar’s usage is sleep, which is what the kernel does anyway when it’s not doing anything. Note below that the first sar sample of only a second showed an average cpu of 3%. The longer samples, averaging over a longer period, show that 6% is probably more of an accurate average, at this time. The web pages I’ve seen so far feed each other with this 1 second sample thing, almost like someone is afraid sar might bog the system down. It won’t. The same two sets of kernel reads happens no matter what the interval is:
    time sar 1 1; time sar 10 1; time sar 100 1
    Linux 2.6.18-194.el5 (blahblah) 10/07/14
    12:04:51 CPU %user %nice %system %iowait %steal %idle
    12:04:52 all 3.00 0.00 0.75 0.00 0.00 96.25
    Average: all 3.00 0.00 0.75 0.00 0.00 96.25
    sar 1 1 0.00s user 0.00s system 0% cpu 1.005 total
    Linux 2.6.18-194.el5 (blahblah) 10/07/14
    12:04:52 CPU %user %nice %system %iowait %steal %idle
    12:05:02 all 6.21 0.00 0.93 0.20 0.00 92.67
    Average: all 6.21 0.00 0.93 0.20 0.00 92.67
    sar 10 1 0.00s user 0.00s system 0% cpu 10.005 total
    Linux 2.6.18-194.el5 (blahblah) 10/07/14
    12:05:02 CPU %user %nice %system %iowait %steal %idle
    12:06:42 all 6.32 0.00 0.97 0.24 0.00 92.47
    Average: all 6.32 0.00 0.97 0.24 0.00 92.47
    sar 100 1 0.00s user 0.00s system 0% cpu 1:40.01 total
    From the man page example it shows each hour having 3 20 minute samples. This provides accurate averaging and small sa## files. A 1 second interval each 10 minutes is 1/600th of the information available.
    EXAMPLES
    To create a daily record of sar activities, place the following entry
    in your root or adm crontab file:
    0 8-18 * * 1-5 /usr/lib/sa/sa1 1200 3 &

    Reply Report comment

  34. Hi,
    Its a nice tool. I have few questions
    1. How to generate the monthly or weekly report ? i mean in a single chart i should a get a monthly report ?
    2. -addHTML option is not working. I am not able to generate any HTML report.

    Anyone help me

    Thanks

    Reply Report comment

Leave a Comment

Tagged as: Tags , , , , , , , ,