Troubleshooting KVM Virtualization Problem With Log Files in Linux

There are various log files stored on the host system to assist with debugging KVM related problems. In this tutorial, I will cover log file locations and tools used to track down KVM problems.

The files are as follows:

1. $HOME/.virtinst/virt-install.log – virt-install tool log file.
2. $HOME/.virt-manager/virt-manager.log – virt-manager tool log file.
3. /var/log/libvirt/qemu/ – Log files for each running virtual machine. If centos is virtual machine name, than log file is /var/log/libvirt/qemu/centos.log.

You can use the grep and other Linux tools to view this files:
# tail -f /var/log/libvirt/qemu/freebsd.log
# grep something $HOME/.virtinst/virt-install.log
$ sudo tail -f /var/log/libvirt/qemu/openbsd.log
Sample outputs:

2016-11-08 17:14:46.909+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.5 (Stefan Bader <stefan.bader@canonical.com> Thu, 06 Oct 2016 13:07:20 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.5), hostname: server1.cyberciti.biz
....
.....
..
=29,id=hostnet1,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:8f:31:5f,bus=pci.0,addr=0x4 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:1 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
char device redirected to /dev/pts/8 (label charserial0)

Connecting To Console

Use the virsh command to connect to guest serial console as follows :
$ virsh list
$ virsh console freebsd
This is useful to troubleshoot problem such as networking and much more from host itself. However, you need to configure guest operating system for a serial console. See how to setup a serial console for FreeBSD and CentOS virtual machines.

KVM Configuration Files

You can edit configuration file to define hardware properties for VMs or other setup. They are located in /etc/libvirt/qemu/ directory. Here is sample FreeBSD guest configuration file.
# less /etc/libvirt/qemu/freebsd.xml
OR
$ virsh dumpxml freebsd
Sample outputs:

When you create guests with the virt-manager or virt-install, the guests configuration files are created automatically in the /etc/libvirt/qemu/ directory. You can use this file error checking. You can edit this file using a text editor or virsh command itself:
virsh edit freebsd
virsh edit centos

KVM tools

Type the following command to install kvm-tools package which contains some diagnostics and debugging tools for KVM, such as kvmtrace and kvm_stat, enter:
# yum -y install kvm_tools
You need to mount debugfs, enter:
# mount -t debugfs debugfs /sys/kernel/debug
# kvm_stat
Sample outputs:

You can use normal tools to troubleshoot and get other information using the following tools:

• ps, pstree, and top
• vmstat, iostat, and lsof
• tcpdump, brctl, ip, and ifconfig

Category	List of Unix and Linux commands
File Management	cat
Firewall	Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04
Network Utilities	dig • host • ip • nmap
OpenVPN	CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04
Package Manager	apk • apt
Processes Management	bg • chroot • cron • disown • fg • jobs • killall • kill • pidof • pstree • pwdx • time
Searching	grep • whereis • which
User Information	groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w
WireGuard VPN	Alpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04