Linux Configure Netconsole To Log Messages Over UDP Network

by on July 2, 2008 · 4 comments· LAST UPDATED July 2, 2008

in , ,

Linux can be configured to log dmesg output to another system via network using syslog. It is done using kernel level networking stuff ia UDP port 514. There is module called netconsole which logs kernel printk messages over udp allowing debugging of problem where disk logging fails and serial consoles are impractical. Most modern distro has this netconsole as a built-in module. netconsole initializes immediately after NIC cards. There are two steps to configure netconsole:

  • Syslogd server - Let us assume 192.168.1.100 IP having FQDN - syslogd.nixcraft.in. Please note that the remote host can run either 'netcat -u -l -p <port>' or syslogd.
  • All other systems running netconsole module in kernel

Step # 1: Configure Centralized syslogd

Login to syslogd.nixcraft.in server. Open syslogd configuration file. Different UNIX / Linux variant have different configuration files

Red Hat / CentOS / Fedora Linux Configuration

If you are using Red Hat / CentOS / Fedora Linux open /etc/sysconfig/syslog file and set SYSLOGD_OPTIONS option for udp logging.
# vi /etc/sysconfig/syslog
Configure syslogd option as follows:
SYSLOGD_OPTIONS="-m 0 -r -x"
Save and close the file. Restart syslogd, enter:
# service syslog restart

Debian / Ubuntu Linux Configuration

If you are using Debian / Ubuntu Linux open file /etc/default/syslogd set SYSLOGD option for udp logging.
# vi /etc/default/syslogd
Configure syslogd option as follows:
SYSLOGD_OPTIONS="-r"
# /etc/init.d/sysklogd restart

FreeBSD configuration

If you are using FreeBSD open /etc/rc.conf and set syslogd_flags option option for udp logging. Please note that FreeBSD by default accepts network connections. Please refer to syslogd man page for more information.

Firewall configuration

You may need to open UDP port 514 to allow network login. Sample iptables rules to open UDP port 514:
MYNET="192.168.1.0/24"
SLSERVER="192.168.1.100"
iptables -A INPUT -p udp -s $MYNET --sport 1024:65535 -d $SLSERVER --dport 514 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p udp -s $SLSERVER --sport 514 -d $MYNET --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT

Step # 2: Configure Linux Netconsole

You need to configure netconsole service. Once this service started, you are allowed a remote syslog daemon to record console output from local system. The local port number that the netconsole module will use 6666 (default). You need to set the IP address of the remote syslog server to send messages.

Open /etc/sysconfig/netconsole file under CentOS / RHEL / Fedora Linux, enter:
# vi /etc/sysconfig/netconsole
Set SYSLOGADDR to 192.168.1.100 (IP address of remote syslog server)
SYSLOGADDR=192.168.0.1
Save and close the file. Restart netconsole service, enter:
# /etc/init.d/netconsole restart

A note about Debian / Ubuntu Linux

Red Hat has netconsole init script. However, under Debian / Ubuntu Linux, you need to manually configure netconsole. Type the following command to start netconsole by loading kernel netconsole module, enter:
# modprobe netconsole 6666@192.168.1.5/eth0,514@192.168.1.100/00:19:D1:2A:BA:A8
Where,

  • 6666 - Local port
  • 192.168.1.5 - Local system IP
  • eth0 - Local system interface
  • 514 - Remote syslogd udp port
  • 192.168.1.100 - Remote syslogd IP
  • 00:19:D1:2A:BA:A8 - Remote syslogd Mac

You can add above modprobe line to /etc/rc.local to load module automatically. Another recommend option is create /etc/modprobe.d/netconsole file and append following text:
# echo 'options netconsole netconsole=6666@192.168.1.5/eth0,514@192.168.1.100/00:19:D1:2A:BA:A8 '> /etc/modprobe.d/netconsole

How do I verify netconsole is logging messages over UDP network?

Login to remote syslog udp server (i.e. 192.168.1.100 our sample syslogd system), enter:
# tail -f /var/log/messages
/var/log/messages is default log file under many distributions to log messages. Refer to /etc/syslog.conf for exact location of your file.

How do I use nc / netcat instead of messing with syslogd?

This is called one minute configuration. You can easily get output on 192.168.1.100 without using syslogd. All you have to do is run netcat (nc) command, on 192.168.1.100:
$ nc -l -p 30000 -u
Login to any other box, enter command:
# modprobe netconsole 6666@192.168.1.5/eth0,30000@192.168.1.100/00:19:D1:2A:BA:A8
Output should start to appear on 192.168.1.100 from 192.168.1.5 without configuring syslogd or anything else.

Further readings:

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 4 comments… read them below or add one }

1 Srinivas G September 1, 2009 at 9:33 am

The above configuration is not working in Fedora Core 10 with 2.6.28 and 2.6.30 kernel versions.

Please let me know the correct procedure for the above kernel versions.

Thanks and Regards,
Srinivas G

Reply

2 Jeff Goldschrafe October 29, 2010 at 12:09 pm

One small note: In RHEL/CentOS 4, you’re looking for /etc/sysconfig/netdump (and /etc/init.d/netdump) rather than netconsole.

Reply

3 Dr. Edward Morbius September 16, 2011 at 7:08 pm

It’s also helpful to trigger a kernel message to see if netconsole is logging properly.

One way to do this is to enable sysrq (‘echo 1 > /proc/sys/kernel/sysrq’, or configure via /etc/sysctl.conf), then trigger sysrq’s help output: ‘echo h > /proc/sysrq-trigger’.

This should log sysrq’s help output on your configured logging host, and is unlikely to affect system stability.

Reply

4 George R. Goffe July 18, 2012 at 4:35 am

Am I seeing things or is all the text in this URL truncated on the right? I’m using Firefox Nightly 16.0a1. I think it’s a Nightly problem. Opera-next works find. :(

Reply

Leave a Comment

Tagged as: , , , , , , , , , , , , , , ,

Previous post:

Next post: