Linux Configure Netconsole To Log Messages Over UDP Network

Posted on in Categories CentOS, Debian Linux, File system, FreeBSD, Gentoo Linux, GNU/Open source, Hardware, kernel, Linux, Linux distribution, Linux Log Management, Security, Sys admin, Tips, Troubleshooting, Ubuntu Linux last updated July 2, 2008

Linux can be configured to log dmesg output to another system via network using syslog. It is done using kernel level networking stuff ia UDP port 514. There is module called netconsole which logs kernel printk messages over udp allowing debugging of problem where disk logging fails and serial consoles are impractical. Most modern distro has this netconsole as a built-in module. netconsole initializes immediately after NIC cards. There are two steps to configure netconsole:

  • Syslogd server – Let us assume IP having FQDN – Please note that the remote host can run either ‘netcat -u -l -p <port>’ or syslogd.
  • All other systems running netconsole module in kernel

Step # 1: Configure Centralized syslogd

Login to server. Open syslogd configuration file. Different UNIX / Linux variant have different configuration files

Red Hat / CentOS / Fedora Linux Configuration

If you are using Red Hat / CentOS / Fedora Linux open /etc/sysconfig/syslog file and set SYSLOGD_OPTIONS option for udp logging.
# vi /etc/sysconfig/syslog
Configure syslogd option as follows:
SYSLOGD_OPTIONS="-m 0 -r -x"
Save and close the file. Restart syslogd, enter:
# service syslog restart

Debian / Ubuntu Linux Configuration

If you are using Debian / Ubuntu Linux open file /etc/default/syslogd set SYSLOGD option for udp logging.
# vi /etc/default/syslogd
Configure syslogd option as follows:
# /etc/init.d/sysklogd restart

FreeBSD configuration

If you are using FreeBSD open /etc/rc.conf and set syslogd_flags option option for udp logging. Please note that FreeBSD by default accepts network connections. Please refer to syslogd man page for more information.

Firewall configuration

You may need to open UDP port 514 to allow network login. Sample iptables rules to open UDP port 514:
iptables -A INPUT -p udp -s $MYNET --sport 1024:65535 -d $SLSERVER --dport 514 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p udp -s $SLSERVER --sport 514 -d $MYNET --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT

Step # 2: Configure Linux Netconsole

You need to configure netconsole service. Once this service started, you are allowed a remote syslog daemon to record console output from local system. The local port number that the netconsole module will use 6666 (default). You need to set the IP address of the remote syslog server to send messages.

Open /etc/sysconfig/netconsole file under CentOS / RHEL / Fedora Linux, enter:
# vi /etc/sysconfig/netconsole
Set SYSLOGADDR to (IP address of remote syslog server)
Save and close the file. Restart netconsole service, enter:
# /etc/init.d/netconsole restart

A note about Debian / Ubuntu Linux

Red Hat has netconsole init script. However, under Debian / Ubuntu Linux, you need to manually configure netconsole. Type the following command to start netconsole by loading kernel netconsole module, enter:
# modprobe netconsole [email protected]/eth0,[email protected]/00:19:D1:2A:BA:A8

  • 6666 – Local port
  • – Local system IP
  • eth0 – Local system interface
  • 514 – Remote syslogd udp port
  • – Remote syslogd IP
  • 00:19:D1:2A:BA:A8 – Remote syslogd Mac

You can add above modprobe line to /etc/rc.local to load module automatically. Another recommend option is create /etc/modprobe.d/netconsole file and append following text:
# echo 'options netconsole [email protected]/eth0,[email protected]/00:19:D1:2A:BA:A8 '> /etc/modprobe.d/netconsole

How do I verify netconsole is logging messages over UDP network?

Login to remote syslog udp server (i.e. our sample syslogd system), enter:
# tail -f /var/log/messages
/var/log/messages is default log file under many distributions to log messages. Refer to /etc/syslog.conf for exact location of your file.

How do I use nc / netcat instead of messing with syslogd?

This is called one minute configuration. You can easily get output on without using syslogd. All you have to do is run netcat (nc) command, on
$ nc -l -p 30000 -u
Login to any other box, enter command:
# modprobe netconsole [email protected]/eth0,[email protected]/00:19:D1:2A:BA:A8
Output should start to appear on from without configuring syslogd or anything else.

Further readings:

6 comment

  1. The above configuration is not working in Fedora Core 10 with 2.6.28 and 2.6.30 kernel versions.

    Please let me know the correct procedure for the above kernel versions.

    Thanks and Regards,
    Srinivas G

  2. It’s also helpful to trigger a kernel message to see if netconsole is logging properly.

    One way to do this is to enable sysrq (‘echo 1 > /proc/sys/kernel/sysrq’, or configure via /etc/sysctl.conf), then trigger sysrq’s help output: ‘echo h > /proc/sysrq-trigger’.

    This should log sysrq’s help output on your configured logging host, and is unlikely to affect system stability.

  3. Am I seeing things or is all the text in this URL truncated on the right? I’m using Firefox Nightly 16.0a1. I think it’s a Nightly problem. Opera-next works find. :(

  4. A coupe of things that I ran into getting this to work on a debian wheezy box with bridging over a bond.

    1st, netconsole does not work on a bridge interface (bridge interfaces don’t support polling). You must specify the underlying bond or eth device for the bridge.

    2nd, You must use a different ip in netconsole than the ip assigned via bridge for the host.

  5. Thanks a lot for this page!
    At least using rsyslog as syslog daemon, on port 514, logging does not work on openSUSE (tried 13.1 and some others). However, “netcat -u -l 6666 | tee -a remote_log” works perfect. Just remember to open that port in SuSEfirewall2. I don’t know the reason, maybe rsyslog is not that compatible.

    I don’t know (yet) if this procedure will work on boot, but at least the “etc/modprobe.d/netconsole.conf” file is copied to the initrd file, and the module is added if listed in “/etc/sysconfig/kernel” (INITRD_MODULES variable).

Leave a Comment