How to show dropped packets per interface on Linux

How do I display dropped packets per interface on Linux operating systems from the command line option? How can I determine why a Linux server is dropping packets?

We can use the ip command or netstat command or ethtool command to show dropped packets statistics per network interface on Linux. Let us see how to use both commands list dropped packets per interface.
Tutorial details
Difficulty Advanced (rss)
Root privileges Yes
Requirements Linux with GCC compilers
Time Depends upon your skills and situation

ADVERTISEMENTS

Displaying show dropped packets per interface on Linux using the netstat

The netstat command is mostly obsolete. Replacement for netstat is ss and ip command. However, netstat still available on older Linux distros, which are in productions. Hence, I will start with netstat but if possible, use the ip/ss tools. The syntax is:
netstat -i
netstat --interfaces

netstat show dropped packets per interface on Linux

Displaying network stats per network interface on LInux

To display summary statistics for each protocol, run:

netstat -s
netstat --statistics

Outputs:

Ip:
    Forwarding: 1
    101759568 total packets received
    70289211 forwarded
    0 incoming packets discarded
    31287093 incoming packets delivered
    136164545 requests sent out
    22 outgoing packets dropped
    220 reassemblies required
    110 packets reassembled ok
    2364 fragments received ok
    3345 fragments failed
    4728 fragments created
Icmp:
    295517 ICMP messages received
    6 input ICMP message failed
    ICMP input histogram:
        destination unreachable: 145
        timeout in transit: 187
        echo requests: 289750
        echo replies: 5435
    298725 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 3408
        echo requests: 5567
        echo replies: 289750
IcmpMsg:
        InType0: 5435
        InType3: 145
        InType8: 289750
        InType11: 187
        OutType0: 289750
        OutType3: 3408
        OutType8: 5567
Tcp:
    19006 active connection openings
    14619 passive connection openings
    2268 failed connection attempts
    393 connection resets received
    1 connections established
    2215735 segments received
    2511500 segments sent out
    6067 segments retransmitted
    182 bad segments received
    13173 resets sent
Udp:
    28543977 packets received
    63 packets to unknown port received
    287687 packet receive errors
    22106848 packets sent
    287687 receive buffer errors
    0 send buffer errors
UdpLite:
TcpExt:
    10 invalid SYN cookies received
    2264 resets received for embryonic SYN_RECV sockets
    42 packets pruned from receive queue because of socket buffer overrun
    14095 TCP sockets finished time wait in fast timer
    21 packetes rejected in established connections because of timestamp
    16908 delayed acks sent
    13 delayed acks further delayed because of locked socket
    Quick ack mode was activated 4346 times
    756194 packet headers predicted
    441344 acknowledgments not containing data payload received
    618096 predicted acknowledgments
    TCPSackRecovery: 87
    Detected reordering 418 times using SACK
    TCPDSACKUndo: 1
    14 congestion windows recovered without slow start after partial ack
    TCPLostRetransmit: 3994
    TCPSackFailures: 1
    121 fast retransmits
    8 retransmits in slow start
    TCPTimeouts: 5158
    TCPLossProbes: 789
    TCPLossProbeRecovery: 66
    TCPSackRecoveryFail: 3
    TCPBacklogCoalesce: 8617
    TCPDSACKOldSent: 4359
    TCPDSACKOfoSent: 1
    TCPDSACKRecv: 127
    3870 connections reset due to unexpected data
    244 connections reset due to early user close
    487 connections aborted due to timeout
    TCPDSACKIgnoredNoUndo: 33
    TCPSackShifted: 37
    TCPSackMerged: 115
    TCPSackShiftFallback: 731
    TCPRcvCoalesce: 225465
    TCPOFOQueue: 29252
    TCPOFOMerge: 1
    TCPChallengeACK: 193
    TCPSYNChallenge: 186
    TCPAutoCorking: 26574
    TCPFromZeroWindowAdv: 8
    TCPToZeroWindowAdv: 8
    TCPWantZeroWindowAdv: 37
    TCPSynRetrans: 647
    TCPOrigDataSent: 1526711
    TCPACKSkippedSynRecv: 153
    TCPKeepAlive: 53
    TCPDelivered: 1539034
    TCPAckCompressed: 2559
IpExt:
    InNoRoutes: 16
    InBcastPkts: 4
    InOctets: 92596603587
    OutOctets: 263001759492
    InBcastOctets: 310
    InNoECTPkts: 121775194
    InECT1Pkts: 1
    InECT0Pkts: 51506
    InCEPkts: 25

Show tcp stats

netstat --statistics --tcp
netstat -s -t

Display udp stats

netstat --statistics --udp
netstat -s -u

Showing dropped packets statistics per network interface on Linux using the ip

Let us see how to see link device stats using the ip command. The syntax is:
ip -s link
ip -s link show {interface}
ip -s link show eth0

Displaying network link stats on Linux using ip
In this example display link stats for wg0:
ip -s link show wg0

4: wg0:  mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none 
    RX: bytes  packets  errors  dropped overrun mcast   
    1889086196 11451163 8413    62869   0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    56342032204 41609374 0       5685    0       0       

It is clear that TX is Transmit and RX is Receive. Wireguard creates the wg0 interface. So either Wireguard or firewall dropping packets as per policy.

Queries the specified network device for NIC- and driver-specific statistics with ethtool

Pass the -S or --statistics option to display stats. Again the syntax is straightforward:
ethtool -S {device}
ethtool -S eth0

NIC statistics:
     rx_queue_0_packets: 94804582
     rx_queue_0_bytes: 92123064799
     rx_queue_0_drops: 0
     rx_queue_0_xdp_packets: 0
     rx_queue_0_xdp_tx: 0
     rx_queue_0_xdp_redirects: 0
     rx_queue_0_xdp_drops: 0
     rx_queue_0_kicks: 1499
     tx_queue_0_packets: 94616365
     tx_queue_0_bytes: 93565559918
     tx_queue_0_xdp_tx: 0
     tx_queue_0_xdp_tx_drops: 0
     tx_queue_0_kicks: 40246533

Another option is to directly query the /proc/net/dev file either using the cat command or column command:
cat /proc/net/dev
column -t /proc/net/dev

And this is what we see:

Inter-|  Receive      |         Transmit
face     |bytes       packets   errs      drop   fifo  frame  compressed  multicast|bytes  packets      errs      drop  fifo    colls  carrier  compressed
eth0:    92123116754  94805122  0         0      0     0      0           0                93565689124  94617058  0     0       0      0        0           0
wg0:     1889086196   11451163  8413      62869  0     8413   0           0                56342032204  41609374  0     5685    0      0        0           0
lo:      52141452     150908    0         0      0     0      0           0                52141452     150908    0     0       0      0        0           0
tun0:    1650631998   16914416  0         0      0     0      0           0                30143956312  22000354  0     660246  0      0        0           0

Finding out why a Linux server is dropping packets

We need to use dropwatch:

It is a project I started in an effort to improve the ability for developers and system administrator to diagnose problems in the Linux Networking stack, specifically in our ability to diagnose where packets are getting dropped.

Building dropwatch

Install required tools, libs and gcc compiler collection on Ubuntu or Debian Linux:
sudo apt-get install libpcap-dev libnl-3-dev libnl-genl-3-dev binutils-dev libreadline6-dev autoconf libtool pkg-config build-essential
Next, clone the repo and then compile it:
git clone https://github.com/nhorman/dropwatch
cd dropwatch
./autogen.sh
./configure
make
make install

Session:

Making install in src
make[1]: Entering directory '/tmp/dropwatch/src'
make[2]: Entering directory '/tmp/dropwatch/src'
 /usr/bin/mkdir -p '/usr/local/bin'
  /bin/bash ../libtool   --mode=install /usr/bin/install -c dropwatch dwdump '/usr/local/bin'
libtool: install: /usr/bin/install -c dropwatch /usr/local/bin/dropwatch
libtool: install: /usr/bin/install -c dwdump /usr/local/bin/dwdump
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/tmp/dropwatch/src'
make[1]: Leaving directory '/tmp/dropwatch/src'
Making install in doc
make[1]: Entering directory '/tmp/dropwatch/doc'
make[2]: Entering directory '/tmp/dropwatch/doc'
make[2]: Nothing to be done for 'install-exec-am'.
 /usr/bin/mkdir -p '/usr/local/share/man/man1'
 /usr/bin/install -c -m 644 dropwatch.1 '/usr/local/share/man/man1'
make[2]: Leaving directory '/tmp/dropwatch/doc'
make[1]: Leaving directory '/tmp/dropwatch/doc'
Making install in tests
make[1]: Entering directory '/tmp/dropwatch/tests'
make[2]: Entering directory '/tmp/dropwatch/tests'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/tmp/dropwatch/tests'
make[1]: Leaving directory '/tmp/dropwatch/tests'
make[1]: Entering directory '/tmp/dropwatch'
make[2]: Entering directory '/tmp/dropwatch'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/tmp/dropwatch'
make[1]: Leaving directory '/tmp/dropwatch'

Run it as follows:
# dropwatch -l kas
Using netstat and dropwatch to observe packet loss on Linux servers
You see nf_hook_slow or icmpv6_rcv are Linux kernel functions. It means I need to search Linux kernel trees and see what is going on. Naturally, you must understand C programming and have a good understanding of Linux kernel internals including TCP/IP stack. Now, you know how to see if packets are dropped at the IP layer, the link layer, the UDP/TCP layer, or the application layer. If packets are dropped in TCP/IP, we need to look into the Linux kernel source code documentation. See man page and source code of dropwatch for futher information:
man dropwatch
I would also suggest trying out tcpdump to dump traffic on a network interface. Often it gives hints about packets and easily analyzed in wireshark tool:
man tcpdump

Conclusion

You learned about various Linux commands to see packet loss on Linux per-interface, including excellent tools such as dropwatch. We can also use Linux profiling with performance counters utility called perf. Check out perf examples for further information.

🐧 Get the latest tutorials on SysAdmin, Linux/Unix, Open Source/DevOps topics:
CategoryList of Unix and Linux commands
File Managementcat
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network Utilitiesdig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNAlpine CentOS 8 Debian 10 Firewall Ubuntu 20.04

ADVERTISEMENTS
0 comments… add one

Leave a Reply

Your email address will not be published.

Use HTML <pre>...</pre>, <code>...</code> and <kbd>...</kbd> for code samples.