Seeking help for packet loss, network latency on lan
suny4t4
Member Posts: 1 ■□□□□□□□□□
in Networking
Hi,
For days now I am trying to find the cause for packet loss on my own lan to any other IP on the lan, also the router/gateway IP. I am running debian 12 os on a poweredge r420 server. I run mtr 192.168.178.1 and ghet a 15% packet loss which can also be noticed in the games which are served by the server to the internet. I have no idea what top trouble shoot next. There is a xen-bridge on the nic.
--------------------------------------
my /etc/network/interfaces is this:
auto eno1
iface eno1 inet dhcp (no cable is connected to eno1)
iface eno2 inet manual
auto xenbr0
iface xenbr0 inet dhcp
iface eno1 inet dhcp (no cable is connected to eno1)
iface eno2 inet manual
auto xenbr0
iface xenbr0 inet dhcp
-----------------------
ip link shows
ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether f0:1f:af:e8:bc:7b brd ff:ff:ff:ff:ff:ff
altname enp2s0f0
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 1000
link/ether f0:1f:af:e8:bc:7c brd ff:ff:ff:ff:ff:ff
altname enp2s0f1
4: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 62:59:16:4a:b5:a1 brd ff:ff:ff:ff:ff:ff
inet 192.168.178.80/24 brd 192.168.178.255 scope global dynamic xenbr0
valid_lft 519321sec preferred_lft 519321sec
5: vif1.0-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master xenbr0 state UNKNOWN group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
6: vif1.0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr0 state DOWN group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
10: vif3.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether f0:1f:af:e8:bc:7b brd ff:ff:ff:ff:ff:ff
altname enp2s0f0
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 1000
link/ether f0:1f:af:e8:bc:7c brd ff:ff:ff:ff:ff:ff
altname enp2s0f1
4: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 62:59:16:4a:b5:a1 brd ff:ff:ff:ff:ff:ff
inet 192.168.178.80/24 brd 192.168.178.255 scope global dynamic xenbr0
valid_lft 519321sec preferred_lft 519321sec
5: vif1.0-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master xenbr0 state UNKNOWN group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
6: vif1.0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr0 state DOWN group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
10: vif3.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
------------------------------------------------
ip route shows
default via 192.168.178.1 dev xenbr0
192.168.178.0/24 dev xenbr0 proto kernel scope link src 192.168.178.80
192.168.178.0/24 dev xenbr0 proto kernel scope link src 192.168.178.80
------------------------------------------------
lsof -i shows
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
avahi-dae 886 avahi 12u IPv4 33868 0t0 UDP *:mdns
avahi-dae 886 avahi 13u IPv6 33869 0t0 UDP *:mdns
avahi-dae 886 avahi 14u IPv4 33870 0t0 UDP *:37540
avahi-dae 886 avahi 15u IPv6 33871 0t0 UDP *:50366
dhclient 987 root 7u IPv4 25652 0t0 UDP *:bootpc
dhclient 1184 root 7u IPv4 24719 0t0 UDP *:bootpc
glances 1234 root 4u IPv4 12433 0t0 TCP localhost:61209 (LISTEN)
sshd 1272 root 3u IPv4 34897 0t0 TCP *:ssh (LISTEN)
sshd 1272 root 4u IPv6 34899 0t0 TCP *:ssh (LISTEN)
apache2 1303 root 3u IPv4 41041 0t0 TCP *:http (LISTEN)
master 1718 root 13u IPv4 12475 0t0 TCP *:smtp (LISTEN)
master 1718 root 14u IPv6 12476 0t0 TCP *:smtp (LISTEN)
xen-qemu- 2052 root 11u IPv4 37129 0t0 TCP localhost:5900 (LISTEN)
apache2 4842 www-data 3u IPv4 41041 0t0 TCP *:http (LISTEN)
apache2 4844 www-data 3u IPv4 41041 0t0 TCP *:http (LISTEN)
apache2 4845 www-data 3u IPv4 41041 0t0 TCP *:http (LISTEN)
cupsd 4901 root 7u IPv4 82038 0t0 TCP localhost:ipp (LISTEN)
sshd 10134 root 4u IPv4 97733 0t0 TCP DsrvSKYNET:ssh->98-155-210-87.ftth.glasoperator.nl:35868 (ESTABLISHED)
sshd 10165 sunyata 4u IPv4 97733 0t0 TCP DsrvSKYNET:ssh->98-155-210-87.ftth.glasoperator.nl:35868 (ESTABLISHED)
WSServer- 13149 soulmask 15u IPv4 143397 0t0 TCP localhost:44989 (LISTEN)
WSServer- 13149 soulmask 19u IPv4 143399 0t0 UDP *:27017
WSServer- 13149 soulmask 23u IPv4 140915 0t0 TCP localhost:18888 (LISTEN)
WSServer- 13149 soulmask 35u IPv4 122628 0t0 UDP *:7784
WSServer- 13149 soulmask 37u IPv4 140916 0t0 TCP *:27020 (LISTEN)
WSServer- 13149 soulmask 45u IPv4 143412 0t0 TCP DsrvSKYNET:37253->185.25.182.52:27037 (ESTABLISHED)
xen-qemu- 13906 root 11u IPv4 149739 0t0 TCP localhost:5901 (LISTEN)
sshd 15664 root 4u IPv4 163974 0t0 TCP DsrvSKYNET:ssh->98-155-210-87.ftth.glasoperator.nl:52162 (ESTABLISHED)
sshd 15672 sunyata 4u IPv4 163974 0t0 TCP DsrvSKYNET:ssh->98-155-210-87.ftth.glasoperator.nl:52162 (ESTABLISHED)
--------------------------------------------------------------
netstat -s shows
Ip:
Forwarding: 2
1261245 total packets received
9 with invalid headers
4 with invalid addresses
0 forwarded
0 incoming packets discarded
1248166 incoming packets delivered
837064 requests sent out
240 dropped because of missing route
Icmp:
8288 ICMP messages received
6 input ICMP message failed
ICMP input histogram:
destination unreachable: 315
timeout in transit: 519
echo requests: 47
echo replies: 7407
9757 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 297
echo requests: 9413
echo replies: 47
IcmpMsg:
InType0: 7407
InType3: 315
InType8: 47
InType11: 519
OutType0: 47
OutType3: 297
OutType8: 9413
Tcp:
2515 active connection openings
44 passive connection openings
1175 failed connection attempts
3 connection resets received
3 connections established
947497 segments received
658901 segments sent out
2599 segments retransmitted
1 bad segments received
1847 resets sent
Udp:
211857 packets received
8 packets to unknown port received
0 packet receive errors
169041 packets sent
0 receive buffer errors
0 send buffer errors
IgnoredMulti: 79823
UdpLite:
TcpExt:
992 TCP sockets finished time wait in fast timer
7 packets rejected in established connections because of timestamp
2037 delayed acks sent
1 delayed acks further delayed because of locked socket
Quick ack mode was activated 42 times
39832 packet headers predicted
16467 acknowledgments not containing data payload received
40030 predicted acknowledgments
TCPSackRecovery: 144
TCPDSACKUndo: 77
56 congestion windows recovered without slow start after partial ack
TCPLostRetransmit: 24
TCPSackFailures: 2
300 fast retransmits
377 retransmits in slow start
TCPTimeouts: 398
TCPLossProbes: 1683
TCPLossProbeRecovery: 614
TCPSackRecoveryFail: 3
TCPBacklogCoalesce: 2861
TCPDSACKOldSent: 179
TCPDSACKOfoSent: 49
TCPDSACKRecv: 654
301 connections reset due to unexpected data
2 connections reset due to early user close
1 connections aborted due to timeout
TCPDSACKIgnoredNoUndo: 358
TCPSackShifted: 91
TCPSackMerged: 123
TCPSackShiftFallback: 536
TCPRcvCoalesce: 538691
TCPOFOQueue: 6110
TCPOFOMerge: 49
TCPChallengeACK: 2
TCPSYNChallenge: 1
TCPSpuriousRtxHostQueues: 3
TCPAutoCorking: 16873
TCPFromZeroWindowAdv: 3
TCPToZeroWindowAdv: 3
TCPSynRetrans: 18
TCPOrigDataSent: 75702
TCPHystartTrainDetect: 2
TCPHystartTrainCwnd: 46
TCPHystartDelayDetect: 2
TCPHystartDelayCwnd: 87
TCPACKSkippedPAWS: 3
TCPKeepAlive: 2
TCPDelivered: 77072
TCPAckCompressed: 4842
TcpTimeoutRehash: 397
TCPDSACKRecvSegs: 654
IpExt:
InMcastPkts: 1260
OutMcastPkts: 814
InBcastPkts: 79829
InOctets: 4851330675
OutOctets: 114849976
InMcastOctets: 97650
OutMcastOctets: 43656
InBcastOctets: 22505209
InCsumErrors: 9
InNoECTPkts: 3794573
InECT0Pkts: 1
Forwarding: 2
1261245 total packets received
9 with invalid headers
4 with invalid addresses
0 forwarded
0 incoming packets discarded
1248166 incoming packets delivered
837064 requests sent out
240 dropped because of missing route
Icmp:
8288 ICMP messages received
6 input ICMP message failed
ICMP input histogram:
destination unreachable: 315
timeout in transit: 519
echo requests: 47
echo replies: 7407
9757 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 297
echo requests: 9413
echo replies: 47
IcmpMsg:
InType0: 7407
InType3: 315
InType8: 47
InType11: 519
OutType0: 47
OutType3: 297
OutType8: 9413
Tcp:
2515 active connection openings
44 passive connection openings
1175 failed connection attempts
3 connection resets received
3 connections established
947497 segments received
658901 segments sent out
2599 segments retransmitted
1 bad segments received
1847 resets sent
Udp:
211857 packets received
8 packets to unknown port received
0 packet receive errors
169041 packets sent
0 receive buffer errors
0 send buffer errors
IgnoredMulti: 79823
UdpLite:
TcpExt:
992 TCP sockets finished time wait in fast timer
7 packets rejected in established connections because of timestamp
2037 delayed acks sent
1 delayed acks further delayed because of locked socket
Quick ack mode was activated 42 times
39832 packet headers predicted
16467 acknowledgments not containing data payload received
40030 predicted acknowledgments
TCPSackRecovery: 144
TCPDSACKUndo: 77
56 congestion windows recovered without slow start after partial ack
TCPLostRetransmit: 24
TCPSackFailures: 2
300 fast retransmits
377 retransmits in slow start
TCPTimeouts: 398
TCPLossProbes: 1683
TCPLossProbeRecovery: 614
TCPSackRecoveryFail: 3
TCPBacklogCoalesce: 2861
TCPDSACKOldSent: 179
TCPDSACKOfoSent: 49
TCPDSACKRecv: 654
301 connections reset due to unexpected data
2 connections reset due to early user close
1 connections aborted due to timeout
TCPDSACKIgnoredNoUndo: 358
TCPSackShifted: 91
TCPSackMerged: 123
TCPSackShiftFallback: 536
TCPRcvCoalesce: 538691
TCPOFOQueue: 6110
TCPOFOMerge: 49
TCPChallengeACK: 2
TCPSYNChallenge: 1
TCPSpuriousRtxHostQueues: 3
TCPAutoCorking: 16873
TCPFromZeroWindowAdv: 3
TCPToZeroWindowAdv: 3
TCPSynRetrans: 18
TCPOrigDataSent: 75702
TCPHystartTrainDetect: 2
TCPHystartTrainCwnd: 46
TCPHystartDelayDetect: 2
TCPHystartDelayCwnd: 87
TCPACKSkippedPAWS: 3
TCPKeepAlive: 2
TCPDelivered: 77072
TCPAckCompressed: 4842
TcpTimeoutRehash: 397
TCPDSACKRecvSegs: 654
IpExt:
InMcastPkts: 1260
OutMcastPkts: 814
InBcastPkts: 79829
InOctets: 4851330675
OutOctets: 114849976
InMcastOctets: 97650
OutMcastOctets: 43656
InBcastOctets: 22505209
InCsumErrors: 9
InNoECTPkts: 3794573
InECT0Pkts: 1
im out of idea's to investigate, please help ?
Comments
-
JDMurray Admin Posts: 13,101 AdminCisco says to start troubleshooting at Layer 1 and work your way up the stack. Check your cables, hubs, switches etc. for breaks, frays, corrosion, exposure to sources of RFI, etc. Try moving or swapping out cables, power-cycling network midpoint devices, applying firmware patches, etc. If Wi-Fi is involved, note if the packet loss is better or worse on those links.