[FreeBSD] Randomly Ether link down/up issue with "route -n monitor"(kernel Netlink enable /sbin/route ,14.x and later, some USB Ether interface)

[Change History]

2025/09/25 change [Reproduce Steps] from Wireguard to "route -n monitor"

2025/09/27 add NOTE 2(why so many ARP, request in this environment), NOTE3 and workaround

Reproduce Steps

1. connect network
2. start route -n monitor
3. start iperf -s on other machine
4. start iperf on target machine

#!/bin/sh

IPERF_SERVER_IP=[IPERF Server IP]

IPERF=/usr/local/bin/iperf

while [ true ]; do

        ${IPERF} -c ${IPERF_SERVER_IP} -e -i 1

        sleep 1

        ${IPERF} -c ${IPERF_SERVER_IP} -u -e -i 1 --isochronous=60:100m,10m --realtime

        sleep 1

done

expected Result

no link down/up on USB Ethernet(ue)

Result

sometime link down/up on USB Ethernet(ue)

/var/log/messages

Sep 25 09:17:48 amd64 kernel: ue0: link state changed to DOWN

Sep 25 09:17:48 amd64 kernel: ue0: link state changed to UP

Sep 25 09:25:13 amd64 kernel: ue0: link state changed to DOWN

Sep 25 09:25:13 amd64 kernel: ue0: link state changed to UP

output from "route -n monitor"

09:17:48.536 PID 2391 add/repl neigh 192.168.10.254 state REACHABLE lladdr 00:a0:de:xx;xx:xx iface ue0

09:17:48.547 PID 2391 add/repl iface iface#3 ue0 admin UP oper UP mtu 1500

09:17:48.547 PID    0 add/repl iface iface#3 ue0 admin UP oper UP mtu 1500

09:17:48.547 PID    0 add/repl iface iface#3 ue0 admin UP oper UP mtu 1500

09:17:48.547 PID    0 add/repl iface iface#3 ue0 admin UP oper UP mtu 1500

09:17:49.537 PID 2391 add/repl neigh 192.168.10.254 state REACHABLE lladdr 00:a0:de:xx;xx:xx iface ue0

[SNIP]

09:25:12.572 PID 2391 add/repl neigh 192.168.10.254 state REACHABLE lladdr 00:a0:de:xx;xx:xx iface ue0

09:25:13.573 PID 2391 add/repl neigh 192.168.10.254 state REACHABLE lladdr 00:a0:de:xx;xx:xx iface ue0

09:25:13.583 PID 2391 add/repl iface iface#3 ue0 admin UP oper UP mtu 1500

09:25:13.583 PID    0 add/repl iface iface#3 ue0 admin UP oper UP mtu 1500

09:25:13.583 PID    0 add/repl iface iface#3 ue0 admin UP oper UP mtu 1500

09:25:13.583 PID    0 add/repl iface iface#3 ue0 admin UP oper UP mtu 1500

09:25:14.867 PID 2391 add/repl neigh 192.168.10.254 state REACHABLE lladdr 00:a0:de:xx;xx:xx iface ue0

09:25:16.777 PID 2391 add/repl neigh 192.168.10.254 state REACHABLE lladdr 00:a0:de:xx;xx:xx iface ue0

[NOTE 1]

"route -n monitor" command is used on WireGuard(/usr/local/bin/wg-quick shell script)

[NOTE 2]

Here's the reason why so many "state REACHABLE" entries (one to two, or even more, per second) are being outputted by "route -n monitor" in this environment.

When the Ricoh ScanSnap application "ScanSnap Home" is running on a PC (192.168.10.198) and the ScanSnap scanner is powered off, the application sends ARP Request commands to search for the scanner. The FreeBSD box on same subnet reacts to these packets, and they are outputted by route -n monitor. (The application continues to send ARP Requests until it's force-closed).

ScanSnap Home-related Packets

tcpdump results:

14:12:00.058366 ARP, Request who-has 192.168.10.250 tell 192.168.10.198, length 50

14:12:00.058413 ARP, Request who-has 192.168.10.251 tell 192.168.10.198, length 50

14:12:00.058426 ARP, Request who-has 192.168.10.252 tell 192.168.10.198, length 50

route -n monitor results from roughly the same time:

14:12:00.512 PID    0 add/repl neigh 192.168.10.198 state REACHABLE lladdr 00:50:b6:nn:nn:nn iface ue0

14:12:00.029 PID    0 add/repl neigh 192.168.10.198 state REACHABLE lladdr 00:50:b6:nn:nn:nn iface ue0

14:12:00.059 PID    0 add/repl neigh 192.168.10.198 state REACHABLE lladdr 00:50:b6:nn:nn:nn iface ue0

Additionally, ARP Requests are also periodically occurring for a NAT server in the route that is currently offline.

Router's Server Search Packets

tcpdump results:

14:17:35.772158 ARP, Request who-has 192.168.10.10 tell 192.168.10.254, length 50

14:17:36.773378 ARP, Request who-has 192.168.10.10 tell 192.168.10.254, length 50

14:17:37.774597 ARP, Request who-has 192.168.10.10 tell 192.168.10.254, length 50

route -n monitor results from roughly the same time:

14:18:06.709 PID    0 add/repl neigh 192.168.10.254 state REACHABLE lladdr 00:a0:de:nn:nn:nn iface ue0

14:18:08.711 PID    0 add/repl neigh 192.168.10.254 state REACHABLE lladdr 00:a0:de:nn:nn:nn iface ue0

[NOTE 3]

The issue (USB Ethernet link down/up) is that when an Ethernet driver without this problem (e.g., awg0) receives an ARP request, and NetLink processes it and reports it to route -n monitor, the affected USB Ethernet adapter (which is linked up and communicating separately) experiences a link down/up phenomenon despite not receiving the ARP packet on USB Ethernet.

Workaround

The root cause seems to be an issue between Netlink and the Ethernet Driver, but I don't have time to investigate it. Therefore, we're currently using the following workaround for FreeBSD-14 and later.

Workaround1 change /sbin/route

don't printout the neigh change

-

-- sbin/route/route_netlink.c.org      2025-09-27 14:26:39.289721000 +0900

+++ sbin/route/route_netlink.c  2025-09-27 14:27:21.644803000 +0900

@@ -753,7 +753,9 @@

        int groups[] = {

                RTNLGRP_LINK,

+#if 0

                RTNLGRP_NEIGH,

+#endif

                RTNLGRP_NEXTHOP,

 #ifdef INET

                RTNLGRP_IPV4_IFADDR,

Workaround2

just kill "route -n monitor" process after staring the Wireguard.

Current status

FreeBSD Version	CPU	Ethernet Interface	enable route monitor	reproduce?	Notes
15.0-ALPHA2	amd64	ue(axe)AX88772	YES	YES[NG]
			NO	NO[OK]
		ue(axe)AX88178	YES	YES[NG]
			NO	NO[OK]
		ue(axge) ELECOM EDC-GUC3L-W	YES	YES[NG]
			NO	NO[OK]
		ue(ure) Lenovo USB-C to LAN)	YES	YES[NG]
			NO	NO[OK]
		ue(ure) Planex USB-LAN2500R	YES	NO[OK]
			NO	-
		ue(cdce)Lenovo USB-C to LAN	YES	NO[OK]
			NO	-
		ue(cdce)Planex USB-LAN2500R	YES	NO[OK]
			NO	-
		run(Wi-Fi)	YES	NO[OK]
			NO	-
		em0(On board)	YES	NO[OK]
			NO	-
15.0-ALPHA3	arm64	ue(axe)AX88772	YES	?
			NO	NO[OK]
		ue(axe)AX88178	YES	?
			NO	?
		ue(axge) ELECOM EDC-GUC3L-W	YES	unstable, packet loss about 37%(maybe other issue) ,even when disable all options(-txcsum -rxcsum)
			NO
		ue(ure) Lenovo USB-C to LAN)	YES	YES[NG]
			NO	NO[OK]
		ue(ure) Planex USB-LAN2500R	YES	NO[OK]	write-fin failed: No buffer space available
			NO	-
		ue(cdce)Lenovo USB-C to LAN	YES	NO[OK]
			NO	-
		ue(cdce)Planex USB-LAN2500R	YES	NO[OK]
			NO	-
		run(Wi-Fi)	YES	NO[OK]
			NO	-
		Internal IF(On Board)	YES	NO[OK]
			NO	-
14.3-RELEASE	amd64	ue(axe)	YES	?
			NO	?
	arm64	ue(axe)AX88772	YES	YES[NG]
			NO	NO[OK]
		Internal IF	YES	NO[OK]	Oracle cloud
13.5-RELEASE	amd64	ue(axe)AX88772	YES	NO[OK]
	arm64		YES	NO[OK]

• Could changing the order of the interfaces shown by ifconfig from wg0 to ue0 solve the problem?
• Result: It won't solve it[NG].
• disable the ifconfig options (eg. -txcsum -rxcsum)
• Result: It won't solve it[NG].

Enviroment

USB Ethernet

• axe: 
• BUFFALO LUA3-U2-ATX AX88772
  <ASIX Elec. Corp. AX88772, rev 2.00/0.01, addr 2> on usbus0
  ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
          options=80008<VLAN_MTU,LINKSTATE>
  miibus1: <MII bus> on axe0
  ukphy0: <Generic IEEE 802.3u media interface> PHY 16 on miibus1
  ukphy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
  
  
• BUFFALO LUA3-U2-AGT AX88178
  <BUFFALO INC. product 0x006e> at usbus1  ue0:flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=80008<VLAN_MTU,LINKSTATE> 
  miibus1: <MII bus> on axe0
  e1000phy0: <Marvell 88E1111 Gigabit PHY> PHY 24 on miibus1
  e1000phy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
  
  
• axge
• ELECOM EDC-GUC3L-W AX8179
  axge0: <ASIX Elec. Corp. AX88179, rev 2.10/1.00, addr 2> on usbus7 miibus0: <MII bus> on axge0 rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 3 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
• ure
• Lenovo USB-C to LAN RTL8153
  ure0: <Lenovo Lenovo USB-C to LAN, class 0/0, rev 2.10/31.20, addr 2> on usbus7
  miibus0: <MII bus> on ure0
  rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 0 on miibus0
  rgephy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
  options=68009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
• Planex USB-LAN2500R
  ure0: <Planex USB 10/100/1G/2.5G LAN, class 0/0, rev 2.10/30.00, addr 2> on usbus7
  options=60009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  
  
• cdce (setup info FreeBSD Bugzilla 252165)
• Lenovo USB-C to LAN RTL8251/8153 
  options=80000<LINKSTATE>
• Planex USB-LAN2500R
  
  

• run0: <Ralink 802.11 n WLAN, class 0/0, rev 2.00/1.01, addr 2> on usbus7
  run0: MAC/BBP RT3071 (rev 0x021C), RF RT3022 (MIMO 2T2R), address 4c:xx:xx:xx:xx
• BUFFALO WLI-UC-G300HP

amd64 machine

• Lenovo ThinkPad T400s

arm64 machines

• PINE A64
• Raspberry Pi 4
• Raspberry Pi Zero 2W

Related articles

FreeBSD Forums:

• Internet connection going up and down constantly
• Charlie Brown reported using the wg0 with ue0
• Interface flapping on ethernet USB adapter
  
• no information about Wireguard, finally use cdce.
• Link rapidly goes UP and DOWN

FreeBSD Bugzilla

• Bug 252165 - usb network and mii bus media status race condition
  
• Bug 267514 - AXGE(4) ASIX AX88179A ue0: link state changed to DOWN
• Bug 275026 - net/wireguard-tools: after service wireguard stop, route monitor keeps running in the background
  
• Bug 276414 - rtnetlink: destroying an interface generates spurious RTM_NEWLINKs
• Bug 272144 - netlink sets ifi_type to FreeBSD if_type instead of ARPHRD_*

[FreeBSD] use fib for specific wireguard interface

Background

My FreeBSD box has two interface, net0 is main fiber line(fast speed), net1  is backup line(Japan KDDI povo slow line).

At first, I set static route for backup wireguard line.  Unfortenately, all packet to the static route([End Point IP address]) uses slow line[NG].

for Example previous config(/etc/rc.conf)

static_routes="backup_line"

route_backup_line="-net [End Point IP address] -gateway 192.168.220.1"

(192.168.220.1 is backup line default router)

Now I know the fib(Forwarding Informations Base), I use fib 1 for backup wireguard line. 

Backup line is only used by below wireguard line[OK].

And I can remove above static route.

Step1:  /etc/sysctl.conf

add below things into /etc/sysctl.conf(enable fibs)

net.fibs=2

net.add_addr_allfibs=1

Step2:  /etc/rc.conf

# net1 is for fib1 

ifconfig_ue0_name="net1"

ifconfig_net1="192.168.220.3/24 fib 1"

# set default route for fib 1

static_routes="fib1_default"

route_fib1_default="-fib 1 default -gateway 192.168.220.1"

Step3:  /usr/local/etc/wireguard/wg0.conf

use PostUp line

[Interface]

Privatekey = [Privatekey]

Address = 10.8.41.1/24

table = off

ListenPort = 1111

PostUp = /sbin/ifconfig wg0 tunnelfib 1

[Peer]

PublicKey = IOx7n4S1rBZPlj6an2rMQSddj07RMQFkWPAyvtJsYnY=

AllowedIPs = 10.8.0.0/16, 192.168.0.0/24

Endpoint = [End Point IP address]:1111

PersistentKeepAlive = 30

NOTES:

When you uses multiple wireguard lines, if you want selectively to use net0(fast line)for one of wireguard line, you use wg*.conf without "PostUp = /sbin/ifconfig wg* tunnelfib 1".