Fixing Solana Validator Packet Loss:
A FreeBSD RSS Rabbit Hole
How a ten-year-old FreeBSD decision, a NIC hardware quirk, and a VLAN tagging assumption combined to silently destroy validator performance — and how we fixed it.
At Solana Vibe Station we operate validators in both Amsterdam and Atlanta, sitting behind OPNsense HA firewall pairs on custom hardware. For months our Amsterdam validator was under-performing during leader slots — missing vote transactions that should have arrived, filling leader slots partially instead of fully. What looked like a colocation issue, then a firewall misconfiguration, turned out to be a ten-year-old FreeBSD kernel decision interacting with an Intel E810 hardware behavior in a way nobody had bothered to fix. This post documents the full journey.
Votes Vanishing During Leader Slots
A healthy Solana validator expects around 760 vote transactions per leader slot. Our Amsterdam validator was consistently landing closer to 480–493 — a 35–40% shortfall that didn't show up in any single obvious metric. The validator wasn't crashing. The firewall wasn't logging drops. UDP connections looked fine from the outside.
The pain was specifically during leader slots — the windows where our validator is producing blocks and needs to receive the highest volume of inbound QUIC/UDP traffic. Outside of leader slots, the validator appeared to perform normally, which made this harder to diagnose and easy to initially dismiss as network jitter.
First Wave: Surface-Level Fixes
The initial investigation followed a logical progression — check the most common causes first. Each fix provided partial improvement but didn't solve the underlying problem.
netstat. Increased ring buffers to 4096 (E810 maximum) via override_nrxds/override_ntxds.
All of these fixes were real improvements but none of them solved the core issue. The fundamental problem was that our firewall NIC was failing to distribute packet processing across CPU cores, creating a single-threaded bottleneck that saturated under leader slot load.
netstat and netstat -Q can mislead you. NIC-level hardware drops (rx_missed_errors, rx_no_desc) happen before the OS ever sees the packets. Always check driver-level counters first with sysctl dev.ice | grep rx_no_desc.
The DDoS Rabbit Hole
With local fixes exhausted, we turned suspicion toward the upstream network. Solana validators during leader slots receive traffic spikes that can look indistinguishable from a UDP flood to automated DDoS mitigation systems — potentially hundreds of thousands of packets per second on QUIC and legacy TPU ports, all from distributed peers worldwide.
Our Amsterdam infrastructure sits behind a major European colocation provider with multiple upstream transit providers, all of which run their own automated filtering tiers. Many such facilities are also adjacent to large internet exchanges that provide their own Anti-DDoS services in the path.
The Theory
A Solana validator receiving 500K+ UDP packets/second during a leader slot looks exactly like a DDoS attack to any filtering system that isn't blockchain-aware. Automated scrubbing at the carrier or IX layer could be silently dropping packets without any notification — no ICMP unreachables, no visible errors on our end.
We explored DDoS mitigation policies with our upstream provider and considered options for direct cross-connects that would bypass automated scrubbing. While this remains a theoretically valid concern for any Solana operator in a major datacenter, it turned out not to be the primary cause of our specific problem. The real issue was entirely within our own firewall stack — the upstream network was delivering the packets, but our firewall couldn't process them fast enough.
The Real Problem: RSS Completely Broken on E810 + VLAN
The breakthrough came when we started looking at actual queue-level packet distribution using vmstat -i and netstat -Q. The data was damning.
Interface Queue Interrupts Packets ice3 rxq0 4,821,032 18.2% ← WAN, properly distributed ice3 rxq1 4,799,441 18.1% ice3 rxq2 4,803,218 18.1% ice3 rxq3 4,788,901 18.1% ice3 rxq4-7 ≈18.6% even split ice0 rxq0 31,402,871 99.7% ← LAN, completely broken ice0 rxq1 44,821 0.14% ice0 rxq2 31,024 0.09% ice0 rxq3-7 <0.05% ghost queues
The WAN interface (ice3) was distributing traffic nearly perfectly across all 8 queues. The LAN interface (ice0) was jamming 99.7% of all packets onto a single CPU core. That one CPU was saturating under leader slot load, creating a software bottleneck after the NIC.
Why Was LAN Broken But WAN Fine?
Our Amsterdam setup has the validator traffic on a tagged VLAN interface layered on top of ice0. The WAN interface (ice3) receives untagged traffic directly. This difference turned out to be critical.
The Intel E810 NIC's RSS engine computes its hash on the packet headers it sees at the MAC layer. When an 802.1q VLAN tag is present, the 4-byte VLAN header sits between the Ethernet header and the IP header. The E810 was hashing the VLAN tag bytes instead of the actual IP source/destination, producing effectively random (but consistent) hash values that happened to map almost entirely to queue 0.
Without VLAN tag (ice3 — WAN): [ Eth Header ][ IP Header ][ UDP Header ][ Payload ] ↑ RSS hashes here correctly With 802.1q VLAN tag (ice0 — LAN): [ Eth Header ][ VLAN Tag ][ IP Header ][ UDP Header ][ Payload ] ↑ RSS hashes HERE — wrong bytes! → almost everything maps to rxq0
This explained why ATL (our Atlanta location using Intel X710 / ixl driver) had proper RSS distribution — the X710 handles VLAN-tagged RSS correctly and shows healthy queue spread. The E810's specific RSS implementation had a known but undocumented behavior with VLAN-tagged flows.
| Location | NIC / Driver | VLAN Config | RSS Distribution | rxq0 Load |
|---|---|---|---|---|
| Amsterdam (AMS) | E810 / ice | 802.1q tagged VLAN | Broken | 99.7% |
| Atlanta (ATL) | X710 / ixl | Native (no VLAN tag) | Working | ~51% (highest queue) |
ice driver) is unique among our NIC fleet in that it actually requests the kernel's desired hash configuration and applies it. Other drivers (ixl, ax) use their own internal defaults. This distinction — the E810 deferring to the FreeBSD kernel's hash config — is what made the next part possible.
Writing a FreeBSD Kernel Patch
Here's the part that makes this situation remarkable: FreeBSD has had UDP 4-tuple RSS hashing disabled in the kernel since 2014. The original commit comment cited concerns about IP fragmentation and stated that "other things need to be put in place" before UDP RSS could be safely enabled. They never came back to it.
The kernel's RSS hash configuration (rss_config.c) tells NICs what fields to use when computing RSS hashes. For TCP it specifies 4-tuple hashing (src IP, dst IP, src port, dst port). For UDP, FreeBSD returns a config that only uses the IP pair — no ports. This means all UDP traffic between the same two hosts always lands on the same queue, regardless of port numbers.
For most workloads this doesn't matter. For a Solana validator receiving QUIC/UDP traffic from hundreds of peers simultaneously — all targeting the same destination IP and port — it's catastrophic. All of that traffic hashes identically and piles onto one queue.
/* UDP RSS hash type — pre-patch behavior */ /* UDP 4-tuple hashing disabled since 2014 */ /* Only source/dest IP used for UDP flows */ rss_hashtype |= RSS_HASHTYPE_RSS_UDP_IPV4; /* 2-tuple only */ /* RSS_HASHTYPE_RSS_UDP_IPV4_EX never set */
/* New RDTUN sysctl — must be set at boot in loader.conf */ static int rss_udp_4tuple = 0; SYSCTL_INT(_net_inet_rss, OID_AUTO, udp_4tuple, CTLFLAG_RDTUN, &rss_udp_4tuple, 0, "Enable UDP 4-tuple (src/dst IP + port) RSS hashing"); /* In rss_gethashconfig(): */ if (rss_udp_4tuple) { rss_hashtype |= RSS_HASHTYPE_RSS_UDP_IPV4_EX; rss_hashtype |= RSS_HASHTYPE_RSS_UDP_IPV6_EX; }
Because the E810 is the only NIC in our stack that requests hash config from the kernel (vs. using hardcoded driver defaults), this patch directly controlled what hash keys the E810 was programming into its RSS lookup table at boot. Enabling 4-tuple UDP hashing meant the NIC would now factor source port into the RSS calculation — giving genuinely distinct hash values for different QUIC connection flows even when they share the same destination.
Why This Is Safe for Our Use Case
The 2014 concern about IP fragmentation is valid in general: IP fragments don't carry port information, so a fragmented UDP packet's second fragment can't be 4-tuple hashed and may end up on a different queue than the first fragment. However, Solana's QUIC/UDP traffic isn't fragmented in practice — QUIC handles its own packetization at a layer above UDP, and all packets fit within MTU. The IP fragmentation risk is real but irrelevant for this specific traffic pattern.
The patch was submitted upstream to FreeBSD: github.com/freebsd/freebsd-src/pull/2057
Compiling the Kernel on Live Hardware
To deploy the patch we needed to compile a custom FreeBSD kernel and install it on the OPNsense firewall. Rather than setting up a cross-compilation environment, we compiled directly on the firewall hardware itself — keeping the toolchain simple and ensuring the kernel config matched the running system exactly.
kern.conftxt sysctl exports the current kernel's complete configuration — a perfect starting point.
# Install FreeBSD source matching current OPNsense version pkg install git llvm binutils git clone --depth=1 \ -b releng/14.1 \ https://github.com/freebsd/freebsd-src.git \ /usr/src
# This exports the EXACT config of the currently running kernel # Use it directly as the build config — no guessing required sysctl kern.conftxt > /usr/src/sys/amd64/conf/SMP
# Apply UDP 4-tuple RSS patch to rss_config.c cd /usr/src patch -p1 < /tmp/rss-udp-4tuple.patch # Build only the kernel (not the full world — much faster) make -j$(sysctl -n hw.ncpu) \ KERNCONF=SMP \ buildkernel 2>&1 | tee /tmp/kernel-build.log
# Install to a separate kernel directory make KERNCONF=SMP DESTDIR=/ installkernel KODIR=/boot/kernel.test # Boot into it ONCE ONLY — if it panics or hangs, next boot # reverts to the previous kernel automatically nextboot -k kernel.test reboot
The nextboot approach is crucial for a production firewall. If the patched kernel causes any instability, the system automatically falls back to the known-good kernel on the following boot. No risk of bricking the firewall with a bad kernel.
# Confirm new sysctl exists (RDTUN — only settable at boot) sysctl net.inet.rss.udp_4tuple net.inet.rss.udp_4tuple: 1 # Confirm it's being applied (check hash type bitmask) sysctl net.inet.rss.hash_type net.inet.rss.hash_type: 843 # bit for UDP_IPV4_EX should now be set
Layered Tuning: The Final Stack
The kernel patch fixed the fundamental RSS distribution problem, but achieving stable high-performance operation required a stack of additional tunables. Each one addressed a specific failure mode discovered during testing.
The rss.bits Problem
Setting net.inet.rss.bits to values below 4 caused a mysterious pathology: the majority of packets were being deferred (queued to a software queue) instead of directly dispatched to the mapped CPU core. With hybrid dispatch mode, packets should land directly on the core that owns the RSS bucket. Instead they were hitting the wrong cores and getting passed off.
With rss.bits = 3 (8 buckets mapped to CPUs 0–7): ice3 WAN queues → CPUs 8–15 (from MSI-X IRQ affinity) RSS buckets → CPUs 0–7 (from rss.bits=3 mapping) Mismatch! Packet arrives on CPU 12, bucket says CPU 4 → netisr defers to software queue instead of direct dispatch → 90%+ of packets get queued instead of directly processed → QDisp'd spikes, latency accumulates ──────────────────────────────────────────────────────── With rss.bits = 4 (16 buckets mapped to CPUs 0–15): ice3 WAN queues → CPUs 0–15 RSS buckets → CPUs 0–15 Aligned! Packet arrives on its mapped CPU → Direct dispatch (HDisp'd) >99% → Zero software queue buildup
Hardware Offloading: Disable Everything
This one was counterintuitive. Hardware offloading (rxcsum, txcsum, TSO, LRO, etc.) is supposed to reduce CPU load by pushing work onto the NIC. In practice, on our setup it was causing latency spikes and instability. Disabling all hardware offloads via the OPNsense interface produced an immediate and significant improvement in network stability — not raw throughput, but consistency and latency variance.
The likely cause: offloads interact poorly with PF's packet inspection. PF needs to see complete, reassembled packets to evaluate firewall rules. When LRO coalesces multiple packets before PF sees them, it can cause state table inconsistencies. When TSO segments packets after PF has already evaluated them, rule evaluation may have been based on data that no longer matches the actual wire frames. The result is subtle corruption of the processing pipeline at high packet rates.
Final Tunable Reference
loader.conf.local — boot-time tunables
# RSS core configuration net.inet.rss.enabled=1 net.inet.rss.bits=4 # 16 buckets — matches 16-core system net.inet.rss.udp_4tuple=1 # Our kernel patch — RDTUN, boot-only # netisr — allow all threads, bind to specific CPUs net.isr.maxthreads=-1 net.isr.bindthreads=1 # Software queue depth before QDrops start net.inet.ip.intr_queue_maxlen=4096 # PF state table — reduce hash collisions # Should be at least 2x your average concurrent states net.pf.states_hashsize=262144 # NIC ring buffer descriptors — max out both RX and TX # Default is 1024; E810 supports up to 4096 dev.ice.0.iflib.override_nrxds=4096 dev.ice.3.iflib.override_nrxds=4096 dev.ice.0.iflib.override_ntxds=4096 dev.ice.3.iflib.override_ntxds=4096
sysctl dev.ice | grep credits
Essential Debugging Commands
# HDisp'd = directly processed (good) | Queued = deferred (latency) # QDrops = software drops (bad, means intr_queue_maxlen is too small) # WMark hitting intr_queue_maxlen means QDrops are happening netstat -Q | grep -E "WSID| ip "
vmstat -i # Look for uneven distribution across rxq0–rxqN
sysctl dev.ice | grep rx_no_desc # Any value > 0 means NIC drops sysctl dev.ice | grep credits # Should show ~4095 if override worked
CARP HA Pair Instability: The Hyperthreading Trap
While tuning the primary firewall, we also disabled hyperthreading via machdep.hyperthreading_allowed=0 to attempt to reduce CPU contention between netisr threads and other workloads. This triggered an unexpected failure mode in the CARP HA pair.
net.isr.maxthreads=-1 tells netisr to create one thread per CPU — but it reads the CPU count at boot time, before hyperthreading is restricted. On some kernel configurations this caused netisr to create threads for logical CPUs that were now unschedulable, leaving those threads permanently blocked trying to run on cores the OS had hidden.
The CARP protocol depends on timely heartbeat delivery between the primary and secondary firewall. CARP heartbeats travel through netisr. If netisr threads are blocked or delayed, CARP heartbeats miss their deadline, the secondary declares the primary dead, and the CARP pair flaps — the secondary promotes itself, both nodes think they're primary, and chaos follows.
Sequence of events: 1. machdep.hyperthreading_allowed=0 set in loader.conf → Kernel boots, sees 8 physical CPUs (not 16) 2. net.isr.maxthreads=-1 → netisr spawns threads for CPUs 0–7 ✓ → (In some cases) netisr also tries CPUs 8–15 ✗ → Threads for 8–15 can never be scheduled 3. netisr thread for CARP packets → blocked on CPU 12 (doesn't exist) → CARP heartbeat delayed by >500ms 4. Secondary node misses heartbeats → Declares primary dead → Promotes itself to MASTER state 5. Split-brain: both nodes respond to CARP VIP → Network instability until one yields
The fix: don't disable hyperthreading while using net.isr.maxthreads=-1. Either leave hyperthreading enabled, or explicitly cap maxthreads to the actual physical core count to prevent netisr from creating phantom threads. The performance benefit of disabling HT for our use case was marginal anyway — the CPU contention issue was better addressed by RSS bucket alignment.
OPNsense Secondary: A Different Set of Problems
The secondary firewall uses Intel X710 NICs (ixl driver) rather than the E810. Different driver, different problems.
RSS Bucket-to-CPU Mismatch on ixl
The ixl driver assigns MSI-X interrupts differently from ice. On our secondary box, ixl NIC queue interrupts were landing on CPUs 0, 2, 4, and 6 (physical cores), while RSS buckets were mapped to CPUs 0–3. This mismatch meant packets arriving on CPU 6's queue were being handed to RSS bucket CPU 2 — constant cross-CPU shuffling that prevented direct dispatch.
Normally you'd fix this by setting net.inet.rss.bucket_mapping — but this sysctl is defined as CTLFLAG_RD (read-only) in the FreeBSD source. It's computed from rss_getcpu() at boot and cannot be overridden via loader.conf. The bucket mapping is a derived value, not a configuration parameter.
#!/bin/sh # /usr/local/etc/rc.d/ixl_irq_affinity # Pin ixl NIC queue IRQs to CPUs matching RSS buckets # Run after: netif # Get IRQ numbers for ixl0 queues for i in 0 1 2 3; do irq=$(vmstat -i | awk "/ixl0:rxq${i}/{print \$1}" | tr -d ':') cpu=$i # Map rxq0→CPU0, rxq1→CPU1, rxq2→CPU2, rxq3→CPU3 if [ -n "$irq" ]; then cpuset -l $cpu -x $irq fi done
pfsync Interface RSS Concentration
The secondary box uses igc0 as its pfsync interface (the dedicated link that synchronizes PF state tables between primary and secondary). igc0 was concentrating 98% of its traffic on a single queue — which also happens to share CPU resources with the primary NIC's RSS processing. The pfsync traffic was competing with regular firewall traffic for interrupt time on the same core.
Fixing this required either moving pfsync to a dedicated interface with better RSS support, or explicitly pinning igc0's interrupt to a CPU that isn't shared with ixl queue processing — similar to the IRQ affinity fix above.
Results and What's Still Left
After the kernel patch, rss.bits=4 alignment, descriptor ring scaling, and hardware offload disabling, the queue distribution on the primary firewall went from 99.7% concentration to near-perfect spread:
After fix — ice0 LAN queue distribution: rxq0 ████████████████ 6.3% rxq1 ███████████████ 6.2% rxq2 ████████████████ 6.3% rxq3 ███████████████ 6.2% rxq4 ████████████████ 6.3% rxq5 ████████████████ 6.3% rxq6 ███████████████ 6.2% rxq7 ████████████████ 6.3% rxq8 ████████████████ 6.3% rxq9 ███████████████ 6.2% rxq10 ████████████████ 6.3% rxq11 ████████████████ 6.3% rxq12 ███████████████ 6.2% rxq13 ████████████████ 6.3% rxq14 ████████████████ 6.3% rxq15 ████████████████ 6.2%
What's Still Left
There are still occasional rx_no_desc errors on ice0 (the LAN interface). Our current hypothesis: the ax driver interfaces (LAGG for secondary LAN traffic) are mapped to overlapping CPU cores with ice0, causing interrupt contention. When both NICs try to post interrupts on the same core simultaneously, one gets delayed, the descriptor ring doesn't drain fast enough, and a packet drops at the hardware level.
The cleanest fix is to migrate all LAN traffic to the ice interfaces and disable the ax ports entirely, eliminating the cross-driver CPU contention. If descriptor drops persist after that, reducing the interrupt throttle timer (dev.ice.0.vsi.0.rx_itr from the current 50µs toward 10–20µs) would cause the NIC to raise interrupts more aggressively and drain the ring buffer faster.
Broader Takeaways
The core lesson from this debugging journey is that packet loss in high-throughput UDP applications often hides at the hardware boundary — in places that typical monitoring tools don't surface. Metrics that look fine at the application layer can mask catastrophic drops happening before the OS even sees the packets. The diagnostic path for any unexplained packet loss should start at the NIC ring buffer (rx_no_desc, rx_missed_errors), then work up through RSS distribution (vmstat -i), through netisr (netstat -Q), and only then to the application layer.
For anyone running Solana validators or other high-volume UDP applications behind an OPNsense or vanilla FreeBSD firewall with Intel E810 NICs: the combination of this kernel patch and the tuning stack above represents the current state of the art for high-performance UDP forwarding on this platform. The FreeBSD PR is open — hopefully the upstream project picks it up so future operators don't have to patch their own kernels.
Quick Reference: Diagnostic Toolkit
# 1. Check netisr queue health (QDrops = problem, HDisp'd/Queued ratio) netstat -Q | grep -E "WSID| ip " # 2. Check per-queue RSS distribution (should be even across all queues) vmstat -i # 3. Check NIC-level descriptor drops (invisible to netstat!) sysctl dev.ice | grep rx_no_desc # 4. Verify descriptor ring sizes are actually applied sysctl dev.ice | grep credits # Should read ~4095 after override # 5. Verify RSS tunables are active sysctl net.inet.rss sysctl net.isr