FreeBSD Packet Lifecycle
Traversal Layers
- 1. Physical Wire & PHY
- 2. NIC MAC / DMA Engine
- 3. Rx Ring Buffer & RSS
- 4. Interrupt → CPU Pickup
- 5. netisr — Protocol Dispatch
- 6. IP / TCP / UDP Stack
- 7. pf / IPFW Firewall
- 8. Socket Buffer → Userland
End-to-End Pipeline
Everything starts with electrical signals (copper) or light pulses (fiber) hitting the physical port of your E810 NIC. The PHY transceiver is the first component that touches the packet — it converts the analog signal into a digital bitstream.
The PHY checks for valid signal lock (link up/down), performs auto-negotiation for speed and duplex, and feeds the raw bitstream into the MAC layer of the NIC. If there's no link lock, nothing goes further — this is where "cable unplugged" or "no carrier" events originate.
At this point the bitstream is just raw Ethernet frames — the PHY doesn't understand IP, TCP, or anything above L2. It simply ensures the electrical/optical signal is correctly recovered into bits and hands them off.
Once the PHY delivers clean bits, the MAC (Media Access Control) engine — an ASIC on the E810 itself — takes over. This is all happening on the NIC hardware, before the CPU or OS is involved at all.
The MAC engine performs several critical steps in order:
Frame Validation: It checks the Ethernet FCS (Frame Check Sequence / CRC-32). Bad CRC → frame is silently dropped right here on the NIC. You'll see this as ifconfig ice0 → Ierrors. The frame is never DMA'd to host memory.
L2 Parsing: The MAC strips and identifies the EtherType (0x0800 for IPv4, 0x86DD for IPv6, etc.), handles VLAN tag stripping if configured, and parses the Ethernet header to determine the destination MAC for filtering.
Header Parsing for RSS: This is critical for your E810 — the NIC's packet classification pipeline parses deep into the packet to extract the flow key. On the E810, this is done by the Dynamic Device Personalization (DDP) engine, which can parse custom protocols. For standard traffic it extracts: src IP, dst IP, src port, dst port, and protocol number.
Hardware Offloads: Before the packet goes anywhere, the NIC can compute and verify IP/TCP/UDP checksums in hardware (rxcsum), perform TCP segmentation offload (TSO) and Large Receive Offload (LRO), and set hardware timestamp if PTP is configured.
This is where your RSS distribution issues live. After classification, the NIC must decide which Rx queue to place the packet into, and then DMA it into host memory.
RSS (Receive Side Scaling) Hash Computation: The E810 takes the extracted flow tuple (src IP, dst IP, src port, dst port) and feeds it through a Toeplitz hash function using a configurable 40-byte secret key. The output is a 32-bit hash value. The lower N bits of this hash are used as an index into the Indirection Table (RETA), which maps hash values → Rx queue numbers.
The Indirection Table (RETA): On the E810 this is a 512-entry table. Each entry holds a queue index. When a packet hashes to bucket N, the NIC looks up RETA[N] to get the target queue. You can inspect and modify this:
DMA Transfer: Once the target queue is selected, the NIC's DMA engine writes the packet data directly into pre-allocated mbuf memory in host RAM using the Rx descriptor ring for that queue. The ring is a circular buffer of descriptors — each descriptor points to an mbuf where the NIC can write packet data.
The driver (ice(4)) pre-allocates these mbufs and programs the ring with their physical addresses. The NIC writes the packet, then updates the descriptor with metadata (packet length, RSS hash, checksum status, VLAN info) and advances the write pointer.
Key point: each Rx queue is bound to a specific CPU core via MSI-X interrupt vector. This is the essence of RSS — by hashing flows to queues, and pinning queues to CPUs, you get parallel packet processing without lock contention. But if the hash distribution is skewed, one core gets hammered while others sit idle.
dev.ice.0.rx_ring_sizeiqdrops at the interface level. Packet lost before the OS ever sees it.The packet is now sitting in host memory inside the Rx ring buffer. Now we need to tell the CPU about it. This happens via MSI-X interrupts (Message Signaled Interrupts — Extended), where each Rx queue has its own dedicated interrupt vector mapped to a specific CPU core.
The Interrupt Sequence:
1. The NIC fires the MSI-X interrupt for the queue that received the packet. On E810, each queue gets its own vector — no sharing unless you configure it that way.
2. The CPU core receives the interrupt and runs the interrupt handler (ithread in FreeBSD). This is a very short, hard-interrupt context — it does minimal work.
3. The handler acknowledges the interrupt and immediately schedules a taskqueue (or calls into the driver's Rx processing path). On the ice(4) driver, this uses iflib which implements a polling/interrupt hybrid model.
Interrupt Coalescing: The E810 supports ITR (Interrupt Throttle Rate) — instead of firing one interrupt per packet, the NIC waits to accumulate a batch or waits a small time interval, then fires once. This is critical for high-throughput scenarios. You can tune this:
iflib Polling Model: FreeBSD's iflib framework (which the ice driver uses) implements a hybrid interrupt/polling approach. On the first interrupt, it disables further interrupts for that queue and enters a polling loop that drains packets from the ring. If the ring is empty, it re-enables interrupts. This amortizes interrupt overhead under load.
After the driver extracts the mbuf from the ring and fills in the metadata, it calls ether_input() which is the entry point into the FreeBSD network stack proper. This is the boundary between "NIC driver land" and "kernel network stack land."
dev.cpu.*.cx_lowest=C1), and whether the CPU was busy with something else. For your validator, you want minimal coalescing and shallow C-states.This is where your qdrops happen. netisr is FreeBSD's network interrupt service routine framework — it's the dispatch layer that routes packets from the Ethernet input path to the correct protocol handler (IPv4, IPv6, ARP, etc.).
How ether_input() reaches netisr: When the driver calls ether_input(), the Ethernet header is parsed, the EtherType is examined, and the packet is queued into the appropriate netisr protocol queue. For IPv4 traffic, this goes to the ip netisr. For IPv6, the ip6 netisr. For ARP, the arp netisr.
The netisr Queue: Each protocol handler has a per-CPU queue with a configurable maximum depth. When a packet is dispatched to netisr, it's placed into this queue. If the queue is full, the packet is dropped — this is a qdrop.
Dispatch Policies: netisr supports different dispatch modes that determine which CPU's queue a packet goes to:
net.isr.dispatch=directmbuf→m_pkthdr.flowid (the RSS hash from the NIC) to pick the target CPU. This maintains flow affinity — same flow, same CPU.net.isr.defaultqlimit reached), the packet is dropped. This shows up in netstat -Q as drops. Common causes: (1) queue limit too low for burst traffic, (2) CPU can't drain the queue fast enough (protocol processing is too slow), (3) RSS imbalance funnels too many packets to one CPU's netisr queue.When net.isr.dispatch=direct, the netisr queue is bypassed entirely — ip_input() runs directly in the driver's context. This eliminates qdrops at this layer but means the driver holds the CPU longer (can't process more packets from the ring until IP processing completes). For latency-sensitive workloads like your validator, direct dispatch is often better — but monitor that your driver ring doesn't overflow instead.
Once netisr dispatches the packet, we enter the actual network protocol stack. This is where L3 (IP) and L4 (TCP/UDP) processing happens.
ip_input() — Layer 3:
The function validates the IP header: version field, header length, total length, and header checksum (unless the NIC already verified it in hardware — which the E810 does, flagged in the mbuf). It then checks if the packet is destined for a local address or needs to be forwarded. For a firewall doing routing/forwarding, both paths matter.
This is also where pf/IPFW hook in — the firewall is called via the pfil (packet filter) framework as a hook inside ip_input() and ip_output(). More on that in the next section.
After firewall processing (assuming the packet is allowed), the IP layer strips the IP header and looks at the protocol field to determine the L4 handler: protocol 6 → TCP, protocol 17 → UDP, protocol 1 → ICMP.
TCP Processing (tcp_input()): For TCP, the kernel performs connection lookup in the TCP control block table (hash-based), validates sequence numbers, manages the TCP state machine (SYN, ESTABLISHED, FIN-WAIT, etc.), processes window scaling and SACK options, computes and updates RTT estimates, handles ACK processing and triggers output if needed, and deposits data into the socket receive buffer.
UDP Processing (udp_input()): Much simpler — UDP is stateless. The kernel does a socket lookup based on (dst IP, dst port), verifies the UDP checksum if present (E810 did this in HW), and deposits the datagram into the socket receive buffer. If no socket is listening → ICMP port unreachable (unless suppressed).
The firewall doesn't exist as a separate "box" the packet passes through — it's a hook inside the IP processing path. FreeBSD uses the pfil (packet filter interface layer) framework to allow multiple packet filters to register hooks at specific points in the stack.
pfil Hook Points:
ip_input() after basic IP validation but before any protocol processing. This is where inbound rules apply.ip_output() just before the packet is sent to the interface. Outbound rules apply here.ip_forward() for routed/forwarded packets. Separate from in/out so you can have different rules for forwarded traffic.pf Evaluation (if using pf): When the pfil hook fires, pf's pf_test() function is called. It evaluates the packet against the loaded ruleset. pf maintains a state table — for stateful rules (keep state), pf first checks if the packet matches an existing state entry (hash lookup). If it matches a state, the rules aren't re-evaluated — the cached verdict applies. This is the fast path.
If there's no state match, pf walks the rule list sequentially (last match wins). For each rule it checks: direction (in/out), interface, protocol, source/destination addresses, ports, flags, and any other qualifiers. If the packet matches a pass ... keep state rule, a new state entry is created for the flow.
IPFW Evaluation (if using IPFW): IPFW uses a first-match-wins model (opposite of pf). Rules are evaluated in numbered order. The first matching rule determines the action. IPFW also supports check-state / keep-state for dynamic rules.
pfctl -s info to check state table size and search rates.NAT happens here too: If you have NAT rules in pf, address translation is performed at the pfil hook point. For inbound NAT (rdr), the destination is rewritten before further processing. For outbound NAT (nat), the source is rewritten on the way out.
After the packet passes the firewall and protocol processing is complete, the payload data lands in a socket receive buffer (so_rcv). This is the final kernel-side queue before userland gets the data.
Socket Buffer: Each socket has a send buffer (so_snd) and a receive buffer (so_rcv). These are sized by kern.ipc.maxsockbuf and per-socket options (SO_RCVBUF). The socket buffer holds mbuf chains until the application calls recv() / read() / recvfrom().
If the socket buffer is full and the application isn't reading fast enough: for TCP, the kernel stops advertising window space → sender slows down (backpressure). For UDP, new datagrams are silently dropped — this is a major source of UDP packet loss at the application layer and shows up in netstat -s -p udp as "dropped due to full socket buffers."
Waking Up the Application: When data arrives in the socket buffer, the kernel checks if any thread is blocked in recv() or select() / poll() / kqueue() on this socket. If so, the thread is made runnable. For kqueue (which your Solana stack likely uses), a EVFILT_READ event is posted.
And that's the complete journey. The packet has traveled from photons/electrons on the wire, through NIC hardware classification, DMA into host memory, CPU interrupt handling, netisr dispatch, IP/TCP/UDP processing, firewall evaluation, and finally into the application's hands.
netstat -Q and netstat -s -p udp regularly.