Networking Stack Internals

The sk_buff (socket buffer, always called "skb" in code) is the central data structure of the entire Linux networking stack. Every packet that passes through the kernel — received from hardware, being built by a socket, or being forwarded — lives inside an skb. Understanding it deeply is the single most important thing you can do before reading any networking code.

The genius of skb design is that it separates the metadata about a packet (headers, tail pointers, protocol information, timestamps, checksums) from the actual packet bytes. The actual bytes live in a separate contiguous memory region called the data buffer (or linear data area). The skb structure itself is a large C struct that points into this buffer using four key pointers:

• head: the very beginning of the allocated data buffer (never moves)
• data: the start of the current valid packet data (moves forward as headers are consumed)
• tail: the end of the current valid packet data (moves forward as data is appended)
• end: the very end of the allocated data buffer (never moves)

Think of the buffer as a hotel hallway. head and end are the walls. data is the front door (moves in as the concierge strips your coat — i.e., as headers are parsed and consumed). tail is the back wall of your room (moves out as you add more luggage — i.e., as payload is appended).

This layout enables zero-copy header manipulation. To prepend an IP header onto a TCP segment, the stack does not copy the segment — it simply moves data backward by sizeof(struct iphdr) bytes and writes the IP header into that headroom. This is the skb_push() operation. Conversely, to strip the Ethernet header when passing up from the link layer to the network layer, the stack calls skb_pull() which moves data forward — no copy, just pointer arithmetic.

For large packets (like those built from sendfile() or zero-copy sends), the linear area may not hold everything. The skb has a skb_shared_info structure appended just past end, which contains an array of skb_frag_t fragments — pointers into page cache pages. This is the fragmented skb or paged data model, and it is how the kernel implements scatter-gather I/O without copying megabytes of data.

The skb also carries a rich set of metadata fields: sk (the owning socket, or NULL for forwarded packets), dev (the net_device it arrived on or will be sent via), protocol (the EtherType), ip_summed (checksum offload status), tstamp (hardware or software timestamp), mark (for netfilter/iptables marking), priority (for QoS), and many more. Reading these fields is how every layer of the stack makes decisions without re-parsing lower-layer headers it has already consumed.

/* Simplified sk_buff layout (from include/linux/skbuff.h) */
struct sk_buff {
    /* -- These four pointers define the data area -- */
    unsigned char   *head;   /* start of allocated buffer        */
    unsigned char   *data;   /* start of packet data (moves up)  */
    unsigned char   *tail;   /* end   of packet data (moves up)  */
    unsigned char   *end;    /* end   of allocated buffer        */

    /* -- Ownership / routing -- */
    struct sock     *sk;     /* owning socket (NULL if forwarded) */
    struct net_device *dev;  /* device packet arrived on / sent via */

    /* -- Protocol info -- */
    __be16          protocol;   /* EtherType: ETH_P_IP, ETH_P_IPV6 … */
    __u16           transport_header; /* offset from head to L4 hdr  */
    __u16           network_header;   /* offset from head to L3 hdr  */
    __u16           mac_header;       /* offset from head to L2 hdr  */

    /* -- Checksum offload -- */
    __u8            ip_summed;  /* CHECKSUM_NONE/PARTIAL/COMPLETE  */

    /* skb_shared_info lives just past 'end' and holds page frags */
};

/* Prepend 20-byte IPv4 header — no memory copy! */
struct iphdr *iph = (struct iphdr *)skb_push(skb, sizeof(*iph));
/* skb->data now points 20 bytes earlier; iph points to that space */

/* Strip 14-byte Ethernet header — again, no copy */
skb_pull(skb, ETH_HLEN);   /* skb->data += 14 */

/* Access the current network-layer header */
struct iphdr *iph2 = ip_hdr(skb); /* returns skb->head + skb->network_header */

The Linux kernel must support thousands of different network cards, from ancient ISA cards to modern 400 Gbit NICs, from Wi-Fi chips to virtual tunnel interfaces. It achieves this via struct net_device — the universal network device abstraction. Every network interface in the system (physical or virtual) is represented by exactly one net_device.

A net_device is to a network card what a struct inode is to a file: it is the kernel's canonical object representing that resource. When you run ip link show, you are listing all net_device objects in the current network namespace.

The net_device contains:

1. Identity: name (eth0, lo), MAC address, MTU, flags (IFF_UP, IFF_RUNNING, etc.)
2. Statistics: rx_packets, tx_bytes, etc. — what ip -s link shows
3. Operations table (net_device_ops): a struct of function pointers — the driver's interface to the kernel. The most important are:
   • ndo_open / ndo_stop: bring the interface up/down
   • ndo_start_xmit: transmit a packet (takes an skb, returns NETDEV_TX_OK or NETDEV_TX_BUSY)
   • ndo_get_stats64: fetch statistics
   • ndo_set_mac_address, ndo_change_mtu, etc.
4. NAPI structures: struct napi_struct instances for each receive queue (explained in the NAPI section)
5. Queue state: TX and RX queue arrays (struct netdev_queue, struct netdev_rx_queue)
6. Network namespace pointer: struct net *nd_net — which namespace this device belongs to

The separation of net_device_ops function pointers from the net_device struct is the classic strategy pattern: the kernel core calls dev->netdev_ops->ndo_start_xmit(skb, dev) without knowing anything about the hardware. The driver registers these pointers at probe time. This is how the same ip command works for Intel NICs, Realtek NICs, Wi-Fi chips, and VXLAN tunnels.

When a driver is loaded, it calls alloc_etherdev() (for Ethernet) or alloc_netdev() (generic) to allocate a net_device plus private driver data in one allocation, then fills in netdev_ops, sets up NAPI, and calls register_netdev(). From that point, the kernel knows about the device.

/* Minimal Ethernet driver skeleton */
#include <linux/netdevice.h>
#include <linux/etherdevice.h>

struct my_priv {
    /* hardware-specific state */
    void __iomem *regs;
};

static netdev_tx_t my_xmit(struct sk_buff *skb, struct net_device *dev)
{
    struct my_priv *priv = netdev_priv(dev);
    /* hand skb->data / skb->len to hardware DMA ring */
    /* ... */
    dev_kfree_skb(skb); /* or defer until TX completion interrupt */
    return NETDEV_TX_OK;
}

static int my_open(struct net_device *dev)
{
    netif_start_queue(dev); /* allow ndo_start_xmit to be called */
    return 0;
}

static const struct net_device_ops my_ops = {
    .ndo_open        = my_open,
    .ndo_start_xmit  = my_xmit,
};

static int my_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
    struct net_device *dev = alloc_etherdev(sizeof(struct my_priv));
    dev->netdev_ops = &my_ops;
    /* set MAC, IRQ, etc. */
    return register_netdev(dev); /* now visible as eth0, eth1, … */
}

When a packet arrives, the NIC raises a hardware interrupt. The CPU drops whatever it was doing, saves state, and jumps to the interrupt handler registered by the driver. This is fine for occasional packets. But at 10 Gbps, packets arrive every 67 nanoseconds. If the CPU spent all its time taking hardware interrupts, it would never get any other work done — a problem called interrupt livelock.

The Linux solution is two-phase interrupt processing:

Phase 1 — Hard IRQ (minimal, fast): The hardware interrupt handler does the absolute minimum: acknowledges the interrupt to the hardware, schedules deferred work, and returns. Specifically for networking, the driver calls napi_schedule() which places its NAPI structure on a per-CPU poll list and raises a NET_RX_SOFTIRQ.

Phase 2 — Softirq (deferred, still in interrupt context but preemptible by hard IRQs): Softirqs are a lightweight deferred execution mechanism. There are only 10 softirq types in the kernel (HI, TIMER, NET_TX, NET_RX, BLOCK, IRQ_POLL, TASKLET, SCHED, HRTIMER, RCU). They run after the hard IRQ handler returns, still in "interrupt context" (no sleeping, no user access), but hard IRQs are re-enabled so the CPU can still respond to new hardware events.

The NET_RX_SOFTIRQ handler is net_rx_action() in net/core/dev.c. It iterates over the per-CPU NAPI poll list and calls each registered poll() function, which is the driver's receive function. This is the heart of NAPI.

Why not just use a kernel thread? Threads have scheduling latency. Softirqs run immediately when hard IRQs finish, with no scheduler involvement, giving sub-microsecond latency from hardware interrupt to packet processing. For network performance, this matters enormously.

Why not just do everything in the hard IRQ? Hard IRQ handlers run with interrupts disabled on that CPU. If you spend 10ms processing packets in a hard IRQ, you miss 10ms worth of timer ticks, keyboard input, and other interrupts. The split model gives you both: fast response (raised by hard IRQ) and non-blocking processing (done in softirq).

Softirqs are strictly per-CPU. Each CPU has its own softirq pending bitmask and processes its own softirq backlog. This is why networking is naturally CPU-parallel: each receive queue can be handled by a different CPU with no locking.

/* Simplified flow: hard IRQ → softirq → packet processing */

/* Step 1: Driver's hard IRQ handler (runs with IRQs disabled on this CPU) */
static irqreturn_t my_irq_handler(int irq, void *dev_id)
{
    struct net_device *dev = dev_id;
    struct my_napi *napi = &my_priv(dev)->napi;

    /* Disable NIC interrupts so we don't keep re-entering */
    my_hw_disable_rx_irq(dev);

    /* Schedule NAPI poll — sets NET_RX_SOFTIRQ pending */
    napi_schedule(napi);

    return IRQ_HANDLED;
}

/* Step 2: NET_RX_SOFTIRQ → net_rx_action() calls our poll function */
static int my_napi_poll(struct napi_struct *napi, int budget)
{
    int received = 0;

    while (received < budget) {
        struct sk_buff *skb = my_hw_receive_one_packet(napi);
        if (!skb) break;

        /* Hand packet up the stack */
        netif_receive_skb(skb);
        received++;
    }

    if (received < budget) {
        /* No more packets — re-enable IRQ and stop polling */
        napi_complete_done(napi, received);
        my_hw_enable_rx_irq(napi);
    }
    return received;
}

NAPI (New API, introduced in Linux 2.6, backported to late 2.4) is the standard receive model for all modern network drivers. It solves the interrupt livelock problem through interrupt coalescing via polling: instead of taking one interrupt per packet, the driver takes one interrupt to signal "packets are arriving", then polls the hardware ring buffer until it is empty or a budget is exhausted.

The central object is struct napi_struct. Each receive queue on a NIC should have its own napi_struct (modern NICs have 8, 16, or more queues). The struct contains:

• poll: the driver's function pointer, called by net_rx_action()
• weight: the maximum packets to process per poll call (default 64)
• state: bitmask (NAPI_STATE_SCHED, NAPI_STATE_DISABLE, etc.)
• dev_list: node in the per-CPU poll list

The NAPI State Machine:

1. Idle: NIC interrupts are enabled. Hardware raises an IRQ when a packet arrives.
2. Scheduled: The IRQ handler calls napi_schedule(). This adds the napi_struct to the current CPU's poll list, raises NET_RX_SOFTIRQ, and disables further NIC interrupts for this queue. More packets can arrive in the DMA ring without causing more interrupts.
3. Polling: net_rx_action() calls poll() with a budget (typically 64). The driver processes up to budget packets, calling netif_receive_skb() for each.
4. Complete: If the driver processes fewer than budget packets (queue is empty), it calls napi_complete_done(), which removes the napi_struct from the poll list and re-enables NIC interrupts. If it hit the budget, the softirq reschedules it for the next softirq run — preventing starvation of other tasks.

The budget system is a fairness mechanism. If one NIC is receiving a flood of traffic, it should not monopolize the CPU's softirq time. net_rx_action() tracks the total budget across all NAPI instances per softirq invocation (default: 300 packets). If a NAPI instance uses its full per-poll weight, it goes back to the tail of the list. This ensures other NAPI instances and other softirqs get CPU time.

GRO (Generic Receive Offload) is the receive counterpart of GSO. In the NAPI poll loop, before calling netif_receive_skb(), drivers can call napi_gro_receive() instead. GRO coalesces multiple small TCP segments belonging to the same flow into a single large skb before passing up the stack, dramatically reducing the per-packet overhead of protocol processing.

Multi-queue and RSS (Receive Side Scaling): Modern NICs hash incoming flows (by src/dst IP and port) to different hardware receive queues. Each queue has its own NAPI instance, and the kernel assigns each NAPI instance to a specific CPU via IRQ affinity. This allows true parallel packet processing without any locking — a 16-queue NIC can use 16 CPUs simultaneously.

/* Setting up NAPI in a driver (probe or open path) */

/* 1. Initialize the napi_struct */
netif_napi_add(dev,           /* net_device */
               &priv->napi,  /* napi_struct to init */
               my_napi_poll, /* poll function */
               64);           /* weight (max pkts per poll) */

/* 2. Enable it when interface comes up */
napi_enable(&priv->napi);

/* 3. In napi_poll — the budget loop */
static int my_napi_poll(struct napi_struct *napi, int budget)
{
    struct my_priv *priv = container_of(napi, struct my_priv, napi);
    int work = 0;

    while (work < budget) {
        struct sk_buff *skb = my_rx_ring_get_next(priv);
        if (!skb)
            break;

        skb->dev      = priv->dev;
        skb->protocol = eth_type_trans(skb, priv->dev); /* sets mac_header, protocol */

        /* GRO path: coalesce small segments before handing up */
        napi_gro_receive(napi, skb);
        work++;
    }

    /* If we drained the queue, stop polling and re-arm IRQ */
    if (work < budget)
        napi_complete_done(napi, work);

    return work;
}

/* 4. Disable when interface goes down */
napi_disable(&priv->napi);

Once the NAPI poll function has an skb ready, it calls netif_receive_skb(). This function is the ingress entry point of the kernel networking stack — the moment a packet transitions from driver territory to core stack territory.

netif_receive_skb() does several important things in sequence:

1. RX timestamp: If the device or software timestamping is enabled, record when the packet was received.

2. tc_classify / ingress qdisc: If TC (Traffic Control) ingress filters are configured (via tc filter add dev eth0 ingress), the packet is classified here. eBPF programs attached via tc are invoked at this point.

3. XDP (eXpress Data Path) — actually earlier: XDP programs run even before netif_receive_skb, directly in the NAPI poll loop, on the raw DMA buffer before an skb is even allocated. This is why XDP can achieve line-rate packet processing. By the time we reach netif_receive_skb, any XDP_DROP or XDP_TX decisions have already been made.

4. Protocol demultiplexing: The function calls __netif_receive_skb_core(), which looks up the skb->protocol (the EtherType set by eth_type_trans()) in the ptype_base hash table. This table maps EtherTypes to registered protocol handlers (struct packet_type). For ETH_P_IP (0x0800), it finds the IPv4 handler. For ETH_P_ARP (0x0806), the ARP handler. For ETH_P_IPV6 (0x86DD), the IPv6 handler. AF_PACKET sockets (used by tcpdump) register a packet_type for ETH_P_ALL and see a copy of every packet here.

5. Bridging and bonding: If the incoming device is a bridge port or bond slave, bridge/bonding code gets the packet first and may forward or consume it without ever reaching the IP stack.

The packet_type lookup and dispatch is conceptually a switch on EtherType, but implemented as a hash table for O(1) lookup with many protocols registered. The handler function pointer func is called with (skb, dev, pt, orig_dev). For IPv4, func is ip_rcv().

/* How protocol handlers are registered (from net/ipv4/af_inet.c) */
static struct packet_type ip_packet_type __read_mostly = {
    .type = cpu_to_be16(ETH_P_IP),
    .func = ip_rcv,           /* called for every IPv4 packet    */
    .list_func = ip_list_rcv, /* called for batched packets (GRO) */
};

static int __init inet_init(void)
{
    /* ... */
    dev_add_pack(&ip_packet_type); /* register into ptype_base hash */
    /* ... */
}

/* Registering a raw AF_PACKET socket (tcpdump uses this) */
static struct packet_type tcpdump_pt = {
    .type = htons(ETH_P_ALL), /* receive EVERY EtherType         */
    .func = packet_rcv,       /* deliver copy to AF_PACKET socket */
    .dev  = NULL,             /* NULL = all devices               */
};
dev_add_pack(&tcpdump_pt);

/* The dispatch loop in __netif_receive_skb_core (conceptually) */
type = skb->protocol;                     /* e.g. ETH_P_IP = 0x0800  */
pt = ptype_base[ntohs(type) & PTYPE_HASH_MASK]; /* O(1) hash lookup */
pt->func(skb, skb->dev, pt, skb->dev);   /* call ip_rcv() etc.      */

Once ip_rcv() is called with an Ethernet-stripped skb, the IPv4 layer begins its processing. The network layer's job is to make routing decisions: should this packet be delivered locally (to a socket on this machine) or forwarded to another host?

ip_rcv() — sanity checking: Before doing anything useful, ip_rcv() validates the packet:

• Check that skb->len >= sizeof(struct iphdr) (at least 20 bytes)
• Verify IP header checksum (unless hardware already did it via ip_summed == CHECKSUM_COMPLETE)
• Verify IP version == 4 and header length is valid
• Check tot_len does not exceed actual skb length
• Trim the skb if it is longer than tot_len (Ethernet may pad short packets)

If any check fails, the packet is dropped and statistics counters incremented.

Netfilter NF_INET_PRE_ROUTING hook: After validation, ip_rcv() calls NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, ...). This is where iptables PREROUTING rules and conntrack (connection tracking) run. iptables DNAT rules (port forwarding) apply here. If the packet is accepted, processing continues.

ip_rcv_finish() and routing: ip_rcv_finish() calls ip_route_input_noref() to perform a FIB (Forwarding Information Base) lookup — essentially a longest-prefix-match lookup of the destination IP in the routing table. The result is stored in skb_dst(skb). The routing result is either:

• RTN_LOCAL: deliver to a local socket → call ip_local_deliver()
• RTN_UNICAST: forward to another host → call ip_forward()
• RTN_MULTICAST, RTN_BROADCAST: multicast/broadcast handling

ip_local_deliver(): For packets destined to this machine, ip_local_deliver() first handles IP fragmentation reassembly (using ip_defrag()), then hits the NF_INET_LOCAL_IN hook (iptables INPUT chain). After that, it calls ip_local_deliver_finish() which does the L4 demultiplexing: it looks up the IP protocol field (TCP=6, UDP=17, ICMP=1) in the inet_protos[] array and calls the registered L4 handler.

/* The IPv4 receive path — simplified call chain */

/* net/ipv4/ip_input.c */
int ip_rcv(struct sk_buff *skb, struct net_device *dev,
           struct packet_type *pt, struct net_device *orig_dev)
{
    struct iphdr *iph;

    /* Ensure we have a writable header */
    if (!pskb_may_pull(skb, sizeof(struct iphdr)))
        goto drop;

    iph = ip_hdr(skb);   /* skb->head + skb->network_header */

    if (iph->ihl < 5 || iph->version != 4)
        goto drop;

    if (!pskb_may_pull(skb, iph->ihl * 4))
        goto drop;

    if (ip_fast_csum((u8 *)iph, iph->ihl))
        goto csum_error; /* checksum mismatch */

    /* NF_INET_PRE_ROUTING: iptables PREROUTING, DNAT, conntrack */
    return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING,
                   net, NULL, skb, dev, NULL,
                   ip_rcv_finish);  /* continuation after hook */
drop:
    kfree_skb(skb);
    return NET_RX_DROP;
}

/* After routing lookup determines packet is for us */
static int ip_local_deliver_finish(struct net *net, struct sock *sk,
                                   struct sk_buff *skb)
{
    __skb_pull(skb, skb_network_header_len(skb)); /* strip IP header */

    int protocol = ip_hdr(skb)->protocol; /* TCP=6, UDP=17, ... */
    const struct net_protocol *ipprot = inet_protos[protocol];

    if (ipprot)
        return ipprot->handler(skb); /* tcp_v4_rcv() or udp_rcv() */
    /* else: send ICMP protocol unreachable */
}

The L4 handler for TCP is tcp_v4_rcv(), registered in net/ipv4/tcp_ipv4.c. This is where TCP protocol logic takes over: sequence number tracking, acknowledgment generation, flow control, and ultimately delivering data to the application's socket buffer.

Socket lookup: The first thing tcp_v4_rcv() does is find the struct sock (socket) that owns this connection, using a 4-tuple lookup: (src_ip, src_port, dst_ip, dst_port). This is done via __inet_lookup_skb() which searches the established connection hash table first (O(1) average case), then the listening socket table if no match. If no socket is found, the packet gets a TCP RST or is dropped silently.

TCP state machine: Depending on the socket state (ESTABLISHED, SYN_SENT, SYN_RECV, TIME_WAIT, LISTEN, etc.), different processing functions are called. For an established connection receiving data, tcp_rcv_established() is the fast path.

Sequence number validation: The TCP header's sequence number is checked against tp->rcv_nxt (the next expected byte). Out-of-order segments are placed into tp->out_of_order_queue (an RB-tree sorted by sequence number) for later reassembly. In-order segments can be delivered immediately.

Checksum verification: If hardware didn't compute the TCP checksum (ip_summed != CHECKSUM_COMPLETE), tcp_checksum_complete() computes it in software.

Data delivery to socket receive buffer: For in-order data, tcp_rcv_established() calls tcp_data_queue() which calls skb_set_owner_r() to associate the skb with the socket and account for its memory against the socket's receive buffer (sk->sk_rcvbuf). The skb is placed on sk->sk_receive_queue.

Sending an ACK: __tcp_ack_snd_check() decides whether to send an immediate ACK or use delayed ACK (wait 40ms to potentially piggyback the ACK on outgoing data). The delayed ACK timer is sk->sk_timer.

Waking the application: After queuing data, sk->sk_data_ready(sk) is called (usually sock_def_readable()). This wakes up any process blocked in read() / recv() / epoll_wait() on this socket. The process then calls tcp_recvmsg() which copies data from sk_receive_queue into userspace — or for zero-copy receives, maps the skb pages directly.

This entire path — from tcp_v4_rcv() to waking the blocked process — typically completes in under 5 microseconds on modern hardware.

/* Highly simplified tcp_v4_rcv flow */

int tcp_v4_rcv(struct sk_buff *skb)
{
    const struct iphdr *iph = ip_hdr(skb);
    const struct tcphdr *th;
    struct sock *sk;

    /* Pull TCP header into linear area */
    if (!pskb_may_pull(skb, sizeof(struct tcphdr)))
        goto discard;

    th = tcp_hdr(skb);

    /* 4-tuple socket lookup: (sip, sport, dip, dport) */
    sk = __inet_lookup_skb(&tcp_hashinfo, skb,
                           __tcp_hdrlen(th),
                           th->source, th->dest,
                           sdif, &refcounted);
    if (!sk)
        goto no_tcp_socket; /* send RST */

    /* Hand off to per-state handler */
    if (sk->sk_state == TCP_ESTABLISHED)
        return tcp_rcv_established(sk, skb);
    else
        return tcp_rcv_state_process(sk, skb);

no_tcp_socket:
    tcp_v4_send_reset(NULL, skb);
discard:
    kfree_skb(skb);
    return 0;
}

/* Inside tcp_rcv_established — the fast receive path */
/* After sequence/checksum checks, in-order data is queued: */
static void tcp_data_queue(struct sock *sk, struct sk_buff *skb)
{
    struct tcp_sock *tp = tcp_sk(sk);

    /* In-order: add to receive queue */
    if (TCP_SKB_CB(skb)->seq == tp->rcv_nxt) {
        __skb_queue_tail(&sk->sk_receive_queue, skb);
        tp->rcv_nxt += skb->len;  /* advance expected sequence */
        sk->sk_data_ready(sk);    /* wake blocked read() calls  */
    }
    /* Out-of-order: insert into RB-tree for later reassembly */
}

The transmit path is the mirror of the receive path, but with more moving parts because the kernel must handle flow control, congestion control, segmentation, and queuing disciplines (qdiscs). Here is the journey of data from a send() call to the physical wire.

1. System call entry: send(fd, buf, len, 0) enters the kernel as sys_sendto() → sock_sendmsg() → for TCP: tcp_sendmsg(). This function copies data from userspace into skbs using sk_stream_alloc_skb() and adds them to the socket's send queue (sk->sk_write_queue). Each skb is limited to MSS (Maximum Segment Size) — the maximum data a single TCP segment can carry.

2. TCP layer — congestion and flow control: tcp_push() → tcp_write_xmit() applies TCP's congestion window (tp->snd_cwnd) and receiver window (tp->snd_wnd) to decide how many skbs can actually be sent. Skbs that cannot be sent yet stay in sk_write_queue until window opens. Sent skbs are copied to sk_send_head (the retransmit queue).

3. GSO (Generic Segmentation Offload): For large writes, TCP may build a single large skb (up to 64 KB, a "super-segment" or TSO skb). If the NIC supports TSO (TCP Segmentation Offload), this single skb is handed to the hardware which splits it into MTU-sized segments itself. If TSO is not available, the kernel's GSO layer (net/core/gso.c) does the splitting in software just before transmission. GSO/TSO dramatically reduces the number of descriptor ring entries and per-packet processing overhead.

4. IP layer: ip_queue_xmit() adds the IP header, performs the routing lookup to find the output device and next-hop, applies IP options, and calls ip_local_out() which hits NF_INET_LOCAL_OUT (iptables OUTPUT chain). Then ip_output() → ip_finish_output() handles fragmentation if the packet exceeds the path MTU.

5. Traffic Control (qdisc): dev_queue_xmit() enqueues the skb into the device's qdisc (queuing discipline). The default qdisc is pfifo_fast (a simple priority FIFO). More sophisticated qdiscs like fq_codel (Fair Queuing with CoDel) or htb (Hierarchical Token Bucket for bandwidth shaping) can be plugged in via tc. The qdisc is the kernel's traffic shaping and scheduling mechanism.

6. Driver and hardware: When the qdisc dequeues the skb, it calls dev->netdev_ops->ndo_start_xmit(skb, dev). The driver places the skb into the NIC's TX descriptor ring, writes the ring tail register to tell the hardware about new work, and returns. When transmission completes, the NIC raises a TX completion interrupt, the driver calls dev_kfree_skb() on the completed skbs and, if the TX ring was full (netif_stop_queue was called), restarts the queue with netif_wake_queue().

/* Transmit path call chain (grep-friendly function names) */

/*
 * Userspace:  send(fd, buf, len, 0)
 *               │
 * Syscall:    sock_sendmsg()
 *               │
 * TCP:        tcp_sendmsg()           copy data into skbs
 *               └─ tcp_push()
 *                   └─ tcp_write_xmit()   apply cwnd/rwnd limits
 *                       └─ tcp_transmit_skb()
 *                           │
 * IP:             ip_queue_xmit()     add IP header, route lookup
 *                   └─ ip_local_out()
 *                       └─ NF_HOOK(NF_INET_LOCAL_OUT)  [iptables OUTPUT]
 *                           └─ ip_output()
 *                               └─ ip_finish_output()
 *                                   └─ ip_finish_output2()
 *                                       └─ neigh_output()  [ARP resolution]
 *                                           └─ dev_queue_xmit()
 *                                               │
 * Qdisc:          sch_direct_xmit()   dequeue from qdisc
 *                   │
 * Driver:      ndo_start_xmit()       put in TX ring, ring doorbell
 */

/* The crucial point: skbs in sk_write_queue are cloned into
 * the retransmit queue. The clone is freed on TX ACK.
 * If the NIC ring is full, ndo_start_xmit returns NETDEV_TX_BUSY,
 * the qdisc holds the skb, and netif_stop_queue() prevents new calls. */

Netfilter is the kernel's packet filtering and manipulation framework. It defines hook points at strategic locations in the packet path where external code can inspect and modify packets. Understanding netfilter is essential for anyone working on firewalls, NAT, connection tracking, or eBPF tc/nftables programs.

There are five standard IPv4 netfilter hook points:

1. NF_INET_PRE_ROUTING: Very first, before routing decision. Used for DNAT (redirect incoming packets to different IP/port).
2. NF_INET_LOCAL_IN: After routing, for packets going to a local socket. Used for INPUT firewall rules.
3. NF_INET_FORWARD: For packets being forwarded (not local). Used for FORWARD rules.
4. NF_INET_LOCAL_OUT: Locally generated packets, after they leave the socket but before routing. Used for OUTPUT rules.
5. NF_INET_POST_ROUTING: Just before a packet goes to the wire. Used for SNAT/MASQUERADE.

Each hook is implemented with NF_HOOK() macros in the packet path (as seen in ip_rcv() and ip_output()). The macro calls nf_hook() which iterates over all registered hook functions for that (protocol family, hook number) pair in order of priority.

iptables is built on top of netfilter. The iptables tables (filter, nat, mangle, raw) each register callback functions at specific hooks. xt_tables (the extended tables framework) handles matching (via match modules like -m tcp, -m conntrack) and targeting (via target modules like -j ACCEPT, -j DNAT, -j MARK).

nftables is the modern replacement, with a more flexible bytecode-based evaluation engine (nft_do_chain()) that replaces the per-table linear rule scanning with a more efficient structure.

conntrack (connection tracking) is a separate netfilter module that tracks the state of every connection (NEW, ESTABLISHED, RELATED, INVALID). It runs at NF_INET_PRE_ROUTING and NF_INET_LOCAL_OUT. The conntrack entry (struct nf_conn) is associated with the skb via nfct pointer. Stateful firewalling (-m state --state ESTABLISHED) depends entirely on conntrack.

Hook verdict: Every hook function returns a verdict: NF_ACCEPT (continue), NF_DROP (discard), NF_STOLEN (hook took ownership of skb), NF_QUEUE (send to userspace via nfqueue), or NF_REPEAT. If any function returns NF_DROP, the skb is freed immediately.

/* Registering a custom netfilter hook (kernel module) */
#include <linux/netfilter.h>
#include <linux/netfilter_ipv4.h>

static unsigned int my_hook_fn(void *priv,
                               struct sk_buff *skb,
                               const struct nf_hook_state *state)
{
    struct iphdr *iph = ip_hdr(skb);

    /* Drop all packets to 10.0.0.1 */
    if (iph->daddr == htonl(0x0A000001)) {
        pr_info("dropping packet to 10.0.0.1\n");
        return NF_DROP;
    }
    return NF_ACCEPT;
}

static struct nf_hook_ops my_hook_ops = {
    .hook     = my_hook_fn,
    .pf       = NFPROTO_IPV4,          /* IPv4 packets only         */
    .hooknum  = NF_INET_PRE_ROUTING,   /* hook point                */
    .priority = NF_IP_PRI_FIRST,       /* run before iptables       */
};

static int __init my_init(void)
{
    return nf_register_net_hook(&init_net, &my_hook_ops);
}

static void __exit my_exit(void)
{
    nf_unregister_net_hook(&init_net, &my_hook_ops);
}

XDP (eXpress Data Path) is the highest-performance packet processing framework in the Linux kernel, introduced in Linux 4.8. It allows eBPF programs to process packets at the earliest possible point — directly in the NIC driver's NAPI poll loop, on the raw DMA buffer, before even allocating an sk_buff.

The key insight: sk_buff allocation is expensive (cache-missy memory access, field initialization). For drop-heavy workloads (DDoS mitigation, load balancer fast-path), allocating an skb only to immediately drop it wastes significant CPU. XDP bypasses this entirely.

How it works:

1. An XDP eBPF program is loaded with ip link set dev eth0 xdp obj prog.o. (Or via BPF_PROG_TYPE_XDP + BPF_XDP_LINK APIs.)
2. The driver calls the XDP program for each received frame, passing an xdp_buff struct containing pointers to the raw DMA buffer (data, data_end, data_meta), the receive ring descriptor context, and the net_device.
3. The eBPF program inspects the raw bytes, makes a decision, and returns a verdict:
   • XDP_PASS: hand the frame to the normal stack (an skb will be allocated)
   • XDP_DROP: drop the frame silently — no skb allocation, no stats (unless you instrument them)
   • XDP_TX: retransmit the frame back out the same interface (useful for echo servers, load balancers)
   • XDP_REDIRECT: forward the frame to another interface or CPU (via bpf_redirect()), or to a userspace AF_XDP socket
   • XDP_ABORTED: drop with trace event (debugging)

AF_XDP (kernel 5.2+): A zero-copy path from NIC to userspace. Userspace applications map the NIC's DMA buffers via a shared memory area (UMEM) and communicate with the kernel via lock-free SPSC rings (TX, RX, fill, completion). Throughput rivals DPDK while being integrated into the kernel networking stack.

XDP driver modes:

• Native XDP: program runs inside the driver's NAPI poll loop. Requires driver support. Maximum performance.
• Generic XDP (SKB mode): kernel allocates the skb first, then calls the XDP program. Works with any driver but loses most of the performance benefit. Useful for development/testing.
• Offloaded XDP: some SmartNICs can run the eBPF program in NIC firmware, completely offloading processing from the CPU.

XDP is the foundation of projects like Cloudflare's DDoS mitigation, Facebook's Katran load balancer, Cilium CNI, and many high-performance packet generators.

/* Minimal XDP program — drop all UDP traffic, pass everything else */
/* Compile: clang -O2 -target bpf -c xdp_drop_udp.c -o xdp_drop_udp.o */
/* Load:    ip link set dev eth0 xdp obj xdp_drop_udp.o sec xdp         */

#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/udp.h>
#include <bpf/bpf_helpers.h>

SEC("xdp")
int xdp_drop_udp(struct xdp_md *ctx)
{
    void *data     = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;

    /* Parse Ethernet header */
    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)
        return XDP_PASS;
    if (eth->h_proto != __constant_htons(ETH_P_IP))
        return XDP_PASS;

    /* Parse IP header */
    struct iphdr *iph = (void *)(eth + 1);
    if ((void *)(iph + 1) > data_end)
        return XDP_PASS;

    /* Drop UDP */
    if (iph->protocol == IPPROTO_UDP)
        return XDP_DROP;

    return XDP_PASS;
}

char _license[] SEC("license") = "GPL";

Network namespaces (introduced in Linux 2.6.24) are one of the foundational technologies of modern containerization. A network namespace is an isolated instance of the entire network stack: its own set of network interfaces, routing tables, iptables rules, sockets, and even its own loopback device. Containers (Docker, Kubernetes pods) get their network isolation from network namespaces.

From the kernel's perspective, every net_device and struct sock belongs to a specific struct net (the network namespace object). init_net is the default namespace. System calls like socket() and bind() operate on the namespace of the calling process. The net_device field nd_net tracks which namespace the device belongs to.

Key virtual devices:

veth (virtual Ethernet pair): A pair of connected virtual NICs. Anything sent into one end comes out the other. ip link add veth0 type veth peer name veth1 creates two linked interfaces. When one end is moved into a container's namespace, it becomes that container's network interface. Traffic flows: container → veth1 → veth0 → host bridge/router → internet.

bridge: A virtual L2 switch. Multiple interfaces (real or virtual) can be attached. The bridge learns MAC addresses and forwards frames accordingly. docker0 is a bridge. Kubernetes's cbr0 is a bridge. Implementation in net/bridge/.

tun/tap: Userspace-accessible virtual NICs. A tun device works at L3 (IP packets); a tap device at L2 (Ethernet frames). A userspace program opens /dev/net/tun, creates a tun/tap interface, and then read()/write() calls move packets between kernel and userspace. VPN software (OpenVPN, WireGuard in userspace mode) and QEMU's guest networking use tun/tap.

vxlan/geneve: Overlay network tunnels. Encapsulate L2 Ethernet frames inside UDP packets for transmission over L3 networks. Used extensively in Kubernetes CNI plugins (Flannel uses vxlan; Cilium can use Geneve). Implementation creates a virtual NIC; sending a frame causes it to be encapsulated and forwarded to the remote VTEP (VXLAN Tunnel Endpoint).

loopback: The lo device is entirely in-kernel. Packets sent to 127.0.0.1 go through the full IP stack (routing, netfilter, TCP) but the loopback driver just loops the skb back into netif_receive_skb(). Each network namespace has its own loopback device.

/* Creating a network namespace + veth pair in shell */

# Create a network namespace named 'container'
ip netns add container

# Create a veth pair: veth0 (host) <--> veth1 (container)
ip link add veth0 type veth peer name veth1

# Move veth1 into the container namespace
ip link set veth1 netns container

# Configure IPs
ip addr add 10.0.0.1/24 dev veth0
ip link set veth0 up

ip netns exec container ip addr add 10.0.0.2/24 dev veth1
ip netns exec container ip link set veth1 up
ip netns exec container ip link set lo up

# Ping from host to container
ping 10.0.0.2

# Run a command inside the container namespace
ip netns exec container ss -tuln  # shows sockets isolated from host

# The kernel path: ping sends ICMP to 10.0.0.2
#   → routing table says: via veth0
#   → ndo_start_xmit of veth driver
#   → veth driver calls netif_receive_skb on the PEER (veth1)
#   → packet appears in the container namespace's stack