Linux Networking from Userspace

A socket is the fundamental abstraction for network communication in Unix. The key insight is that it is just a file descriptor — the same integer you get from open(). You can read(), write(), select(), poll(), epoll_wait(), and close() it just like a regular file. This unification is not accidental; it is one of the most deliberate design choices in Unix history.

Under the hood, a socket descriptor points to a struct socket in the kernel, which in turn points to a struct sock (the protocol-independent socket) and a concrete protocol implementation (e.g., struct tcp_sock). The socket lives in the virtual filesystem layer (sockfs), which is why it has an inode and can be polled.

The lifecycle of a TCP connection from the server side demonstrates the full API:

1. socket() — allocate the socket data structures and get an fd
2. bind() — associate a local address and port
3. listen() — mark the socket as passive, create the accept queue
4. accept() — block until a client connects, return a new fd for that connection
5. read()/write() — exchange data
6. close() — tear down the connection (sends TCP FIN)

The client side replaces bind/listen/accept with a single connect(). The connect() call triggers the TCP three-way handshake: SYN → SYN-ACK → ACK. It blocks until the handshake completes (or times out).

The address family (AF_INET for IPv4, AF_INET6 for IPv6, AF_UNIX for local) and socket type (SOCK_STREAM for TCP, SOCK_DGRAM for UDP, SOCK_RAW for raw IP) are chosen at socket() time and shape every subsequent operation.

/* minimal_server.c — a complete TCP echo server */
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>

int main(void) {
    int lfd = socket(AF_INET, SOCK_STREAM, 0);

    /* allow reuse so we can restart without waiting TIME_WAIT */
    int one = 1;
    setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));

    struct sockaddr_in addr = {
        .sin_family = AF_INET,
        .sin_port   = htons(8080),
        .sin_addr.s_addr = INADDR_ANY,
    };
    bind(lfd, (struct sockaddr *)&addr, sizeof(addr));
    listen(lfd, 5);   /* backlog: up to 5 queued connections */

    printf("Listening on :8080\n");
    for (;;) {
        struct sockaddr_in client;
        socklen_t len = sizeof(client);
        int cfd = accept(lfd, (struct sockaddr *)&client, &len);
        printf("Connection from %s\n", inet_ntoa(client.sin_addr));

        char buf[1024];
        ssize_t n;
        while ((n = read(cfd, buf, sizeof(buf))) > 0)
            write(cfd, buf, n);   /* echo back */
        close(cfd);
    }
}

The Internet protocol stack is often drawn as four layers (or seven in the OSI model). What matters is understanding the job each layer has and the header it adds to your data.

Application Layer (your program): You hand a byte stream to the kernel via write(). The kernel does not care what those bytes mean — HTTP, SSH, a JPEG — it just promises reliable, ordered delivery.

Transport Layer (TCP/UDP): TCP adds a 20-byte header containing source port, destination port, sequence number, acknowledgment number, window size, and checksum. Sequence numbers are how TCP achieves ordered, reliable delivery: every byte has a number, and the receiver sends ACKs confirming which bytes it has received. The window size is how TCP does flow control: the receiver advertises how much buffer space it has, and the sender will not exceed that.

Network Layer (IP): IP adds a 20-byte header with source IP, destination IP, TTL (time-to-live, decremented by each router), protocol (6=TCP, 17=UDP, 1=ICMP), and checksum. IP is stateless and best-effort: it will drop packets if a queue is full, deliver them out of order, or deliver duplicates. TCP's job is to hide all of this.

Link Layer (Ethernet): The NIC adds a 14-byte Ethernet frame header: destination MAC address (48 bits), source MAC (48 bits), EtherType (0x0800 = IPv4). The destination MAC is resolved via ARP: the kernel broadcasts "who has IP 192.168.1.1?" and the owner replies with its MAC.

When you call write(fd, data, n), the kernel builds these headers from bottom to top in a struct sk_buff (socket buffer), then hands it to the NIC driver. When a packet arrives, the NIC DMAs it into memory and raises an interrupt (or triggers NAPI polling), and the kernel strips headers from bottom to top until the payload lands in your socket's receive buffer.

The crucial insight: each layer only reads its own header. The IP layer does not know about TCP ports. The Ethernet driver does not know about IP addresses. This layering is what makes it possible to swap out Ethernet for WiFi or TCP for QUIC without rewriting everything else.

/* Show the IP + TCP headers the kernel actually sends.
   Run: sudo tcpdump -i lo -X port 8080
   while this client connects to the echo server above. */
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>

int main(void) {
    int fd = socket(AF_INET, SOCK_STREAM, 0);
    struct sockaddr_in srv = {
        .sin_family = AF_INET,
        .sin_port   = htons(8080),
    };
    inet_pton(AF_INET, "127.0.0.1", &srv.sin_addr);
    connect(fd, (struct sockaddr *)&srv, sizeof(srv));
    write(fd, "hello\n", 6);
    char buf[16];
    read(fd, buf, 6);
    close(fd);
    return 0;
}

Every packet inside the Linux kernel lives in a struct sk_buff (socket buffer), defined in include/linux/skbuff.h. Understanding sk_buff is the master key to reading networking kernel code.

An sk_buff is a metadata structure — it does not contain the packet data itself. Instead it contains pointers into a separately-allocated data buffer:

• head: start of the allocated buffer
• data: start of the current packet payload (moves as headers are pushed/pulled)
• tail: end of the current payload
• end: end of the allocated buffer

When a packet travels down the stack (being sent), headers are pushed by decrementing data and writing the new header there. When a packet travels up the stack (being received), headers are pulled by incrementing data past each consumed header. This means adding a TCP header to an IP packet is just skb_push(skb, sizeof(struct tcphdr)) — no copying of the payload.

The sk_buff also carries metadata used throughout the stack:

• sk: the owning socket (set when a packet is matched to a connection)
• dev: the network device the packet arrived on or will leave from
• protocol: the EtherType
• ip_summed: whether the checksum has been verified (NIC offload)
• cb[]: a 48-byte scratchpad that each protocol layer uses for its own state

sk_buff is allocated from a slab allocator and can be cloned cheaply (clone shares the data buffer, incremented refcount) for multicast or retransmission scenarios.

When you contribute networking patches, you will read and write sk_buff manipulation constantly. The key helper functions: skb_push, skb_pull, skb_put, skb_reserve, skb_clone, kfree_skb.

/* Kernel snippet (net/ipv4/tcp_output.c style) showing
   how a TCP header is prepended to a payload.
   This is simplified — real code does much more. */

// reserve headroom for headers before writing payload
skb_reserve(skb, MAX_TCP_HEADER);

// write application data at skb->tail
unsigned char *payload = skb_put(skb, data_len);
memcpy(payload, data, data_len);

// push TCP header (moves skb->data backward)
struct tcphdr *th = (struct tcphdr *)skb_push(skb, sizeof(*th));
memset(th, 0, sizeof(*th));
th->source = htons(sport);
th->dest   = htons(dport);
th->seq    = htonl(seq);
th->doff   = sizeof(*th) / 4;
th->syn    = 1;

// push IP header on top
struct iphdr *iph = (struct iphdr *)skb_push(skb, sizeof(*iph));
iph->protocol = IPPROTO_TCP;
iph->saddr    = saddr;
iph->daddr    = daddr;

These three tools form your diagnostic toolkit. Together they let you inspect every layer of the network stack from the command line.

ip — the modern replacement for ifconfig and route ip link show — list network interfaces, their flags (UP, LOWER_UP, PROMISC), and MAC addresses. ip addr show — list IP addresses assigned to each interface. ip route show — show the routing table. The kernel consults this table for every outgoing packet to decide which interface to use and what the next-hop IP should be. ip neigh show — show the ARP table (IP→MAC mappings).

ss — socket statistics, the modern replacement for netstat ss -tulpn — show all TCP/UDP listening sockets with the owning process. ss -tnp state established — show established TCP connections. ss -i — show TCP internal state: retransmit timer, congestion window (cwnd), slow-start threshold (ssthresh), round-trip time (rtt). This is extraordinarily useful for diagnosing performance problems.

tcpdump — packet capture and display tcpdump reads packets directly from the kernel via a raw socket on the pcap interface, attaching a BPF filter in the kernel before any packet copy happens (so filtered-out packets cost almost nothing). sudo tcpdump -i any -n port 8080 — capture any packet on any interface involving port 8080, without DNS resolution. sudo tcpdump -i lo -XX -s 0 port 8080 — dump full hex+ASCII of all bytes. sudo tcpdump -w /tmp/cap.pcap port 8080 — write to a file, then open in Wireshark for GUI analysis.

The tcpdump filter language (BPF filter expressions like tcp and host 10.0.0.1 and port 443) is compiled directly to eBPF bytecode and runs in the kernel, making it very efficient.

# diagnose a slow connection in three commands:

# 1. is the route correct?
ip route get 8.8.8.8
# output: 8.8.8.8 via 192.168.1.1 dev eth0 src 192.168.1.100

# 2. watch TCP congestion state in real-time
watch -n 1 'ss -i dst 8.8.8.8'
# look for: cwnd (congestion window), rtt, retrans

# 3. capture the actual packets
sudo tcpdump -i eth0 -n 'host 8.8.8.8 and port 443' -c 20
# SYN, SYN-ACK, ACK = handshake
# then TLS ClientHello, etc.

When your program calls write(fd, buf, n) on a TCP socket, the data takes a precise path through the kernel. Understanding this path tells you where latency comes from and where to look when things go wrong.

1. System call entry: write() invokes the sys_write syscall, which calls sock_sendmsg() in net/socket.c. This dispatches to the protocol-specific sendmsg handler — for TCP, that is tcp_sendmsg() in net/ipv4/tcp.c.

2. Copying into the send buffer: tcp_sendmsg() copies the application data into sk_buff structures in the socket's send buffer (bounded by sk->sk_sndbuf). If the send buffer is full, the write() call blocks (for blocking sockets) or returns EAGAIN (for non-blocking).

3. TCP segmentation: tcp_write_xmit() takes sk_buffs from the send queue and segments them according to the MSS (maximum segment size, negotiated during the handshake, typically 1460 bytes for Ethernet). Congestion control (tcp_cwnd_test()) and the receiver's advertised window are checked — if either would be exceeded, transmission stops.

4. IP layer: ip_queue_xmit() in net/ipv4/ip_output.c looks up the route, fills the IP header, and optionally fragments the packet if it exceeds the MTU.

5. Netfilter hooks: Before the packet leaves the host, it passes through the NF_INET_LOCAL_OUT and NF_INET_POST_ROUTING netfilter hooks — this is where iptables/nftables rules run.

6. Device queue: dev_queue_xmit() pushes the packet onto the network device's qdisc (queuing discipline — tc's domain). The default qdisc is pfifo_fast (a simple three-band priority queue).

7. NIC driver: The driver's ndo_start_xmit() function is called, which programs the NIC's DMA ring with the packet's memory address. The NIC reads the packet from memory via DMA and transmits it. The kernel is not involved in the actual electrical transmission.

8. Completion: The NIC raises an interrupt (or the driver polls it) when transmission is done. The kernel frees the sk_buff and advances the TCP send window.

/* You can observe this path with strace + tcpdump:

   terminal 1: gcc -O2 minimal_server.c -o server && ./server
   terminal 2: strace -e trace=network ./client
   terminal 3: sudo tcpdump -i lo -n port 8080

   strace output will show:
     socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
     connect(3, {sa_family=AF_INET, sin_port=htons(8080), ...}, 16) = 0
     write(3, "hello\n", 6)               = 6
     read(3, "hello\n", 16)               = 6
     close(3)                             = 0

   tcpdump output will show the SYN/SYN-ACK/ACK handshake
   followed by the data segment and FIN/ACK teardown.
*/

The receive path is the reverse journey, but the mechanism is quite different — it is interrupt-driven and uses a technique called NAPI (New API) to avoid interrupt storms under high load.

1. NIC receives a frame: The NIC's hardware receives an Ethernet frame, DMAs it into a pre-allocated ring buffer in kernel memory (these buffers are allocated by the driver at initialization), and raises a hardware interrupt.

2. Hard IRQ handler: The NIC driver's IRQ handler (ndo_start_xmit counterpart on the receive side) runs in interrupt context. It does the minimum work: acknowledges the interrupt and schedules a NAPI poll. Crucially, it disables the NIC's receive interrupt so the CPU is not bombarded with interrupts on every subsequent packet. This is the NAPI design.

3. SoftIRQ / NAPI poll: The kernel's NET_RX_SOFTIRQ soft interrupt runs (deferred, non-atomic context). It calls the driver's napi_poll() function, which drains up to budget (default 64) frames from the DMA ring, converting each into an sk_buff and calling netif_receive_skb(). When the ring is empty, the NIC interrupt is re-enabled.

4. Netfilter hooks: Each sk_buff passes through NF_INET_PRE_ROUTING, then routing lookup. If the packet is destined for this host, it passes through NF_INET_LOCAL_IN. This is where iptables INPUT chain rules run.

5. IP layer: ip_local_deliver() strips the IP header and dispatches to the transport layer based on the protocol field (inet_protos[protocol]).

6. TCP layer: tcp_v4_rcv() in net/ipv4/tcp_ipv4.c looks up the socket (__inet_lookup_skb()), runs the TCP state machine (handles ACKs, out-of-order segments, retransmits), and if the data is in-order, calls tcp_data_queue() to append it to the socket's receive buffer.

7. Waking the application: After adding data to the receive buffer, the kernel calls sk->sk_data_ready() (usually sock_def_readable()), which wakes any process sleeping in read(), select(), or epoll_wait() on this socket.

8. read() call: Your application's read() calls tcp_recvmsg(), which copies data from the kernel's receive buffer into your userspace buffer and advances the TCP receive window (potentially sending a window update ACK to the sender).

/* You can observe the receive buffer fill up with ss.
   Connect the echo client, pause before reading,
   and watch the recv-q column in ss output: */

// in client, before the read() call:
pause(); // SIGCONT to resume

// in another terminal:
// ss -tnp | grep 8080
// you will see Recv-Q grow as data arrives but is not read

Netfilter is the Linux kernel's packet filtering framework. It installs hooks at five strategic points in the IP stack:

1. NF_INET_PRE_ROUTING — immediately after the packet arrives, before routing
2. NF_INET_LOCAL_IN — for packets destined for this machine, after routing
3. NF_INET_FORWARD — for packets being forwarded to another machine
4. NF_INET_LOCAL_OUT — for packets generated by this machine, before routing
5. NF_INET_POST_ROUTING — for all outgoing packets, after routing

At each hook, registered callback functions are called in priority order. Each callback returns a verdict: NF_ACCEPT (continue processing), NF_DROP (discard the packet), NF_STOLEN (I own this packet now), or NF_QUEUE (send to userspace via nfnetlink).

iptables is the classic userspace tool that registers rules at these hooks via the ip_tables kernel module. Rules are organized into tables (filter, nat, mangle, raw) and chains (INPUT, OUTPUT, FORWARD, PREROUTING, POSTROUTING). Each rule matches a set of criteria (source IP, destination port, protocol, connection state) and has a target (ACCEPT, DROP, REJECT, DNAT, SNAT, LOG).

Conntrack (connection tracking, nf_conntrack) is a separate but tightly coupled module that maintains a hash table of all active connections and their state (NEW, ESTABLISHED, RELATED, INVALID). This is what makes stateful firewall rules like -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT work — the kernel knows if this packet belongs to a previously established flow.

nftables is the modern replacement for iptables, introduced in Linux 3.13. It uses a single kernel module (nf_tables) with a much cleaner, more powerful rule language, and a JIT-compiled virtual machine for matching. The nft command replaces iptables, ip6tables, arptables, and ebtables. Distributions are migrating to nftables; new kernel code should target it.

NAT (Network Address Translation) is implemented as a special Netfilter target that modifies the source or destination IP/port of a packet and tells conntrack to apply the reverse translation to reply packets automatically.

# iptables: allow established/related, allow SSH, drop everything else
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -j DROP

# nftables equivalent (preferred on modern systems)
nft add table inet filter
nft add chain inet filter input \
    '{ type filter hook input priority 0; policy drop; }'
nft add rule inet filter input \
    ct state established,related accept
nft add rule inet filter input \
    tcp dport 22 accept

# see what conntrack knows about active connections
cat /proc/net/nf_conntrack
# or: conntrack -L

# trace a packet through the entire netfilter pipeline
# (extremely useful for debugging)
iptables -t raw -A PREROUTING -p tcp --dport 8080 -j TRACE
modprobe nf_log_ipv4
iptables -A INPUT -j LOG --log-prefix "INPUT: "
# then watch: dmesg -w

A network namespace is a complete, isolated copy of the Linux networking stack: its own interfaces, routing table, ARP table, iptables/nftables rules, port number space, and socket table. Two processes in different network namespaces can each bind to port 80 without conflict. This is the foundation of container networking (Docker, Kubernetes pods, LXC).

Network namespaces are one member of the Linux namespace family (others: PID, mount, UTS, IPC, user namespaces). They are created with clone(CLONE_NEWNET) or unshare(CLONE_NEWNET), and managed from userspace with ip netns.

What is in a new, empty network namespace? Only a loopback interface (lo), in the DOWN state. No routes, no external connectivity. You have to wire it up manually.

Connecting namespaces with veth pairs: A veth (virtual Ethernet) pair is a pair of virtual NICs connected back-to-back — packets sent into one end come out the other. The standard pattern:

1. Create a veth pair: ip link add veth0 type veth peer name veth1
2. Move one end into a namespace: ip link set veth1 netns myns
3. Configure both ends with addresses: ip addr add 10.0.0.1/24 dev veth0 and ip netns exec myns ip addr add 10.0.0.2/24 dev veth1
4. Bring both up.

Connecting multiple namespaces to the internet: Use a bridge (virtual switch). Docker creates a docker0 bridge, moves one veth end into each container namespace, and uses MASQUERADE (SNAT) to give containers internet access through the host's IP.

Network namespaces are also used by network engineers to simulate multi-router topologies on a single laptop — a technique heavily used for kernel networking development and testing.

#!/bin/bash
# Build a two-namespace network with a veth pair and ping across it.
# Run as root.

# create namespace
ip netns add red
ip netns add blue

# create veth pair
ip link add veth-red type veth peer name veth-blue

# assign one end to each namespace
ip link set veth-red netns red
ip link set veth-blue netns blue

# configure addresses inside each namespace
ip netns exec red  ip addr add 192.168.100.1/24 dev veth-red
ip netns exec blue ip addr add 192.168.100.2/24 dev veth-blue

# bring interfaces up
ip netns exec red  ip link set veth-red up
ip netns exec red  ip link set lo up
ip netns exec blue ip link set veth-blue up
ip netns exec blue ip link set lo up

# ping across the namespace boundary
ip netns exec red ping -c 3 192.168.100.2

# run a server in 'blue' and connect from 'red'
ip netns exec blue nc -l 9999 &
ip netns exec red  nc 192.168.100.2 9999 <<< "hello from red"

# cleanup
ip netns del red
ip netns del blue

The naive echo server above uses one thread per connection. This breaks down at scale: a server with 10,000 simultaneous connections cannot maintain 10,000 threads (each thread consumes ~8MB of stack by default, plus scheduling overhead).

The solution is non-blocking I/O with event notification. Instead of blocking in read() waiting for data, you tell the kernel "notify me when any of these sockets have data ready, then I will read all of them in a single thread".

The Linux mechanism for this is epoll (event poll), introduced in Linux 2.5.44. The API:

1. epoll_create1(0) — create an epoll instance (returns an fd)
2. epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &event) — register interest in events on fd (EPOLLIN = data ready to read, EPOLLOUT = writable, EPOLLET = edge-triggered mode)
3. epoll_wait(epfd, events, maxevents, timeout) — block until at least one registered fd has an event; returns the list of ready fds

This scales to hundreds of thousands of connections because epoll_wait runs in O(1) — the kernel maintains a red-black tree of watched fds and a linked list of ready fds. When data arrives on a socket, the socket's ready state is added to the ready list (this is called from sk_data_ready, the same callback mentioned in the receive path).

Edge-triggered vs level-triggered: Level-triggered (default) notifies you as long as data is available. Edge-triggered (EPOLLET) notifies you only when the state changes (transition from not-ready to ready). Edge-triggered is more efficient but requires you to read() in a loop until EAGAIN to drain the buffer completely.

Modern high-performance servers (nginx, Redis, Node.js's libuv) are built on epoll. The lesson: the socket API is simple, but the performance model requires understanding the event loop pattern.

/* epoll echo server skeleton — handles many connections in one thread */
#include <sys/epoll.h>
#include <unistd.h>
#include <fcntl.h>
#include <arpa/inet.h>
#include <errno.h>

static void set_nonblock(int fd) {
    fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_NONBLOCK);
}

int main(void) {
    int lfd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
    int one = 1; setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &one, 4);
    struct sockaddr_in a = {.sin_family=AF_INET,.sin_port=htons(8080),.sin_addr.s_addr=INADDR_ANY};
    bind(lfd,(struct sockaddr*)&a,sizeof(a)); listen(lfd,128);

    int epfd = epoll_create1(0);
    struct epoll_event ev = {.events=EPOLLIN, .data.fd=lfd};
    epoll_ctl(epfd, EPOLL_CTL_ADD, lfd, &ev);

    struct epoll_event evs[64];
    for (;;) {
        int n = epoll_wait(epfd, evs, 64, -1);
        for (int i = 0; i < n; i++) {
            if (evs[i].data.fd == lfd) {
                /* new connection */
                int cfd = accept4(lfd, NULL, NULL, SOCK_NONBLOCK);
                struct epoll_event ce = {.events=EPOLLIN|EPOLLET,.data.fd=cfd};
                epoll_ctl(epfd, EPOLL_CTL_ADD, cfd, &ce);
            } else {
                /* data ready: read until EAGAIN */
                char buf[4096]; ssize_t r;
                while ((r = read(evs[i].data.fd, buf, sizeof(buf))) > 0)
                    write(evs[i].data.fd, buf, r);
                if (r == 0) { close(evs[i].data.fd); }
            }
        }
    }
}

Every time the kernel needs to send a packet, it performs a route lookup: given a destination IP address, which interface should this packet leave on, and what is the next-hop IP (the gateway)?

The Linux routing subsystem maintains multiple data structures. The main one visible to users is the FIB (Forwarding Information Base), shown by ip route show. The kernel also maintains a route cache (removed in Linux 3.6 — now every lookup goes to the FIB) and a neighbour cache (the ARP table, ip neigh show).

A FIB entry looks like: 10.0.0.0/8 via 192.168.1.1 dev eth0 proto static metric 100. The kernel selects the most specific match (longest prefix match). If no match is found, the kernel uses the default route (0.0.0.0/0), or if there is no default route, the packet is dropped with ENETUNREACH.

After selecting a route, the kernel resolves the next-hop IP to a MAC address via ARP (for Ethernet). If the ARP entry is not in the cache, the kernel sends an ARP broadcast, waits for a reply, and queues the packet. This is why the first packet to a new destination has higher latency.

Policy routing: Linux supports multiple routing tables (up to 255) and can select which table to use based on rules (ip rule show). This is used for source-based routing (different default routes based on source IP), VPNs, and container networking.

ECMP (Equal-Cost Multi-Path): Linux can distribute traffic across multiple equal-cost routes using a hash of the 5-tuple (src IP, dst IP, src port, dst port, protocol). This is how Linux implements simple load balancing at the IP level.

# Observe the route a packet actually takes:
ip route get 8.8.8.8
# output: 8.8.8.8 via 192.168.1.1 dev eth0 src 192.168.1.100 uid 1000
#         cache

# The 'cache' line means this result was just computed and may be
# cached for performance. In older kernels there was an explicit
# route cache; now this is ephemeral.

# Add a static route (e.g., reach 10.20.0.0/16 via a specific gateway)
ip route add 10.20.0.0/16 via 192.168.1.254 dev eth0

# See all routes including kernel-generated 'connected' routes:
ip route show table all

# Watch ARP resolution happen in real-time:
ip neigh flush all   # clear cache
sudo tcpdump -i eth0 arp &
ping -c 1 192.168.1.1   # triggers ARP broadcast then unicast reply