A forwarding node on our backbone edge started misbehaving once its connection count climbed past a few hundred thousand: new connections occasionally failed to establish, dmesg filled with nf_conntrack: table full and neighbour: ndisc_cache: neighbor table overflow, and ss -s showed a huge pile of TIME-WAIT entries with local ports running out. The box itself wasn’t CPU- or memory-bound. The ceiling was entirely in the kernel’s default limits.
We see this pattern a lot across our network — edge boxes on the AS37969 backbone, relay nodes on cross-border lines, and CDN/proxy machines fronting large volumes of short-lived connections. The default net.* and fs.* limits are sized for desktops or lightly loaded servers, and they don’t hold up under production traffic. What follows is the /etc/sysctl.conf we’ve settled on for these machines, grouped by purpose, with the reasoning and the tradeoff spelled out for each line.
One blunt warning first: this is not a config you paste wholesale and walk away from. A few of these knobs will reboot the box outright when set wrong, or silently drop connections behind NAT. Pick what fits your case. Don’t paste the whole block.
Memory and OOM behavior
vm.swappiness = 0
vm.panic_on_oom=1
kernel.panic=10
kernel.softlockup_panic = 1
kernel.sysrq = 0
kernel.printk = 5 4 1 7
| Parameter | Value | Effect and cost |
|---|---|---|
vm.swappiness | 0 | Avoid moving pages to swap. On a memory-comfortable network node this keeps latency steady; but once memory actually gets tight the kernel has less room to reclaim, so OOM arrives sooner. Set 0 only when you’re sure there’s headroom. |
vm.panic_on_oom | 1 | Panic on OOM instead of letting the OOM killer pick a process. Pairs with kernel.panic below for a “if it breaks, reboot fast” strategy. Dangerous: on a production box this means a single memory spike reboots the whole machine. If you’d rather kill one process and keep the box up, leave it off. |
kernel.panic | 10 | Reboot automatically 10 seconds after a panic. Only makes sense when you want the machine to self-heal and another node can take over the traffic. |
kernel.softlockup_panic | 1 | Panic and reboot on a detected soft lockup (a CPU stuck without scheduling). Reasonable for an unattended edge box that would otherwise hang, but it turns a fault you could have dumped and investigated into a reboot. |
kernel.sysrq | 0 | Disable the SysRq magic keys. Purely a way to tighten the physical/serial console attack surface; keep it if you rely on SysRq for emergency debugging. |
kernel.printk | 5 4 1 7 | Console log levels. Cuts console noise, but don’t crank it so high that useful warnings get filtered too. |
The core tradeoff in this group: panic_on_oom + kernel.panic is a “reboot rather than hang” policy, right for a redundant cluster where one machine can be sacrificed. On a single box where you need the forensic state, you want the opposite.
File handles
fs.file-max = 65536
High-concurrency proxies, large numbers of SNMP/monitoring sessions, and many open sockets all consume file descriptors. fs.file-max is the system-wide ceiling. Note it’s only the kernel total — what a process can actually open is still bounded by ulimit -n (the nofile limit in /etc/security/limits.conf), and both have to be raised together to matter. On a genuinely heavy box, 65536 is often still low and you’ll want to scale it with your connection count.
Network core buffers
net.core.netdev_max_backlog = 40960
net.core.optmem_max = 921600
net.core.rmem_default = 2097152
net.core.wmem_default = 2097152
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.somaxconn = 65535
| Parameter | Effect and cost |
|---|---|
net.core.netdev_max_backlog | Queue length when the NIC takes packets in faster than the kernel processes them. On 10G ports under small-packet storms, raising it cuts drops; the cost is higher latency and memory use when the queue actually backs up. |
net.core.rmem_max / wmem_max | Per-socket receive/send buffer ceiling. On high bandwidth-delay-product cross-border links, buffers that are too small cap throughput. |
net.core.rmem_default / wmem_default | Default buffer sizes. |
net.core.optmem_max | Ceiling for per-socket ancillary buffers (control data, etc.). |
net.core.somaxconn | Ceiling for the accept queue (handshake complete, waiting for the app to accept). Set too low under a short-connection storm and you drop new connections outright — but the app’s own listen() backlog has to be raised to match, or this does nothing. |
Bigger buffers aren’t strictly better: set them too large and each connection pins a lot of kernel memory, which multiplied across hundreds of thousands of connections can exhaust RAM, and it amplifies bufferbloat. Size them from the link’s bandwidth-delay product rather than just piling on more.
TCP / UDP buffers
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 87380 8388608
net.ipv4.udp_mem = 8388608 12582912 16777216
net.ipv4.udp_rmem_min = 65536
net.ipv4.udp_wmem_min = 65536
net.ipv4.tcp_rmem = 92160 6000000 17476000
tcp_rmem / tcp_wmem are triples: min, default, max. The kernel autotunes each connection’s buffer within that range, so you don’t need — and shouldn’t want — a high default, because raising the default reserves large memory on every idle connection.
A candid note: the source config has net.ipv4.tcp_rmem twice (4096 87380 8388608 and 92160 6000000 17476000), and the second overrides the first. We keep it verbatim because that was the actual state of the machine at the time, but in a real deployment keep only one line. The second raises the minimum to 92160, meaning every TCP connection pins at least 90KB of receive buffer — very memory-hungry once connection counts climb, and only appropriate for a small number of high-throughput, large-BDP connections.
Connections and TIME-WAIT handling
net.ipv4.ip_local_port_range = 1025 65530
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_time = 3600
net.ipv4.tcp_max_syn_backlog = 40960
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1
| Parameter | Effect and cost |
|---|---|
ip_local_port_range | Range of local outbound ports. For a forward proxy or any box making many active outbound connections, port exhaustion is the most common bottleneck; 1025 65530 gives you nearly twenty thousand more ports. |
tcp_fin_timeout | FIN-WAIT-2 timeout. Lower it to reclaim half-closed connections faster, useful for short-connection workloads; too low and you penalize slow peers. |
tcp_max_syn_backlog | SYN (half-open) queue length. Works with syncookies to resist SYN floods. |
tcp_max_tw_buckets | Ceiling on TIME-WAIT slots. Raising it avoids TW buckets overflow drops, at the cost of the memory those slots take. |
tcp_syncookies | Continue the handshake with cookies when the SYN queue is full — the key SYN-flood defense, keep it on. |
tcp_syn_retries / tcp_synack_retries | Handshake retry counts; lower them so failed connections give up sooner. |
tcp_tw_reuse | Allow reusing a TIME-WAIT socket for a new outbound connection. Safe, and valuable on a box making many active outbound connections. |
net.ipv4.tcp_tw_recycle=1 deserves its own line: it was removed entirely in Linux 4.12, and even on older kernels it’s extremely dangerous. It drops packets behind NAT — the kernel treats multiple clients behind the same NAT egress as timestamp regressions and refuses their handshakes, which shows up as “some users randomly can’t connect, and it’s nearly impossible to reproduce.” We keep it here only as a faithful record of the original config. Do not enable it on any modern production box. When you need to recycle TIME-WAIT, tcp_tw_reuse is enough.
Forwarding, reverse path, and the neighbor table
net.ipv4.conf.default.rp_filter = 0
net.ipv4.ip_forward = 1
net.ipv4.neigh.default.gc_thresh1 = 8192
net.ipv4.neigh.default.gc_thresh2 = 4092
net.ipv4.neigh.default.gc_thresh3 = 8192
ip_forward = 1 is a prerequisite for any forwarding/routing/gateway role. rp_filter = 0 turns off reverse-path validation — on a multi-homed box with asymmetric routing (traffic in and out over different links) you have to disable it, or legitimate traffic gets dropped as spoofed source addresses. The cost is losing a layer of source-address anti-spoofing, so a pure host should set it back to 1.
The neighbor table’s (ARP/ndisc) three gc_thresh values govern cache reclamation: gc_thresh3 is the hard ceiling, and when the table fills it logs neighbor table overflow and drops packets. Raise it when you have many devices on the same segment (for example a large block of directly connected servers). Note that in the source config gc_thresh2 (4092) is actually smaller than gc_thresh1 (8192) — the order is backwards. It should satisfy thresh1 < thresh2 < thresh3; we keep it verbatim here, but fix it to ascending values in practice.
IP fragmentation
net.ipv4.ipfrag_high_thresh = 5242880
net.ipv4.ipfrag_low_thresh = 2932160
High/low memory watermarks for reassembling IP fragments. On tunnels, VPNs, or heavy UDP fragmentation, raising these reduces reassembly failures; but fragment reassembly is itself an attack surface, so on a pure application box, avoid fragmentation rather than leaning on these two values to paper over it.
Other network switches
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
tcp_sack, tcp_window_scaling, and tcp_timestamps are the baseline for getting throughput on high-BDP links and should be on by default. tcp_no_metrics_save = 1 stops the kernel from caching the last connection’s performance metrics — better when peer quality is unstable, since one bad history won’t drag the next connection down. icmp_echo_ignore_broadcasts = 1 drops broadcast pings and tightens the attack surface.
Full reference config
Below is the complete /etc/sysctl.conf fragment, each value matching what we run in production. Once more: handle tcp_tw_recycle and the duplicate tcp_rmem according to your own environment, and fix the gc_thresh ordering. Run sysctl -p after editing.
vm.swappiness = 0
vm.panic_on_oom=1
kernel.panic=10
fs.file-max = 65536
kernel.sysrq = 0
kernel.printk = 5 4 1 7
kernel.softlockup_panic = 1
net.ipv4.conf.default.rp_filter = 0
net.ipv4.ip_forward = 1
net.core.netdev_max_backlog = 40960
net.core.optmem_max = 921600
net.core.rmem_default = 2097152
net.core.wmem_default = 2097152
net.core.wmem_max = 8388608
net.core.rmem_max = 8388608
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 87380 8388608
net.ipv4.udp_mem = 8388608 12582912 16777216
net.ipv4.udp_rmem_min = 65536
net.ipv4.udp_wmem_min = 65536
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.ipfrag_high_thresh = 5242880
net.ipv4.ipfrag_low_thresh = 2932160
net.ipv4.ip_local_port_range = 1025 65530
net.ipv4.neigh.default.gc_thresh1 = 8192
net.ipv4.neigh.default.gc_thresh2 = 4092
net.ipv4.neigh.default.gc_thresh3 = 8192
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_time = 3600
net.ipv4.tcp_max_syn_backlog = 40960
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_rmem = 92160 6000000 17476000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.core.somaxconn = 65535
Before you roll this out, change one box, put load on it, and watch dmesg and ss -s to confirm no new drops before you go wider. The gains from kernel tuning are real, and so are the ways it bites you. Verify first, then ship.