Recently, needing a break from spreadsheets at work, I did some performance testing of lighttpd. Using an HTTP load generator client without keep-alive connections, I soon ran out of ephemeral port numbers on the Mac that ran the client. Once the connections in TIME_WAIT cleared up, I could resume load testing for a very short while.
OS X 10.5 has a reasonably large default range for ephemeral port numbers:
net.inet.ip.portrange.hifirst: 49152 net.inet.ip.portrange.hilast: 65535
Other than widening the range, the usual way to support more short-lived connections is to reduce the amount of time that closed connections can spend in TIME_WAIT state. On Linux, for example, this can be done as:
echo timeout_in_seconds > /proc/sys/net/ipv4/tcp_fin_timeout
It took me a while to find the equivalent setting for OS X, although I did find a few discussion forums where people had asked how to change the TIME_WAIT interval to less than 2MSL and been told it couldn’t be done.
What ended up working for me was to change net.inet.tcp.msl:
$ sysctl net.inet.tcp.msl net.inet.tcp.msl: 15000 $ sudo sysctl -w net.inet.tcp.msl=1000 net.inet.tcp.msl: 15000 -> 1000
1000ms is too small a value for an Internet-facing system (the default 15 second interval is arguably aggressive enough already), but when testing over a local network it enabled me to do webserver testing at the rate of several thousand new connections per second from one client host.
