[RndTbl] wacky NFS port problem

[Trevor Cordes]

OK, this is a new one...

Updated my kernel and rebooted tonight.  Everything came up fine.

Except dovecot (IMAP server).  It complained:
Error: service(imap-login): listen(*, 993) failed: Address already in use

(993=imaps)

OK.  Except:

#netstat -tulpn | grep 993
<nothing>
#lsof -i:993
<nothing>
#ss -t -l 'sport = 993'
<nothing>

A wasted hour later I did:

netstat -n |grep 993

Sure enough I had a tcp connection between my workstation and my file 
server on 993 to 2049.  grep 2049 /etc/services... eureka!

2049 is nfs.

>From what I've figured out, nfs, which I have mount something on boot, 
randomly grabbed 993 before dovecot did, and was holding it as long as the 
nfs fs was mounted locally.

And because nfs is in-kernel it didn't show up in lsof et al.

I umounted it, waited for TIME_WAIT to expire and then I could start 
dovecot.

*** This has never happened before on dozens of reboots... I guess it's 
pretty much luck / race condition what port nfs gets, especially with 
systemd possibly trying to do the mount and/or dovecot in parallel.  I 
have no idea when it tries to do the mounts, but I'm guessing before 
dovecot.

Note, I'm using NFSv4 which (unlike older NFS) uses TCP stateful in a 
persistent connection.  Pretty much everything you know about v2/v3 throw 
away when it comes to using v4.

I poked around nfs docs and google and found this option:

resvport / noresvport 
Specifies whether the NFS client should use a privileged source port when 
communicating with an NFS server for this mount point. If this option is 
not specified, or the resvport option is specified, the NFS client uses a 
privileged source port. If the noresvport option is specified, the NFS 
client uses a non-privileged source port. This option is supported in 
kernels 2.6.28 and later.
Refer to the SECURITY CONSIDERATIONS section for important details.

OK, down there is a section:

Using non-privileged source ports

Which basically says using privileged ports is for security so normal 
users can't make/fake their own daemons and pretend to be any user they 
want.  OK, so I know that's a bit hokey from a sec standpoint, but it's 
better than nothing, I guess.  Regardless, having nfs client pick a random 
high port might hurt me also because I use some of them for my own 
purposes/daemons and I need them open also.

The docs say this:

"The exact range of privileged source ports that can be chosen is set by a 
pair of sysctls to avoid choosing a well-known port, such as the port used 
by ssh."

Which would be perfect, except I can't find any further reference to these 
magic sysctls.  I tried searching my box's /proc and /sys fs's for nfs &
ports, nfs & priv, nfs & mount, etc.  Couldn't find anything relevant.

It would be really nice if I could specify the local source port, or at 
least specify the list of no-no ports using this elusive, promised, 
"sysctl".  Anyone have any ideas?

What a strange bug to run into.  Chance of 1 in 1024 and it had to hit me.