[RndTbl] weird apache hit

[Trevor Cordes]

Deeper into the rabbit hole...

I started logging a ton of extra stuff on each apache hit to get a
better idea of what is going on with these weird hits, and finally
after a 30 hour absence one arrived:

1.2.3.4 - - [20/Feb/2020:17:28:35 -0600] "GET / HTTP/1.1" 403 199
"-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/41.0.2228.0 Safari/537.36" 80 foo.com
/var/www/html/ HTTP/1.1 Xk8WIxeqhm968vI6R9xOIwAAAIY GET 443
1.2.3.4 1.2.3.4 1.1.1.2

My log setup:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
%p %v %f %H %L %m %{local}p %a %{c}a %A" foo

No, you don't need to wade through all of that.  There's 2 interesting
bits:

%A says they are hitting our 1.1.1.2 (ip changed to protect the
innocent) when our main IP is 1.1.1.1!  We don't even have any dns
pointing to 1.1.1.2 any more.  Yet apache is set to listen on * (all
nics).  Ok, not the end of the world, but interesting.

Doubly interesting is we don't have a 1.1.1.2:443 virthost in apache!
Apache seems to listen to 1.1.1.2:443 anyhow.

Also note that %p ("canonical port") is saying the hit is on port 80
but %{local}p ("local port") says it's port 443!  What what what?
That's insane!

The clincher was my tcpdump on all interfaces on port 80 was *not*
showing any matching traffic for these hits!  So they must have been
coming in on port 80.

So finally after all of that I've reproduced the problem with simple
telnet of all things, after hammering away with openssl and gnutls-cli
to no avail (and you'll see why)...

telnet 1.1.1.2 443

GET / HTTP/1.1
Host: foo.com

Voila, I get a log entry on can-port 80 (!!!), local port 443 and it
hits my "forbidden" dir of /var/www/html.

So apache will happily talk http (port 80) on port 443, with no ssl/tls
handshake/support, and instead of using my sole virthost on 443
(1.1.1.1:443) *or* the *:80 virthost (!!!) it uses no virthost at all
and defaults to the global docroot!  Ok, it sort of makes sense.

So my next task (ideas appreciated!) is to tell apache to not speak
non-ssl on any 443 port.  It already seems to do this on the
1.1.1.1:443 vhost, but not on the non-specified ips (!!).  I could also
be strict with my listens and interfaces to make sure I'm explicitly
listing every interface and have matching virthosts for all of them.
This may be complicated by the fact the extra ips are on the same
physical interface.  Lastly, I will still change my global docroot to
point to some honeypot dir where I can easily grep for anomalous hits
in the future. (Any other ideas?)

In theory, fudging with SNI should allow similar behavior, but I
couldn't get our server to work trying to fudge with SNI options in
gnutls-cli.  Maybe our apache isn't working with SNI because only 1
:443 virthost exists, I'm not sure.  Or maybe I'm not using it correct
in the client.

The "\x16\x03\x01\x01L\x01" request string hits I reproduced by using
gnutls-cli on the 1.1.1.2 IP which immediately barfs every time,
probably because apache doesn't know to talk ssl on that ip on any port.

I find much of the above somewhat disturbing given that some of the
behavior choices are quite odd and/or dangerous.  I'm sure I'm not the
only one with a config like this.