[RndTbl] MUUG http down last 3 hours

[Trevor Cordes]

On 2017-01-26 Gilbert E. Detillieux wrote:
> I try an "apachectl graceful" on the server, and it has no effect.  I 
> try "apachectl restart" and it's OK for a few seconds, responding to 
> http requests, but after a while it gets jammed up again.  Not sure 
> what's going on, but I have to catch a bus soon...

I'll check it out tonight.  I'm nearly positive Adam had already
tweaked the worker settings because ps shows waaaay more workers than
apache usually does by default.  We may need even more.  There might be
load limits being hit too.

I also have some thoughts about using iptables and/or qos (tc) controls
to give priority to "local" connections (like shaw, mts, les, uofm).
Perhaps even create 3 tiers: local (likely to be muugers/manitobans),
normal, and wtf-are-you-using-us (China, etc).  No one will be blocked,
we'll just give more TCP SYN's and/or egress b/w to the people we are
supposed to be serving first.  We can discuss here or at a board
meeting.

Prioritizing/limiting would also allow us to tune down the load so that
the box doesn't become overall useless for everyone (like we're
experiencing today, sort of).