[RndTbl] mysql crash and non-restart due to RAM?

[Trevor Cordes]

I had a box with 32GB RAM have its mysql just die (no logs, just die).  
init tried to restart it (RHEL6) and I got this:

160104  8:55:55  InnoDB: Initializing buffer pool, size = 3.9G
160104  8:55:55  InnoDB: Error: cannot allocate 4194320384 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 48401248 bytes. Operating system errno: 12
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: Note that in most 32-bit computers the process
InnoDB: memory space is limited to 2 GB or 4 GB.
InnoDB: We keep retrying the allocation for 60 seconds...

The setting of a 4G buffer pool was working for months until just now.

I reduced it to 3G and mysql restarted ok.

Top shows:
Mem:  32827576k total,  2274940k used, 30552636k free,   165052k buffers
Swap:  2097148k total,   276068k used,  1821080k free,   413640k cached

Cacti (up until mysql crashed, wink) shows ps load going up, but that is 
normal for this box, and RAM usage stayed constant, right before the 
crash.  However, we did launch a long ps that spawns a lot of little ps's 
(like maybe 100 at a time).  This is all normal for this box, and it never 
has done this before.

I'm guessing RAM fragmentation:

#cat /proc/buddyinfo
Node 0, zone      DMA      2      1      1      1      1      0      1      0      1      1      3 
Node 0, zone    DMA32   1815   1789   1553   1294   1007    688    415    212    103     67    477 
Node 0, zone   Normal  32859  67280  77301  62304  45390  29754  19144  11069   4786    530    177 
(anything else to check?)

Looks like tons of pages avail of every size, assuming mysql uses 
"normal".

Any ideas?  My only thought now is to reboot server more often to clear 
fragmentation.  Maybe newer kernels solve this "bug" a bit better, but I'm 
stuck on RHEL6 right now.