[RndTbl] Trevor's wacky slow 10MB/s disk I/O update

[Trevor Cordes]

Thanks a ton for all the great suggestions at the meeting the other night 
about my slow disk I/O.  (For those not at the meeting, this is the slow 
I/O issue I've been battling with for months and have asked about here on 
the mailing list numerous times.)

I think Adam(?) suggested testing dd with an option to avoid the cache 
completely, after others thought it sounded like a cache issue.  The 
option for that (after a quick RTFM) is oflag=direct.

Here's the results of that on the system after being up a day (without my 
hourly test running):

spinning rust:
5:15am ~#dd if=/dev/zero of=/dev/sda2 bs=1M count=200 conv=fdatasync
209715200 bytes (210 MB) copied, 9.07724 s, 23.1 MB/s
5:15am ~#dd if=/dev/zero of=/dev/sda2 bs=1M count=200 conv=fdatasync oflag=direct
209715200 bytes (210 MB) copied, 2.04618 s, 102 MB/s

ssd:
5:18am ~#dd if=/dev/zero of=/dev/sdb2 bs=1M count=200 conv=fdatasync 
209715200 bytes (210 MB) copied, 3.38177 s, 62.0 MB/s
5:16am ~#dd if=/dev/zero of=/dev/sdb2 bs=1M count=200 conv=fdatasync oflag=direct
209715200 bytes (210 MB) copied, 0.508799 s, 412 MB/s

Whoa!  Indeed we have a cache issue.  Disks are near full speed when 
writing direct to disk, but slow when going through the cache.

FYI, here's what top looks like after I ran the tests (for mem info):

top - 05:21:49 up 1 day,  4:43,  1 user,  load average: 0.00, 0.03, 0.07
Tasks: 212 total,   1 running, 211 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni, 99.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 16522952 total,  4668792 free,   706636 used, 11147524 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 15709432 avail Mem 

So it looks like buff/cache is getting quite large after all (11GB).  So 
we'll try the next suggestion:

5:23am ~#sync ; echo 1 >> /proc/sys/vm/drop_caches

top
KiB Mem : 16522952 total, 15508640 free,   706136 used,   308176 buff/cache

5:24am ~#dd if=/dev/zero of=/dev/sdb2 bs=1M count=200 conv=fdatasync
209715200 bytes (210 MB) copied, 4.39049 s, 47.8 MB/s

So that doesn't help at all (that's the ssd and should be 400-500MB/s).  
Very strange indeed!

At least I have some better search terms to google with in the next few 
days... maybe I'll get lucky.