[RndTbl] load throttling

[Gilbert E. Detillieux]

On 2012-01-12 15:28, Trevor Cordes wrote:
> (A real question this time!)
>
> I have a web (+ other tasks) server that sometimes does a lot of tasks in
> a short time that spike the load up to large values (like 30).  The tasks
> are all long jobs of lots (tens of thousands) of little (10/s) jobs.  I've
> written code to, between each little job, check the /proc/loadavg load (1
> and 5 min) and insert usleeps (.5+ secs) based on the loadavg (exponential
> relationship).  My goal is to keep the load<  2.0 on a 2 core system.
>
> Anyhow, it seems to be working fine.  The weird thing is, at 2.0 loadavg,
> looking at top, I see the CPUs mostly at idle.  Not even very much wait
> going on.  Why would the load be 2 when both CPUs are mostly at 5-10%
> including wait?

My guess is that your overall processing is disk-bound.  Load average 
includes not only processes in in the run queue, but those waiting for 
disk I/O (normally, fairly short waits).  The MUUG server sees similar 
loads when a lot of people are hammering at the mirror file system (over 
http, ftp and/or rsync), with relatively low CPU percentages.

> I basically want to make sure the system doesn't get overloaded and can
> respond to non-batch request (like a web hit) promptly.  What loadavg
> should I target if not (1.0 * numcores)?  I guess I can arbitrarily pick
> numbers and see what happens, but I like things like this to make sense,
> and to me 1.0 * numcores makes sense, unless I'm not understanding loadavg
> properly.

Yeah, I think you'll have to experiment, since it depends on how much of 
your load is disk vs CPU.  I would certainly make sure to set the target 
threshold higher than 2, and probably lower than 10.  Maybe start with 
4, and if your CPU usage is still low, bring it up toward 8.

YMMV!

Gilbert

-- 
Gilbert E. Detillieux		E-mail: <gedetil at muug.mb.ca>
Manitoba UNIX User Group	Web:	http://www.muug.mb.ca/
PO Box 130 St-Boniface		Phone:  (204)474-8161
Winnipeg MB CANADA  R2H 3B4	Fax:    (204)474-7609