[RndTbl] load throttling

Sean Walberg sean at ertw.com
Thu Jan 12 15:42:18 CST 2012


Load average is an exponentially weighted average of the number of
processes in the run queue. Strange that you're not seeing much iowait,
though maybe not everything shows up there. Usually if you see high load
average and low CPU it means you've got high I/O. Watching "vmstat 2"
usually confirms this with blocked processes on the left.

I'm wondering if usleep works differently from sleep? Not necessarily a
busy loop but it doesn't go to sleep to wait for a signal?

BTW if you're trying to manage jobs, something like beanstalkd might be
more efficient than managing your own load, plus it's network aware.
 There's a Ruby gem called Stalker that makes it really easy to run a
worker daemon that runs the commands you want (bindings in just about every
language out there, too) Then it's just a matter of running the number of
worker instances, say one per CPU.

Sean

On Thu, Jan 12, 2012 at 3:28 PM, Trevor Cordes <trevor at tecnopolis.ca> wrote:

> (A real question this time!)
>
> I have a web (+ other tasks) server that sometimes does a lot of tasks in
> a short time that spike the load up to large values (like 30).  The tasks
> are all long jobs of lots (tens of thousands) of little (10/s) jobs.  I've
> written code to, between each little job, check the /proc/loadavg load (1
> and 5 min) and insert usleeps (.5+ secs) based on the loadavg (exponential
> relationship).  My goal is to keep the load < 2.0 on a 2 core system.
>
> Anyhow, it seems to be working fine.  The weird thing is, at 2.0 loadavg,
> looking at top, I see the CPUs mostly at idle.  Not even very much wait
> going on.  Why would the load be 2 when both CPUs are mostly at 5-10%
> including wait?
>
> I basically want to make sure the system doesn't get overloaded and can
> respond to non-batch request (like a web hit) promptly.  What loadavg
> should I target if not (1.0 * numcores)?  I guess I can arbitrarily pick
> numbers and see what happens, but I like things like this to make sense,
> and to me 1.0 * numcores makes sense, unless I'm not understanding loadavg
> properly.
>
> Thanks!
> _______________________________________________
> Roundtable mailing list
> Roundtable at muug.mb.ca
> http://www.muug.mb.ca/mailman/listinfo/roundtable
>



-- 
Sean Walberg <sean at ertw.com>    http://ertw.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.muug.mb.ca/pipermail/roundtable/attachments/20120112/056ab121/attachment.html>


More information about the Roundtable mailing list