[RndTbl] Parallel curl from the command line

John Lange john at johnlange.ca
Tue Mar 15 16:02:55 CDT 2011


I have some code (written in PHP) that checks on the status of a
single "server" but I need this code to scale up to hundreds of
parallel checks.

For example, you call the script like this:

http://mytest.com/checkserver.php?ip=10.11.12.13

where the ip=X.X.X.X is different every time depending on which server
you are checking.

And the PHP looks something like this:

<?php

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://".$ip."/status.html");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);

?>

To my way of thinking, this should scale easily. Lets say 200 people
hit the page at the same time all passing different "ip=" values, this
should check all 200 different servers in parallel and return results.

It doesn't. I wrote this little bash script to test it (yes I'm using
command line curl to test php-curl, please don't be confused by that):
---
#!/bin/bash

IPS="10.18.136.20
10.18.136.21
10.18.136.22
... ( a hundred more IPs)"

for IP in $IPS ; do
  curl -s http://mytest.com/checkserver.php?ip=$IP &
done
---

All the curl commands launch into the background as you expect and I
see all the apache child threads startup, but the results return one
by one.

I can't for the life of me figure out how this can be possible. Each
apache thread should run it's PHP in a separate thread and return in
parallel. The only thing I can think of is that php is tracking all
the requests as being part of the same session and imposing some limit
on the outbound curl requests... Or maybe it's apache blocking it?

Just thought someone might have had to do something similar in the
past and run into this.

By the way, if I go direct like this:

for IP in $IPS ; do
  curl -s http://".$IP"/status.html &
done

It works just exactly as you'd expect, all results return in parallel
so it's got to be a problem with either PHP or apache.

-- 
John Lange
www.johnlange.ca


More information about the Roundtable mailing list