[RndTbl] Parallel curl from the command line

Helgi Hrafn Gunnarsson helgi at binary.is
Tue Mar 15 16:11:06 CDT 2011


That is most certainly strange. Personally I would suspect the cURL module
first since PHP and Apache are specifically designed to run in parallel.
Perhaps it's implemented in such a way that PHP speaks to only one cURL
process, that's possible, although I'd call that a catastrophic design flaw
if that were the case.

PHP's cURL library can however do asynchronous requests, even from a single
PHP execution. While it's arguably not a proper solution to the weirdness of
your problem, you may have more luck with it.

Here's one tutorial that I found and looks promising:

http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/

Hope it helps! Let us know. :)

Kind regards,
Helgi Hrafn Gunnarsson
helgi at binary.is

On Tue, Mar 15, 2011 at 4:02 PM, John Lange <john at johnlange.ca> wrote:

> I have some code (written in PHP) that checks on the status of a
> single "server" but I need this code to scale up to hundreds of
> parallel checks.
>
> For example, you call the script like this:
>
> http://mytest.com/checkserver.php?ip=10.11.12.13
>
> where the ip=X.X.X.X is different every time depending on which server
> you are checking.
>
> And the PHP looks something like this:
>
> <?php
>
> $ch = curl_init();
> curl_setopt($ch, CURLOPT_URL, "http://".$ip."/status.html");
> curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
> $output = curl_exec($ch);
> curl_close($ch);
>
> ?>
>
> To my way of thinking, this should scale easily. Lets say 200 people
> hit the page at the same time all passing different "ip=" values, this
> should check all 200 different servers in parallel and return results.
>
> It doesn't. I wrote this little bash script to test it (yes I'm using
> command line curl to test php-curl, please don't be confused by that):
> ---
> #!/bin/bash
>
> IPS="10.18.136.20
> 10.18.136.21
> 10.18.136.22
> ... ( a hundred more IPs)"
>
> for IP in $IPS ; do
>  curl -s http://mytest.com/checkserver.php?ip=$IP &
> done
> ---
>
> All the curl commands launch into the background as you expect and I
> see all the apache child threads startup, but the results return one
> by one.
>
> I can't for the life of me figure out how this can be possible. Each
> apache thread should run it's PHP in a separate thread and return in
> parallel. The only thing I can think of is that php is tracking all
> the requests as being part of the same session and imposing some limit
> on the outbound curl requests... Or maybe it's apache blocking it?
>
> Just thought someone might have had to do something similar in the
> past and run into this.
>
> By the way, if I go direct like this:
>
> for IP in $IPS ; do
>  curl -s http://".$IP"/status.html &
> done
>
> It works just exactly as you'd expect, all results return in parallel
> so it's got to be a problem with either PHP or apache.
>
> --
> John Lange
> www.johnlange.ca
> _______________________________________________
> Roundtable mailing list
> Roundtable at muug.mb.ca
> http://www.muug.mb.ca/mailman/listinfo/roundtable
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.muug.mb.ca/pipermail/roundtable/attachments/20110315/2532e1d4/attachment.html 


More information about the Roundtable mailing list