Geek asks and answers

Multi curl cannot handle more than 200 requests at a time

Could you tell me if there are any restrictions on sending a request using multi_curl. When I tried to send a request to more than 200, it received a timeout.

see code below .............. ................................ .........

foreach($newUrlArry as $url){   
            $gatherUrl[] = $url['url'];
        }
        /*...................Array slice----------------------*/

        $totalUrlRequest = count($gatherUrl);
        if($totalUrlRequest > 10){
            $offset = 10;
            $index = 0;
            $matchedAnchors = array();
            $dom = new DOMDocument;
            $NoOfTilesRequest = ceil($totalUrlRequest/$offset);
            for($sl = 0; $sl<$NoOfTilesRequest;$sl++){
                $output = array_slice($gatherUrl, $index, $offset);
                $index = $offset+$index;
                $responseAction = $this->multiRequestAction($output);
                $k=0;
                foreach($responseAction as $responseHtml){
                @$dom->loadHTML($responseHtml);
                $documentLinks = $dom->getElementsByTagName("a");
                $chieldFlag = false;
                for($i=0;$i<$documentLinks->length;$i++) {
                $documentLink = $documentLinks->item($i);
                   if ($documentLink->hasAttribute('href') AND substr($documentLink->getAttribute('href'), 0, strlen($match)) == $match) {
                            $description = $documentLink->childNodes;
                            foreach($description as $words) {
                                $name =  trim($words->nodeName);
                                if($name == 'em' ||  $name == 'b' || $name=="span" || $name=="p") {
                                    if(!empty($words->nodeValue)) {
                                        $matchedAnchors[$sl][$k]['anchor']  = trim($words->nodeValue);
                                        $matchedAnchors[$sl][$k]['img']         = 0;
                                        if($documentLink->hasAttribute('rel'))
                                            $matchedAnchors[$sl][$k]['rel']    = 'Y';
                                        else
                                            $matchedAnchors[$sl][$k]['rel']    = 'N';   
                                        $chieldFlag = true;
                                        break;
                                    }
                                }
                                elseif($name == 'img' ) { 
                                    $alt= $words->getAttribute('alt');
                                    if(!empty($alt)) {
                                        $matchedAnchors[$sl][$k]['anchor']  =  trim($words->getAttribute('alt'));
                                        $matchedAnchors[$sl][$k]['img']         = 1; 
                                        if($documentLink->hasAttribute('rel'))
                                            $matchedAnchors[$sl][$k]['rel']    = 'Y';
                                        else
                                            $matchedAnchors[$sl][$k]['rel']    = 'N';   
                                        $chieldFlag = true;
                                        break;
                                    }
                                }

                            }
                            if(!$chieldFlag){
                                $matchedAnchors[$sl][$k]['anchor']  = $documentLink->nodeValue;
                                $matchedAnchors[$sl][$k]['img']         = 0; 
                                if($documentLink->hasAttribute('rel'))
                                    $matchedAnchors[$sl][$k]['rel']    = 'Y';
                                else
                                    $matchedAnchors[$sl][$k]['rel']    = 'N';   
                            }

                        }

                    }$k++;
                }       
            }
        }
+3
source share
2 answers

Both @Phliplip and @lunixbochs mentioned common cURL errors (maximum runtime and rejected by the target server.)

When sending many cURL requests to the same server, I try to "be beautiful" and set voluntary sleep periods so that I do not bombard the host. For a younger site, 1000+ queries may look like mini DDOS!

, . , NO.

<?php
header('Content-type: text/html; charset=utf-8', true);
set_time_limit(0);
$urls = array(
    'http://www.example.com/cgi-bin/product?id=500', 
    'http://www.example.com/cgi-bin/product?id=501',  
    'http://www.example.com/cgi-bin/product?id=502',  
    'http://www.example.com/cgi-bin/product?id=503',  
    'http://www.example.com/cgi-bin/product?id=504', 
);
$i = 0;
foreach($urls as $url){
    echo $url."\n";
    $curl = curl_init($url);
    $userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';
    curl_setopt($curl, CURLOPT_USERAGENT, $userAgent);
    curl_setopt($curl, CURLOPT_AUTOREFERER, true);
    curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1 );
    curl_setopt($curl, CURLOPT_TIMEOUT, 25 );
    $html = curl_exec($curl);
    $html = @mb_convert_encoding($html, 'HTML-ENTITIES', 'utf-8');  
    curl_close($curl);
    // now do something with info returned by curl 
    $i++;
    if($i%10==0){
        sleep(20);
    } else {
        sleep(2);
    }
}
?>

:

  • new curl init exec .

, () . " " , , :

$urls = array(
    'http://www.example-one.com/', 
    'http://www.example-two.com/', 
    'http://www.example-three.com/', 
    'http://www.example-four.com/', 
    'http://www.example-five.com/', 
    'http://www.example-six.com/'
);

set_time_limit(0);, -, die; try

ini_set('display_errors',1); 
error_reporting(E_ALL);

, .

+4

PHP curl_multi_init, .

memory_limit php.ini , , .

+1

More articles:


All Articles