Timeouts with opening a site (e.g. people lookup)

porter

Rookie

Offline

Posts: 28

Timeouts with opening a site (e.g. people lookup)

« on: September 09, 2006, 02:48:04 PM »

A bit tired and very much tired with how the SLOW response or no response from the AO people lookup almost stops my 0.2.8 today I edited Bot.php and changed the get_site function:

Code:

/*
Gets a URL
*/
function get_site($url)
{
$fp = fopen($url, "r");
if (!$fp) {
$content = ""; // could not connect
} else {
stream_set_blocking($fp, FALSE);
stream_set_timeout($fp, 15);
$content = "";

while ($buffer = fgets($fp, 1024) && stream_get_meta_data($fp)<>'timed_out')
{
$content .= $buffer;
}
if (stream_get_meta_data($fp)=='timed_out') $content='';
}

return $content;
}

I'm sure this is not perfect but it seemed to help me today. (I have not tested this more than a few minutes, though...)

I do not know if just setting that 15 second timeout would result in fgets($fp, 1024)===FALSE in the while loop so I include the "&& stream_get_meta_data($fp)<>'timed_out'"

Maybe this is something that has already been handled in the development branch in which case I suppose this belongs in 0.2.x support.

Please correct me if all of this is just bullshit and the server simply happened to behave better after I had done this... Wink

[edit:] Of course, now I get no whereis results at all and have no way to know whether it is because the 15 sec timeout is too short, the code is bugged, or the people lookup is still dead. At least it does not hang the bot...


« Last Edit: September 09, 2006, 02:53:54 PM by porter »	Logged

jjones666

BeBot Contributor
Champion

Offline

Posts: 353

Re: Timeouts with opening a site (e.g. people lookup)

« Reply #1 on: September 09, 2006, 07:14:57 PM »

I rigged up a "debug" thing to the whois cache so it tells me if something requested hits the lookup table or not.

Noticed that whenever someone enters the private group, whois cache is queried twice, plus an .xml download is also initiated (necessary or not). Just wondered if anyone had any idea why this is?

I did check all of my modules associated with private group have been changed to use the lookup features of the cache.

-jj-


	Logged

Khalem

BeBot Founder
Administrator
Grandmaster

Offline

Gender: Male

Posts: 670

Re: Timeouts with opening a site (e.g. people lookup)

« Reply #2 on: September 10, 2006, 08:29:38 AM »

0.3 already solves this and the code could be adapted.

Here is the 0.3 code for reference

Code:

<?php
/*
Gets a URL
Heavily inspired by Shadowgod's AOXML class and examples from php.net comments.
*/
function get_site($url, $server_timeout = 5, $read_timeout = 10)
{
/*
Parse the URL so we can use it in our raw socket request
*/
$get_url = parse_url($url);

/*
Open the socket
*/
$fd = fsockopen($get_url[host], 80, $errno, $errstr, $server_timeout);

/*
Make sure the socket was created successfully
*/
if (!$fd)
{
$return["error"] = true;
$return["errordesc"] = "Errno: $errno Errstr: $errstr";
return $return;
}
else
{
/*
Set the timeout to prevent it from hanging the bot
*/
socket_set_timeout($fd,$read_timeout);

/*
Send the HTTP request
*/
fputs($fd, "GET $get_url[path] HTTP/1.0\r\n");
fputs($fd, "Host: $get_url[host]\r\n");
fputs($fd, "Connection: Close\r\n");
fputs($fd, "User-Agent: BeBot/$bot_version\r\n\r\n");

/*
Check if the server is giving us what we wanted
*/
$http_response = fgets($fd);

$results = stream_get_meta_data($fd);
/*
Make sure we haven't timed out while waiting for a responce
*/
if ($results[timed_out] == 1)
{
$return["error"] = true;
$return["errordesc"] = "Timed out while reading from $get_url[host]";
$return["content"] = "";
fclose($fd);
return $return;
}

if ( ereg("200 OK", $http_response) )
{
$return["error"] = false;
$return["content"] = "";

/*
Read the contents
*/
while (!feof($fd))
{
$return["content"] .= fgets($fd, 1024);
}

$results = stream_get_meta_data($fd);
/*
Make sure we didn't time out while reading the responce again.
*/
if ($results[timed_out] == 1)
{
$return["error"] = true;
$return["errordesc"] = "Timed out while reading from $get_url[host]";
$return["content"] = "";
fclose($fd);
return $return;
}

fclose($fd);
return $return;
}
else
{
$return["error"] = true;
$return["errordesc"] = "Server returned: $http_response";
fclose($fd);
return $return;
}
}
}
?>


	Logged

BeBot Founder and Fixer Kingpin
Madman coder and destroyer of good code

porter

Rookie

Offline

Posts: 28

Re: Timeouts with opening a site (e.g. people lookup)

« Reply #3 on: September 10, 2006, 10:43:49 AM »

Ahh, thank you! I should have checked the development version. My solution does not work after all, don't try it anyone


	Logged

Khalem

BeBot Founder
Administrator
Grandmaster

Offline

Gender: Male

Posts: 670

Re: Timeouts with opening a site (e.g. people lookup)

« Reply #4 on: September 10, 2006, 10:59:16 AM »

The following code should work for 0.2.
Please provide feedback on it.

Code:

<?php
/*
Gets a URL
Heavily inspired by Shadowgod's AOXML class and examples from php.net comments.
*/
function get_site($url, $server_timeout = 5, $read_timeout = 10)
{
/*
Parse the URL so we can use it in our raw socket request
*/
$get_url = parse_url($url);

/*
Open the socket
*/
$fd = fsockopen($get_url[host], 80, $errno, $errstr, $server_timeout);

/*
Make sure the socket was created successfully
*/
if (!$fd)
{
return 0;
}
else
{
/*
Set the timeout to prevent it from hanging the bot
*/
socket_set_timeout($fd,$read_timeout);

/*
Send the HTTP request
*/
fputs($fd, "GET $get_url[path] HTTP/1.0\r\n");
fputs($fd, "Host: $get_url[host]\r\n");
fputs($fd, "Connection: Close\r\n");
fputs($fd, "User-Agent: BeBot/$bot_version\r\n\r\n");

/*
Check if the server is giving us what we wanted
*/
$http_response = fgets($fd);

$results = stream_get_meta_data($fd);
/*
Make sure we haven't timed out while waiting for a response
*/
if ($results[timed_out] == 1)
{
fclose($fd);
return 0;
}

if ( ereg("200 OK", $http_response) )
{
$content = "";

/*
Read the contents
*/
while (!feof($fd))
{
$content .= fgets($fd, 1024);
}

$results = stream_get_meta_data($fd);
/*
Make sure we didn't time out while reading the response again.
*/
if ($results[timed_out] == 1)
{
fclose($fd);
return 0;
}

fclose($fd);
return $content;
}
else
{
fclose($fd);
return 0;
}
}
}
?>


	Logged

BeBot Founder and Fixer Kingpin
Madman coder and destroyer of good code

porter

Rookie

Offline

Posts: 28

Re: Timeouts with opening a site (e.g. people lookup)

« Reply #5 on: September 10, 2006, 11:34:42 AM »

Yes, that works. I just edited it in and have so far no problems.


	Logged

Dabaron

Apprentice

Offline

Gender: Male

Posts: 145

Re: Timeouts with opening a site (e.g. people lookup)

« Reply #6 on: September 11, 2006, 02:36:02 PM »

using what Khalem posted if it was someone not in my whoiscache I got an insta "FC too slow to respond". It looked like it didn't even try and wait at all for a response. is the $server_timeout and $read_timeout in seconds or is that milliseconds?


	Logged

porter

Rookie

Offline

Posts: 28

Re: Timeouts with opening a site (e.g. people lookup)

« Reply #7 on: September 11, 2006, 04:10:57 PM »

Quote from: Dabaron on September 11, 2006, 02:36:02 PM

My "solution" did that instant give up even after the lookup server started to behave. Using Khalem's latter code with a 0.2.8 I have no problems at all so far. I have not added a whois cache so unless it is included in 0.2.8 that might be what we have different.

Timeouts in the script have to be seconds not ms - I can quote for a fact:

Quote

stream_set_timeout -- Set timeout period on a stream
Description
bool stream_set_timeout ( resource stream, int seconds [, int microseconds] )

Socket_set_timeout aliases to stream_set_timeout. The server timeout is only defined as float in the PHP documentation but a 5ms timeout in a web server context would make no sense, so I trust it to be seconds as well.


	Logged

Khalem

BeBot Founder
Administrator
Grandmaster

Offline

Gender: Male

Posts: 670

Re: Timeouts with opening a site (e.g. people lookup)

« Reply #8 on: September 12, 2006, 01:41:06 AM »

It is in seconds.

Use the following code to get debug info to console.
0.3 has get_site sending the errors back to the calling functions, but this can't be done in 0.2 without breaking backwards compactibility.

Code:

<?php
/*
Gets a URL
Heavily inspired by Shadowgod's AOXML class and examples from php.net comments.
*/
function get_site($url, $server_timeout = 5, $read_timeout = 10)
{
/*
Parse the URL so we can use it in our raw socket request
*/
$get_url = parse_url($url);

/*
Open the socket
*/
$fd = fsockopen($get_url[host], 80, $errno, $errstr, $server_timeout);

/*
Make sure the socket was created successfully
*/
if (!$fd)
{
echo "get_site debug: Creating socket failed\n";
return 0;
}
else
{
/*
Set the timeout to prevent it from hanging the bot
*/
socket_set_timeout($fd,$read_timeout);

/*
Send the HTTP request
*/
fputs($fd, "GET $get_url[path] HTTP/1.0\r\n");
fputs($fd, "Host: $get_url[host]\r\n");
fputs($fd, "Connection: Close\r\n");
fputs($fd, "User-Agent: BeBot/$bot_version\r\n\r\n");

/*
Check if the server is giving us what we wanted
*/
$http_response = fgets($fd);

$results = stream_get_meta_data($fd);
/*
Make sure we haven't timed out while waiting for a response
*/
if ($results[timed_out] == 1)
{
fclose($fd);
echo "get_site debug: Socket timed out while waiting for server response\n";
return 0;
}

if ( ereg("200 OK", $http_response) )
{
$content = "";

/*
Read the contents
*/
while (!feof($fd))
{
$content .= fgets($fd, 1024);
}

$results = stream_get_meta_data($fd);
/*
Make sure we didn't time out while reading the response again.
*/
if ($results[timed_out] == 1)
{
fclose($fd);
echo "get_site debug: Socket timed out while reading data\n";
return 0;
}

fclose($fd);
echo "get_site debug: Success\n";
return $content;
}
else
{
fclose($fd);
echo "get_site debug: Server error, http response was: $http_response\n";
return 0;
}