Other Useful Options

    In addition to headers, there are that the HttpWebRequest and HttpWebResponse classes provide other useful options. Although there are a number of options in the HttpWebRequest and HttpWebResponse classes, most are used only for very rare or obscure situations. In this section we will examine the two most commonly used options in these classes. The two most frequently used are “timeouts” and “redirect following”. The next two sections will cover these options.

Timeout

    While connecting to a URL C# will only wait a specified number of seconds. One timeout value is provided to the HttpWebRequest class. The two timeouts that can occur are:

• Timeout while connecting to the web host.
• Timeout while transferring data with the web host.

    To control the timeout you should use the Timeout property of the HttpWebRequest object. This would be done as follows:

Request.Timeout = 1000;

    The above code would set the timeout to 1000 milliseconds, or one second.

Redirect Following

    One very handy feature in HTTP is “redirect following”. Many web sites make use of the HTTP redirect internally, so you will most likely encounter redirects when writing a bot. The HTTP redirect allows the server to redirect the web browser to a new URL.

    To see an HTTP redirect in action, enter the following URL into your web browser. You would expect the browser to take you to the URL you entered.

http://www.httprecipes.com/1/4/redirect.php

    However, you do not end up on the above URL. You actually end up at the root of the “Recipe Site” at the following URL:

http://www.httprecipes.com/

    This was due to an HTTP redirect. By default, the HttpWebRequest class will follow all such redirects automatically and often. You do not need to even be concerned with them. Web browsers will always follow redirects automatically. However, if you would like to handle the redirects yourself, you can disable auto following. The following line of code would do this:

request.AllowAutoRedirect = false;

    If you disable redirection following, you may manually follow the redirects by looking at the location response header.