Using HttpWebResponse

    The HttpWebResponse. The server headers contain many useful pieces of information. Server headers are commonly used for:

• Determining the type of data at a URL
• Determining the cookies in use
• Determining the web server software in use
• Determining the size of the content at this URL

    For the bots that you create, you will most commonly use server headers to determine the type of data at a URL and to support cookies.

Reading Server Headers

    The HttpWebResponse class is a child class of the WebResponse class. For HTTP connections you can use either class. However, some of the HTTP options are only available with the HttpWebResponse. As a result, most of the examples in this book make use of the HttpWebResponse class.

    Once you retrieve the contents of a URL back from the server, there are headers available to the program. The web server provided this second set of headers. The Headers property of the HttpWebResponse class provides access to these headers. In the next section you will see how to access the headers.

MIME Types

    One very important HTTP response header is the content type. The content-type header tells the web browser what type of data the URL is attached to. For example, to determine the type of content at a URL, you would use the following line of code:

Console.WriteLine(response.Headers["Content-Type"]);

    This type of information is called a Multipurpose Internet Mail Extension (MIME). The “Mail” in MIME is largely historical. MIME types were originally developed for email attachments, long before there was a World Wide Web (WWW). However, they are now applied to many different Internet applications, such as web browsers and servers.

    A MIME type consists of two identifiers separated by a slash (/). For example, text/html is a mime type that identifies a resource as an HTML document. The first part of the type, in this case text, identifies the family of the type. The second identifies the exact type, within that family. Plain text files are also part of the text family, and have the type text/plain. Some of the common MIME types are summarized in Table 4.3.

Table 4.3: MIME Families

    There are many different MIME types under each of these families. However, there is only a handful that you will commonly see. Table 4.4 summarizes these.

Table 4.4: Common MIME Types

    Often, the program will only need to look at the family. For example, if you wanted to download text and binary files differently, you would simply look at the family part of the MIME type. If it is determined that text is the family, you may download the URL as a text file. Any other family would require downloading the information as a binary file. The difference between a binary file and a text file is that binary files are copied exactly to the hard drive, whereas text file’s line endings are reformatted properly for the resident operating system.

Calling Sequence

    As you have seen in this chapter, a variety of operations can be performed on the HttpWebRequest and HttpWebResponse classes. You can set request headers, read response headers, POST data and read response data. Please note, however, that these operations must follow a very specific order. For example, you can’t set a request header, after you are already reading the response. If you are reading the web server’s reply, the request has already been sent. Therefore, all request information must be set before you begin working with the response. The general order that you should follow is:

• Step 1: Obtain a HttpWebRequest object
• Step 2: Set any HTTP request headers.
• Step 3: POST data, if this is a POST request.
• Step 4: Obtain a HttpWebResponse object
• Step 4: Read HTTP response headers
• Step 5: Read HTTP response data

    If you ever face a bug where it seems the request headers are being ignored, check to see if you are not already calling a method related to the response before setting the header. All headers must be set before the request is sent.