Reading from URLs

    C# allows you to read data from URLs. This forms the basis of HTTP programming in C#. In this chapter you will learn how to construct simple requests from web sites. What is a simple request? A simple request is a request where you only request data from a URL. It can get much more complex than that. As you progress through the book you will learn more complex HTTP programming topics such as:

• HTTPS
• Posting Data
• Cookies
• Authentication
• Content Types

    For now, we will focus on getting data from a URL and leave the more complex operations for later. You need to conduct three basic steps to read data from a URL. A summary of these steps is:

• Create a Uri Object
• Open a stream
• Read data from the stream

    We will begin by examining the C# Uri class.

The Uri Class

    C# provides a class to hold URLs. Even though a URL is actually a string, it is convenient to have the Uri class. The Uri class has several advantages over strings. There are methods to:

• Determine if the Uri is valid
• Extract information, such as host or schema
• Open a connection to the Uri

    To create a Uri object, simply pass the string URL string to the constructor of the Uri class, as follows:

Uri url = new Uri("http://www.httprecipes.com");

    This will create a new Uri object that is ready to be used. However, it can throw a checked exception, named UriFormatException. As mentioned previously, to handle the exception, use the following code.

try
{
Uri url = new Uri("http://www.httprecipes.com");
}
catch(UriFormatException e)
{
Console.WriteLine("This Uri is not valid.");
}

    The UriFormatException will be thrown if the URI is invalid. For example, a URL such as the following would throw the exception:

http:////www.httprecipes.com/

    The above URL would throw the exception, because the URL has four slashes (////), which is not valid. It is important to remember that the Uri class only checks to see if the URL is valid. It does NOT check to see if the URL exists on the Internet. Existence of the URL will not be verified until a connection is made. The next section discusses how to open a connection.

Opening the Stream

    C# uses streams to access files and perform other I/O operations. When you access the URI, you will be given a Stream. This Stream is used to download the contents of the URL. The Uri class makes it easy to open a stream for the URL.

    To open a stream, you must first obtain a WebRequest object. This object can be obtained by calling the Create function of the HttpWebRequest class. The Create function accepts a URI to specify which page will be downloaded.

    From the WebRequest object you can obtain an HttpWebResponse object. The HttpWebResponse object allows you to obtain a stream by calling GetResponseStream. The following code shows how this is done.

try
{
WebRequest http = HttpWebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)http.GetResponse();
Stream stream = response.GetResponseStream();
}
catch(UriFormatException e)
{
Console.WriteLine("Invalid URL");
}
catch(IOException e)
{
Console.WriteLine("Could not connect to URL");
}

    As you can see, the above code is similar to the code from the last section. However, an additional line follows the Uri declaration. This line calls the GetResponseStream function, and receives a Stream object. You will see what to do with this object in the next section.

    The above code also has to deal with an additional exception. The IOException can be thrown by the GetResponseStream function, so it is necessary to catch the exception. Remember from the previous section that the constructor of the Uri class does not check to see if a URI actually exists. The URI is checked at this point. If the URI does not exist or if there is any trouble connecting to the web server that holds that URI, then an IOException will be thrown.

    Now that you have constructed the Uri object and opened a connection, you are ready to download the data from that URI.

Downloading the Contents

    Downloading the contents of a web page uses the same procedure you would use to read data from any input stream. If you want to read the entire contents of the URI you should use a StreamReader. The StreamReader class provides the ReadToEnd function that will until the end of the stream has been reached.

Uri uri = new Uri("http://www.httprecipes.com");
WebRequest http = HttpWebRequest.Create(uri);
HttpWebResponse response = (HttpWebResponse)http.GetResponse();
StreamReader stream = new StreamReader(response.GetResponseStream(),System.Text.Encoding.ASCII );

String result = stream.ReadToEnd();
Console.WriteLine( result ); 

response.Close();
stream.Close();
return result;

    As you can see, the above code continues what we have already seen. Just as in the previous code segments, the Uri object is first created. Next a stream is opened to the URL. Finally, the entire contents of the URI are read to a string.