Home › HTTP Programming Recipes for C# Bots › Chapter 5: Secure HTTP Requests ›

Recipes

jeffheaton's picture
Submitted by jeffheaton on Wed, 01/09/2008 - 22:39
in
  Share
   
 
 
 

    This chapter includes two recipes. These two recipes will demonstrate the following:

  • Determining if a URL uses HTTPS, and
  • Using HTTP authentication

    The first recipe will introduce you to some of the things that can be done with the HttpWebRequest class. The second recipe shows how to access a site that uses HTTP authentication.

Recipe #5.1

    This recipe is a fairly simple example of using the HttpWebRequest class. This example simply checks a URL connection to see if it is HTTPS or not. I use this code when I need to be sure that a connection is being made over a secure link. Unless you care about examining HTTPS certificate chains and cipher suites directly, this is probably the only use you will have for the HttpWebRequest object.

    This program is fairly short, so it is implemented entirely in the main method. This HTTPS checker program is shown below in Listing 5.1.

Listing 5.1: Is a Connection HTTPS (IsHTTPS.java)

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Net;

namespace Recipe5_1
{
    /// <summary>
    /// Recipe #5.1: Is URL HTTPS?
    /// Copyright 2007 by Jeff Heaton(jeff@jeffheaton.com)
    ///
    /// HTTP Programming Recipes for C# Bots
    /// ISBN: 0-9773206-7-7
    /// http://www.heatonresearch.com/articles/series/20/
    ///     
    /// This recipe shows how to determine if a URL is using
    /// the HTTPS protocol.
    ///     
    /// This software is copyrighted. You may use it in programs
    /// of your own, without restriction, but you may not
    /// publish the source code without the author's permission.
    /// For more information on distributing this code, please
    /// visit:
    /// http://www.heatonresearch.com/hr_legal.php
    /// </summary>
    class Program
    {
        /// <summary>
        /// The main entry point for the program.
        /// </summary>
        /// <param name="args">Program arguments.</param>
        static void Main(string[] args)
        {
            String strURL = "";

            // obtain a URL to use
            if (args.Length < 1)
            {
                strURL = "https://www.httprecipes.com/1/5/";
            }
            else
            {
                strURL = args[0];
            }

            Uri url = new Uri(strURL);
            WebRequest http = HttpWebRequest.Create(url);
            http.PreAuthenticate = true;
            HttpWebResponse response = (HttpWebResponse)http.GetResponse();

            Console.WriteLine(response.ToString());

            if (String.Compare(url.Scheme,"https",true)==0 )
            {
                Console.WriteLine("Successful HTTPS connection");
            }
            else
            {
                Console.WriteLine("Successful HTTP connection");
            }
        }
    }
}

    This program can be passed a URL to access on the command line; or, if no URL is provided, the program will default to the following URL:

https://www.httprecipes.com/1/5/secure/

    You can also specify a URL. For example, to run this recipe with the URL of http://www.httprecipes.com/
, use the following command:

IsHTTPS http://www.httprecipes.com

    This program begins by obtaining a URL. The URL is either provided from the command line, or defaults to a page on the HTTP Recipes site. The following lines of code do this.

Uri url = new Uri(strURL);
WebRequest http = HttpWebRequest.Create(url);
http.PreAuthenticate = true;
HttpWebResponse response = (HttpWebResponse)http.GetResponse();

    At this point a connection will either have been made or an exception thrown. We can now check to see if this is an HTTPS connection or not.

Console.WriteLine(response.ToString());

if (String.Compare(url.Scheme,"https",true)==0 )
{
Console.WriteLine("Successful HTTPS connection");
}
else
{
Console.WriteLine("Successful HTTP connection");
}

    This recipe will be rarely used; however, it demonstrates one practical purpose of using HTTPS directly.

Recipe #5.2

    The next recipe is very useful if you need to retrieve data from a site that requires HTTP authentication. First, the code downloads the contents of a URL and then saves it to a local file. This recipe can be used either in whole or in part.

    Used in its entirety, this recipe provides a method, named Download. This method requires the URL to download the local file to save to, and the user id and password. The Download method will then download the contents. If the user id or password is invalid, an exception will be thrown.

    If your application doesn’t call for downloading directly to a file, you could use part of this recipe. Perhaps your application needs only to parse data at an HTTP authenticated site. If this is the case, use the code in the Download method that sets up the NetworkCredential.

    I have used this recipe in both ways for several bots that required HTTP authenticated access. The program is shown in Listing 5.2.

Listing 5.2: Download Authenticated URL (AuthDownloadURL.cs)

using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;

namespace Recipe5_2
{
    /// <summary>
    /// Recipe #5.2: Downloading a URL(text or binary)
    /// Copyright 2007 by Jeff Heaton(jeff@jeffheaton.com)
    ///
    /// HTTP Programming Recipes for C# Bots
    /// ISBN: 0-9773206-7-7
    /// http://www.heatonresearch.com/articles/series/20/
    /// 
    /// This recipe shows how to download a text or binary file,
    /// using HTTP authentication.
    /// 
    /// This software is copyrighted. You may use it in programs
    /// of your own, without restriction, but you may not
    /// publish the source code without the author's permission.
    /// For more information on distributing this code, please
    /// visit:
    ///    http://www.heatonresearch.com/hr_legal.php
    /// </summary>
    class AuthDownloadURL
    {
        /// <summary>
        /// Download the specified text page.
        /// </summary>
        /// <param name="response">The HttpWebResponse to download from.</param>
        /// <param name="filename">The local file to save to.</param>
        public void DownloadBinaryFile(HttpWebResponse response, String filename)
        {
            byte[] buffer = new byte[4096];
            FileStream os = new FileStream(filename, FileMode.Create);
            Stream stream = response.GetResponseStream();

            int count = 0;
            do
            {
                count = stream.Read(buffer, 0, buffer.Length);
                if (count > 0)
                    os.Write(buffer, 0, count);
            } while (count > 0);

            response.Close();
            stream.Close();
            os.Close();
        }

        /// <summary>
        /// Download the specified text page.
        /// </summary>
        /// <param name="response">The HttpWebResponse to download from.</param>
        /// <param name="filename">The local file to save to.</param>
        public void DownloadTextFile(HttpWebResponse response, String filename)
        {
            byte[] buffer = new byte[4096];
            FileStream os = new FileStream(filename, FileMode.Create);
            StreamReader reader = new StreamReader(response.GetResponseStream(), System.Text.Encoding.ASCII);
            StreamWriter writer = new StreamWriter(os, System.Text.Encoding.ASCII);

            String line;
            do
            {
                line = reader.ReadLine();
                if (line != null)
                    writer.WriteLine(line);

            } while (line != null);

            reader.Close();
            writer.Close();
            os.Close();
        }


        /// <summary>
        /// Download either a text or binary file from a URL.
        /// The URL's headers will be scanned to determine the
        /// type of tile.  The user id and password are authenticated.
        /// </summary>
        /// <param name="remoteURL">The URL to download from.</param>
        /// <param name="localFile">The local file to save to.</param>
        /// <param name="uid">The user id to use.</param>
        /// <param name="pwd">The password to use.</param>
        public void Download(Uri remoteURL, String localFile,String uid,String pwd)
        {
            NetworkCredential networkCredential = new NetworkCredential(uid, pwd);
            WebRequest http = HttpWebRequest.Create(remoteURL);
            http.PreAuthenticate = true;
            http.Credentials = networkCredential;
            HttpWebResponse response = (HttpWebResponse)http.GetResponse();

            String type = response.Headers["Content-Type"].ToLower().Trim();
            if (type.StartsWith("text"))
                DownloadTextFile(response, localFile);
            else
                DownloadBinaryFile(response, localFile);

        }


        /// <summary>
        /// The main entry point for the program.
        /// </summary>
        /// <param name="args">Program arguments.</param>
        static void Main(string[] args)
        {
            AuthDownloadURL d = new AuthDownloadURL();
            if (args.Length != 4)
            {
                d.Download(new Uri("https://www.httprecipes.com/1/5/secure/"),
                    "test.html",
                    "testuser",
                    "testpassword");
            }
            else
            {

                d.Download(new Uri(args[0]), args[1], args[2], args[3]);
            }
        }
    }
}

    This program can be passed a URL, local filename, user id and password on the command line. If no parameters are provided, the program will default to the following URL:

    The program will also default to a local file of test.html, a user id of testuser, and a password of testpassword.

    You can also specify a URL. For example, to run this recipe with the URL of http://www.httprecipes.com/
, a local file of index.html, a user id of user and a password of password, use the following command:

AuthDownloadURL http://www.httprecipes.com ./index.html user password

    This recipe is very similar to Recipe 4.3, except that it can download from an HTTP authenticated site as well as a regular site. Only the new code relating to HTTP authentication will be discussed here. If you would like to review how this recipe actually downloads a binary or text file, refer to Recipe 4.3. The downloading code is the same in both recipes.

    To access an HTTP authenticated site, the program uses the NetworkCredential class. This Download method begins by creating a NetworkCredential object and using it to connect to the web site. 

NetworkCredential networkCredential = new NetworkCredential(uid, pwd);
WebRequest http = HttpWebRequest.Create(remoteURL);
http.PreAuthenticate = true;
http.Credentials = networkCredential;
HttpWebResponse response = (HttpWebResponse)http.GetResponse();

    From this point forward this recipe is the same as Recipe 4.3. The Content-Type header is checked to see if this is a text or binary file. If this is a text file, then it is downloaded with DownloadTextFile. If this is a binary file, then it is downloaded with DownloadBinaryFile.

String type = response.Headers["Content-Type"].ToLower().Trim();
if (type.StartsWith("text"))
DownloadTextFile(response, localFile);
else
DownloadBinaryFile(response, localFile);

    This recipe can be used as a starting point for any operation requiring HTTP authentication. It does not matter to this recipe if HTTPS is used or not. This recipe supports either HTTP or HTTPS.

Copyright 2005 - 2012 by Heaton Research, Inc.. Heaton Research™ and Encog™ are trademarks of Heaton Research. Click here for copyright, license and trademark information.