web server

    The HttpWebResponse. The server headers contain many useful pieces of information. Server headers are commonly used for:

• Determining the type of data at a URL
• Determining the cookies in use
• Determining the web server software in use
• Determining the size of the content at this URL

    For the bots that you create, you will most commonly use server headers to determine the type of data at a URL and to support cookies.

    In this chapter, network analyzers were demonstrated. A network analyzer shows exactly what data is being exchanged between a web server and a web browser. There are many different network analyzers to choose from. The network analyzer that will be used by this book is WireShark. WireShark was formerly known as Ethereal.

    This chapter includes two recipes. These two recipes demonstrate how to examine two very important request items for bots:

• Cookies
• Forms

    Cookies and forms are used by many websites. This book has an entire chapter devoted to each. Chapter 7, “Responding to Forms” discusses HTML forms. Chapter 8, “Handling Sessions and Cookies” discusses cookies. For now how to examine cookies in a request will be explained.

    WireShark is one of the most popular network analyzers available. WireShark was once known by the name Ethereal, but due to copyright issues changed their name to WireShark. WireShark can be downloaded from the following web site:

http://www.wireshark.org/

    WireShark supports a wide variety of operating systems. To use WireShark, choose the version for the operating system, then download and install that version.

    A network analyzer is a program that allows TCP/IP traffic between the web server and a web browser to be monitored. With a network analyzer, a typical web browser accessing the desired web server can be monitored. This shows exactly what information is transmitted.

    This book is about how to write programs that browse the web, just as a human does. Such programs are called bots. To create bots, it is important to review how web browsing works at a technical level, to form a foundation for the rest of the book. In this chapter, the general structure of web browsing and HTTP requests were explored. The two recipes in this chapter were both web servers. The first web server did nothing more than displays the text “Hello World” to any request; however, the second example implemented a full file based web server.

    In this chapter, the structure of HTTP requests and responses was examined. As shown in Listing 1.3 and 1.4, the structure of HTTP is not terribly complex. As a result, it is relatively easy to create a web server. This is what the two recipes in this chapter will deal with. The first Recipe, 1.1, will show how to create a really simple web server. Next, Recipe 1.2 will show how to extend the simple web server to use HTML and image files, just as a regular web server would.

    As a user uses the web browser there is considerable network activity occurring to support the browsing experience. The Hyper Text Transport Protocol (HTTP) is what allows this to happen. HTTP specifies how web browsers and web servers manage the flurry of requests and responses that occur while a web user is surfing the web. Once it understood how web browsers and servers communicate, the built in HTTP classes provided by C#, can be used to obtain information from a web server programmatically.

    The server headers contain many useful pieces of information. Server headers are commonly used for:

• Determining the type of data at a URL
• Determining the cookies in use
• Determining the web server software in use
• Determining the size of the content at this URL

    For the bots that you create, you will most commonly use server headers to determine the type of data at a URL and to support cookies.

• The HttpURLConnection Class
• Reading HTTP Response Headers
• Setting HTTP Request Headers
• Managing HTTP Timeouts