Person Career

    The HttpWebResponse. The server headers contain many useful pieces of information. Server headers are commonly used for:

• Determining the type of data at a URL
• Determining the cookies in use
• Determining the web server software in use
• Determining the size of the content at this URL

    For the bots that you create, you will most commonly use server headers to determine the type of data at a URL and to support cookies.

    When you call the getTag function of the HTML parse class, you are given an HTMLTag object. This object completely encapsulates the HTML tag that was just parsed. The HTMLTag class is shown in Listing 6.3.

Listing 6.3: HTML Tags (HTMLTag.java)

    The ParseHTML class does HTML parsing. This class is used by all of the recipes in this chapter. Additionally, many recipes through the remainder of the book will use the ParseHTML class. I will begin by showing you how to use the ParseHTML class. A later section will show you how the ParseHTML class was implemented.

Using ParseHTML

    It is very easy to use the ParseHTML class. The following code fragment demonstrates how to make use of the ParseHTML class.

    To properly parse any data, let alone HTML, it is very convenient to have a peekable stream. A peekable stream is a regular Java InputStream, except that you can peek several characters ahead, before actually reading these characters. First we will examine why it is so convenient to use PeekableInputStream.

    Consider parsing the following the following line of HTML.

    This chapter includes two recipes. These two recipes will demonstrate the following:

• Determining if a URL uses HTTPS
• Using HTTP authentication

    The first recipe will introduce you to some of the things that can be done with the HttpsURLConnection class. The second recipe shows how to access a site that uses HTTP authentication.

    This chapter includes four recipes. These four recipes will demonstrate the following:

• Scanning a URL for headers
• Searching a range of IP addresses for web sites
• Downloading a binary or text file
• Monitoring a site to see that it stays up

    These recipes will introduce you to some of the things that can be done with the HttpURLConnection class.

    The server headers contain many useful pieces of information. Server headers are commonly used for:

• Determining the type of data at a URL
• Determining the cookies in use
• Determining the web server software in use
• Determining the size of the content at this URL

    For the bots that you create, you will most commonly use server headers to determine the type of data at a URL and to support cookies.

Teaser

Teaser