The Java bots created in this book are only half of the HTTP communication protocol. These bots must communicate with a web server. For example, a typical HTTP bot may access a popular online bookseller and obtain a price for a certain book. I could write an example bot that accesses Amazon.com and obtains this price. However, there are several problems with this.

• Amazon may prohibit bot access of their site
• Amazon may modify their site and break my bot

    Both issues are important. If the examples in this book were all written to use real-world web sites, a major site redesign to these sites could leave many of the book examples nonfunctional. One minor change to one of these sites, and the related examples in the book would immediately become out of date.

    Additionally, some sites do not allow access by bots. There are two main ways to stop bot access to a web site.

• Lawyers
• Technology

    Some sites specify in their Terms of Service (TOS) agreement that their site may not be accessed programmatically with a bot. From a technical standpoint, they may do nothing to actually prevent bots from accessing their site. They just reserve the right to take legal action for accessing their site programmatically. If you have any questions about the legality of accessing a particular site, always contact a lawyer. Such information is beyond the scope of this book.

    Because of these reasons, the book examples all make use of a special demonstration site owned by Heaton Research. The pages, used by the book’s bots, on the special web site will never change. This special web site contains many pages and applications that simulate many of the real-world web sites that bots may be written for. This web site can be accessed at the following URL:

http://www.httprecipes.com/

    If accessed with a browser, the recipe site will appear as Figure 1.1.

Figure 1.1: The HTTP Recipes Web Site

    As can be seen from Figure 1.1, the first thing the web site presents the surfer with is the edition of this book. If future editions of this book are released, in the years to come, the examples on this site for a particular edition will not change. New areas will be added for new editions. Since this is the first edition of the book, choose the first edition.

    The homepage for the first edition of this book is shown in Figure 1.2.

Figure 1.2: The Homepage for this Book

    From this page links are provided to all of the chapters from this book. Some of the examples will require access the site using a web browser. Other examples will use bots to access this site. Some examples may be a combination of both.

    The examples site will be put to use in the following sections.