Homework 6: Advanced Theseus

This homework will take you beyond the basics and give you a chance to write Theseus plans that require using wrappers, the Xquery operator, and subplans.

IMPORTANT: You will need to download the newest version of Theseus, which has some important bugfixes.

You are going to write a plan that, given a zip code, a business category, and a business subcategory, does the following:

1. Uses Yahoo Yellow Pages to find all businesses in that zip code which match the business category and subcategory specified.
2. Takes the resulting addresses and uses an online Geocoder to determine the polar coordinates (i.e., the lat and lon) of those locations.
3. Takes the resulting coordinates and uses an online source called a "Building Finder" to find the URLs for satellite images of each of the businesses

• Wrap the "Buiding Finder" source to extract satellite imagery for the given coordinates. The Building Finder is located at http://apollo.isi.edu/BuildingFinder/WebForm1.aspx. The input to the building finder ar size of the image, scale, and coordinates. Keep the size of the image and scale constant and the coordinates should be the parameters to the wrapper.  

• Modify your agent for Yahoo Yellow pages, so the agent accepts zipcode, category of business and a subcategory of business as input. (Hint: in your existing yahoo business agent, one wrapper extracts list of categories and another wrapper extracts list of subcategories. Create filters on the list to filter out categories and subcategories that don't match the input parameters.  

• Host Geocoder, Building Finder, and Yahoo Yellow pages agents on your computer (more info on this will be provided in class).

• Write four Theseus plans that help accomplish the task above:  
  • The first plan will be called get_businesses.plan. This plan will take a streamed relation with the schema (zip, biz_cat, biz_subcat) and call the Yahoo Yellow Pages Wrapper to find the businesses in a particular zip code that match specified category and subcategory information. This plan will return a streamed relation with the schema (name, streetaddress, city, state, zip).  

  • The second plan will be called get_coordinates.plan. This plan will take a streamed relation with the schema (name, streetaddress, city, state, zip) and use the Geocoder Wrapper to find the polar coordinates of each of the businesses found. This plan will return a streamed relation consisting of the schema (name, streetaddress, city, state, zip, lat, lon).  

  • The third plan will be called get_imageurls.plan. This plan will take a streamed relation with the schema (name, streetaddress, city, state, zip, lat, lon) and use the Building Finder Wrapper to find the URLs for satellite images of the geocoded businesses. This plan will return a streamed relation with a schema (name, streetaddress, city, state, zip, lat, lon, imageurl).  

  • The fourth plan will be called hw6.plan. This plan will act as a parent plan which calls the other three plans as subplans.

The benefit of using subplans is that your development becomes better modularized. Thus, if you wanted to write a different plan that gathered businesses, geocoded them, and plotted them on a Yahoo map, you would quickly be able to reuse the first two plans as subplans in this new plan.

There are some important notes regarding various parts of this assignment:

• You will need to use the Xwrapper and Xquery operators in this assignment. Make sure that you read the online Theseus documentation to understand how to use these opertors.  

• The first subplan, get_businesses.plan, will require more thought than the other subplans. This is because the XML returned from the xwrapper call is more complicated. You should take a close look at the XML output to identify what part you really want. Then, you should use Xquery to help you reformat the document. In particular, you are going to want to take the result that Xwrapper originally gives you from Yahoo Yellow Pages and use the Xquery operator to convert this document to a form that looks like this:

  <?xml version="1.0" encoding="UTF-8"?>
          <Data>
  
                  <Row>
                          <name id="6.3" type="data">
                                  <Value>Arico's Hallmark</Value>
                          </name>
                          <streetaddress id="6.4" type="data">
  
                                  <Value>13175 Mindanao Way</Value>
                          </streetaddress>
                          <city id="6.5" type="data">
                                  <Value>Marina Del Rey</Value>
                          </city>
  
                          <state id="6.6" type="data">
                                  <Value>CA</Value>
                          </state>
                          <phone id="6.7" type="data">
                                  <Value>(310) 823-3739</Value>
  
                          </phone>
                          <nextlink id="6.56" type="data">
                                  <Value>
                                  </Value>
                          </nextlink>
                  </Row>
  
                  <Row>
                          <name id="6.9" type="data">
                                  <Value>Beyond Scents</Value>
                          </name>
  
                          ... etc. ...
  
  

  This will make it possible for you to use the Theseus xml2rel operator to convert this data from XML to a relation (which you will need for the rest of the plan).  

• Be very careful to watch how the wrappers name their attributes. You will tend to see that if you use xml2rel on the output from the Geocoder wrapper, for example, your relational schema will include pairs like lat and lat_value. One is just a label and one is the actual value.  

Your hw6.plan should have the following interface:

PLAN hw6
{
  INPUT: stream criteria
  OUTPUT: stream imageurls

  BODY 
  {
    /* You will write this! */
  }
}

RELATION criteria: zip char, biz_cat char, biz_subcat char
90292|Retail Shopping|Gifts and Cards

----------------------------------------------
RELATION: hw6_imageurls
   attrs: name, streetaddress, city, state, zip, imageurl, lat, lon
----------------------------------------------
Arico's Hallmark|13175 Mindanao Way|Marina Del Rey|CA|90292|http://apollo.isi.edu/BF/Find.aspx?x=1835&y=18808&z=11&Scale=10&width=2&height=2&toplon=-118.439953394851&toplat=33.9883621706051&botlon=-118.435563250856&botlat=33.9848063332803&displayHouses=1&displayRoads=1|33.9865842519427|-118.437758322853
Girl From Ipanema|103 Washington Blvd|Venice|CA|90292|http://apollo.isi.edu/BF/Find.aspx?x=1822&y=18806&z=11&Scale=10&width=2&height=2&toplon=-118.46803281021&toplat=33.984423152998&botlon=-118.463641742986&botlat=33.9808683410465&displayHouses=1&displayRoads=1|33.9826457470222|-118.465837276598
Heather's Flowers|101 Washington Blvd|Venice|CA|90292|http://apollo.isi.edu/BF/Find.aspx?x=1822&y=18806&z=11&Scale=10&width=2&height=2&toplon=-118.46803281021&toplat=33.984423152998&botlon=-118.463641742986&botlat=33.9808683410465&displayHouses=1&displayRoads=1|33.9826457470222|-118.465837276598
Beyond Scents|4710 Admiralty Way|Marina Del Rey|CA|90292|http://apollo.isi.edu/BF/Find.aspx?x=1834&y=18806&z=11&Scale=10&width=2&height=2&toplon=-118.4420572117&toplat=33.9847303823688&botlon=-118.437667175289&botlat=33.9811746155411&displayHouses=1&displayRoads=1|33.9829524989549|-118.439862193494
Relish Gift Designs Inc|4223 Glencoe Ave|Marina Del Rey|CA|90292|http://apollo.isi.edu/BF/Find.aspx?x=1833&y=18810&z=11&Scale=10&width=2&height=2&toplon=-118.444343904389&toplat=33.9919178397541&botlon=-118.439953394851&botlat=33.9883621706051&displayHouses=1&displayRoads=1|33.9901400051796|-118.44214864962
Marina Del Rey Florist|4072 1/2 Lincoln Blvd|Marina Del Rey|CA|90292|http://apollo.isi.edu/BF/Find.aspx?x=1712&y=0&z=11&Scale=10&width=2&height=2&toplon=-118.416166225139&toplat=0.0018089063233339&botlon=-118.412572628698&botlat=-0.00180890914347243&displayHouses=1&displayRoads=1|-1.4100692661027E-09|-118.414369426919
Jeanne's Hallmark|4726 Admiralty Way|Marina Del Rey|CA|90292|http://apollo.isi.edu/BF/Find.aspx?x=1834&y=18805&z=11&Scale=10&width=2&height=2&toplon=-118.442026753043&toplat=33.9829271666497&botlon=-118.437636813397&botlat=33.9793713952946&displayHouses=1&displayRoads=1|33.9811492809722|-118.43983178322
Star Lane|13737 Fiji Way|Marina Del Rey|CA|90292|http://apollo.isi.edu/BF/Find.aspx?x=1712&y=0&z=11&Scale=10&width=2&height=2&toplon=-118.416166225139&toplat=0.0018089063233339&botlon=-118.412572628698&botlat=-0.00180890914347243&displayHouses=1&displayRoads=1|-1.4100692661027E-09|-118.414369426919
Marina Sensations|13755 Fiji Way|Marina Del Rey|CA|90292|http://apollo.isi.edu/BF/Find.aspx?x=1712&y=0&z=11&Scale=10&width=2&height=2&toplon=-118.416166225139&toplat=0.0018089063233339&botlon=-118.412572628698&botlat=-0.00180890914347243&displayHouses=1&displayRoads=1|-1.4100692661027E-09|-118.414369426919
----------------------------------------------

You should turn in the following:

• hw6.plan, get_businesses.plan, get_coodinates.plan, get_imageurls.plan
• Text file containing the output produced when running hw6.plan
• NOTE: you can use any business category/subcategory pair you want, as long as it produces results (it is strongly suggested that start with something small, such as the example provided above, and make sure it works before attempting a category/subcategory with more data).

If you have any, contact Greg Barish.