Selected controversial TREC10 evaluations of Webclopedia


Question 991: What is cryogenics?
Qtargets: Q-DEFINITION
Qargs: cryogenics

First answer judged incorrect, last one correct and supported.

  • Webster entry for `cryogenics': the branch of physics that deals with very low temperatures
  • Scoring: Webclopedia highly favors the first answer because (1) it contains cryogenics (2) it finds a reliable apposition (3) definition candidate overlaps with WordNet gloss ("the branch of physics that studies the phenomena that occur at very low temperatures")
  • Scoring: Webclopedia gets a very low score for the last answer (in fallback mode): (1) it contains cryogenics (2) there are a couple of words around it with decent information content
  • Compare: What is physics? ... two guys who accidentally get freeze-dried in a physics lab ...
  • Compare: What is cryogenics? ... two guys who accidentally get lost in a cryogenics lab ...
  • Compare: Last answer (get freeze-dried ...) is judged supported, but answer to question 900 is judged unsupported.
    Question 900: George Bush purchased a small interest in which baseball team?
    Qtargets: S-NP, S-NOUN
    Judged correct, but unsupported.

    List question 2: Name 32 countries Pope John Paul II has visited.
    Qtargets: I-EN-PROPER-COUNTRY
    Qargs: John, type
    Qplural: 32

    All answers above judged unsupported.

  • Fact: Thanksgiving will be on November 22 this year.
  • Wishful thinking: Our brave Iraqi troops will crush the Americans in the mother of all battles.
  • Judged supported: What is cryogenics? ... two regular guys who accidentally get freeze-dried in a cryogenics lab and emerge 30 years later ...
    Question 1272: What does CPR stand for?
    Qtargets: Q-ABBREVIATION-EXPANSION
    Qargs: CPR
    First answer judged incorrect.


    Question 1262: What is the Moulin Rouge?
    Qtargets: Q-DEFINITION
    Qargs: Moulin Rouge, organization

    Both judged incorrect.

  • Google: Most top choices returned by Google identify Moulin Rouge as a movie.
  • Google: Large number of web pages identify the Moulin Rouge as a nightclub in Paris.
    Question 1300: What is carbon dioxide?
    Qtargets: Q-DEFINITION
    Qargs: carbon dioxide|dioxide

    All judged incorrect.

  • WordNet: a heavy odorless colorless gas formed during respiration and by the decomposition of organic substances
  • Webster: a colorless, odorless, incombustible gas, CO2, present in the atmosphere and ...
    (smaller font after 50 bytes)
    Question 1256: What is the only artery that carries blue blood from the heart to the lungs?
    Qtargets: S-NP, S-NOUN
    Qargs: artery

    Judged correct, but unsupported.


    Question 1387: What year did the Titanic start on its journey?
    Qtargets: C-TEMP-LOC-WITH-YEAR, I-ENUM-CARDINAL (0.5)

    Judged unsupported.

  • Note: Answer sentence contains `on its maiden voyage'.