These guidelines are the outcome of an international effort on evaluation of Machine Translation and other NLP/CL applications. Earlier related work was performed in the EAGLES Project, funded by the European Community and the Swiss Government, and in the JEIDA Project in Japan. FEMTI was mainly funded through the ISLE Project by the US Government National Science Foundation and the Swiss and Danish Governments.

ISLE Evaluation Working Group

The heads of participating teams were: Eduard Hovy (Information Sciences Institute, University of Southern California, Marina del Rey, CA, USA), Maghi King (ISSCO/TIM/ETI, University of Geneva, Switzerland), Bente Maegaard (Center for Sprogteknologi, Copenhagen, Denmark), and Martha Palmer (Department of Computer Science, University of Pennsylvania, Philadelphia, PA, USA).

Website

The first version of this website was designed by Eduard Hovy (USC/ISI) and Elena Filatova (then at USC/ISI, now at Columbia University). The website is now maintained by Andrei Popescu-Belis (ISSCO/TIM/ETI, University of Geneva), based on XML-encoded data.

Workshops

The FEMTI framework has received considerable input from the participants to the workshops organized through the ISLE Project, listed here starting with the most recent ones. Version 3 of the FEMTI guidelines is the result of collaborative work at the Marina del Rey workshop (February 2003).

• "Towards Systematizing MT Evaluation" - a Workshop at MT Summit IX (New Orleans, Louisiana, USA, Sept. 2003):
   http://www.issco.unige.ch/projects/isle/MTE-at-MTS9.html.

Organizers: Leslie Barett, Maghi King, Keith Miller, Andrei Popescu-Belis. Program committee: Leslie Barrett, Bonnie Dorr, Eduard Hovy, Maghi King, Keith Miller, Ted Pedersen, Andrei Popescu-Belis, Florence Reeder, Nancy Underwood, Michelle Vanni. Authors of presented papers: Necip Fazil Ayan, Bonnie J. Dorr, Okan Kolakonnie; Leslie Barrett; David Farwell, Stephen Helmreich; Eva Forsbom; Florence Reeder, John S. White; Calandra Tate, Sooyon Lee, Clare R. Voss.

• Workshop on the Evaluation of Machine Translation at USC/ISI (Marina del Rey, Feb. 2003):

Organizers: Tony Hartley (U.Leeds), Eduard Hovy (USC/ISI), Maghi King (ISSCO, U.Geneva), Keith Miller (MITRE), Andrei Popescu-Belis (ISSCO, U.Geneva), Florence Reeder (MITRE), Nancy Underwood (ISSCO, U.Geneva), John White (Northrop-Grumman), Sandrine Zufferey (U.Geneva). Guests: Bonnie Dorr (U.Maryland), David Farwell (New Mexico State U.), Hitoshi Isahara (CRL, Kyoto), Daniel Marcu (USC/ISI), Doug Oard (U.Maryland, visiting USC/ISI), Franz Och (USC/ISI), Kishore Papineni (IBM T.J.Watson Research Center), Kathryn B. Taylor (U.S. Government), Michelle Vanni (U.S. Government), Charles Wayne (U.S. Government).

• MT Evaluation: "Human Evaluators Meet Automated Metrics" - a Workshop at LREC 2002 (Las Palmas, Canary Islands):
   http://www.issco.unige.ch/projects/isle/mteval-may02/.

Speakers: Andrei Popescu-Belis, Florence Reeder, George Doddington, Marianne Dabbadie & Widad Mustafa El Hadi, Bonnie Dorr, Eva Forsbom, Tony Hartley, Keith Miller, Cristina Vertan.

• MT Evaluation Workshop "Who Did What to Whom" at MT Summit VIII (Santiago de Compostela, Spain):
   http://issco-www.unige.ch/projects/isle/MT-Summit-wsp.html.

Authors of presented papers: Andrei Popescu-Belis, Sandra Manzi, Maghi King; Elia Yuste-Rodrigo, Francine Braun-Chen; Keith Miller, Donna M. Gates, Josemina Magdalen, Nancy L. Underwood; Michelle Vanni, Keith Miller; Martin Rajman, Anthony Hartley; John S. White; Palmira Marrafa, Antonio Ribeiro; Monika Forner, John S. White; Widad Mustafa El Hadi, Ismail Timimi, Marianne Dabbadie; Florence Reeder, Keith Miller, Jennifer Doyon, John S. White; Christine Bruckner, Mirko Plitt; Florence Reeder.

• Workshop on MT Evaluation at NAACL 2001 (Pittsburgh):
   http://www.cs.cmu.edu/~ref/naacl2001.html

Organizers: Florence Reeder, Eduard Hovy.

• MT-Eval Workshop "An Invitation to get Your Hands Dirty", University of Geneva:
   http://www.issco.unige.ch/projects/isle/mteval-april01

Organizers: Maghi King, Florence Reeder, Sandra Manzi, Andrei Popescu-Belis.

• Hands on MT Evaluation Workshop at AMTA 2000 (Mexico):
   http://www.isi.edu/natural-language/conferences/AMTA2000.html

Organizers: Florence Reeder, Eduard Hovy. Reviewing Committee: Michelle Vanni, Keith Miller, Jack Benoit, Maghi King, Carol Van Ess-Dykema, John White.

• Workshop on the Evaluation of Machine Translation at LREC 2000 (Athens):
   http://www.lrec-conf.org/lrec2000/www.icp.inpg.fr/ELRA/lrec2000.html

Organizers: Bente Maegaard, Maghi King, Eduard Hovy. Authors of presented papers: Bente Maegaard; John White; Remi Zajac; Lynette Hirschman, Florence Reeder, John Burger and Keith Miller; D. Cavar, Uwe Küssner, Dan Tidhar; S. Vogel, Sonja Nießen, Hermann Ney; L. Ahrenberg, Magnus Merkel; E. Hovy