The Framework for Machine Translation Evaluation in ISLE is an attempt to organize the various methods that are used to evaluate MT systems, and to relate them to the purpose and context of the systems. Therefore, FEMTI is made of two interrelated classifications or taxonomies. The first classification enables evaluators to define an intended context of use for the MT system to evaluate. Each feature is then linked to relevant quality characteristics and metrics, defined in the second classification.

To start browsing the framwork, click on CLASSIFICATION in the menu above. You need to expand the overview of the classification in order to have access to the links, or you may download a printable version of FEMTI. You can also find, in the above menu, more information about the authors, a glossary, and a detailed list of references. Feedback can be sent using the comments link.

• A classification of the main features defining the context of use (part 1), that is: the type of user of the MT system, the type of task the system is used for, and the nature of the input to the system.

• A classification of the MT software quality characteristics (part 2), into hierarchies of sub-characteristics, with internal and/or external attributes (i.e., metrics) at the bottom level. The upper levels match the ISO/IEC 9126 characteristics.

• A mapping from the first classification to the second, which defines or suggests the quality characteristics, sub-characteristics and attributes/metrics that are relevant to each context of use.

• People who want to use an MT system. They can select the quality characteristics that are most important to them and thereby choose the MT system that best suits these characteristics.

• People who want to compare several MT systems. They can browse and select the characteristics that best reflect their circumstances, and thereby find associated evaluation measures and tests.

• People who want to design a new MT system or to upgrade an old one. They can learn about the needs of users and find niche applications for their system.

FEMTI is still under work: some parts need completion, while others are updated based on feedback from the community. The taxonomy can be found at two mirrored web sites: http://www.issco.unige.ch/projects/isle/femti/ and http://www.isi.edu/natural-language/mteval/.

Apart from papers presented at MT evaluation workshops and other conferences, the most detailed publication about FEMTI is: Eduard Hovy, Margaret King & Andrei Popescu-Belis (2002) - Principles of Context-Based Machine Translation Evaluation. Machine Translation, 16, pp. 1-33. This article describes the principles underlying FEMTI, overviews previous work, and offers research perspectives for the future.