1 Evaluation requirements

• 1.1 Purpose of evaluation

  • 1.1.1 Feasibility evaluation

  • 1.1.2 Requirements elicitation

  • 1.1.3 Internal evaluation

  • 1.1.4 Diagnostic evaluation

  • 1.1.5 Declarative evaluation

  • 1.1.6 Operational evaluation

  • 1.1.7 Usability evaluation

• 1.2 The object of evaluation

  • 1.2.1 Component of an MT system

  • 1.2.2 MT system considered as a whole

  • 1.2.3 MT system considered as a component of a larger system

• 1.3 Characteristics of the translation task

  • 1.3.1 Assimilation

    • 1.3.1.1 Document routing / sorting

    • 1.3.1.2 Information extraction / summarization

    • 1.3.1.3 Search

  • 1.3.2 Dissemination

    • 1.3.2.1 Internal / in-house publication

      • 1.3.2.1.1 Routine

      • 1.3.2.1.2 Experimental / research

    • 1.3.2.2 External publication

      • 1.3.2.2.1 Single-client

      • 1.3.2.2.2 Multi-client

  • 1.3.3 Communication

    • 1.3.3.1 Synchronous

    • 1.3.3.2 Asynchronous

• 1.4 User characteristics

  • 1.4.1 Machine translation user

    • 1.4.1.1 Education

    • 1.4.1.2 Proficiency in source language

    • 1.4.1.3 Proficiency in target language

    • 1.4.1.4 Computer literacy

  • 1.4.2 Translation consumer

    • 1.4.2.1 Proficiency in source language

    • 1.4.2.2 Proficiency in target language

  • 1.4.3 Organisational user

    • 1.4.3.1 Quantity of translation

    • 1.4.3.2 Number of personnel

    • 1.4.3.3 Time allowed for translation

• 1.5 Input characteristics (author and text)

  • 1.5.1 Document type

    • 1.5.1.1 Genre

    • 1.5.1.2 Domain/field of application

  • 1.5.2 Author characteristics

    • 1.5.2.1 Proficiency in source language

    • 1.5.2.2 Professional training

  • 1.5.3 Characteristics related to sources of error

    • 1.5.3.1 Intentional error sources

    • 1.5.3.2 Medium related error sources

    • 1.5.3.3 Performance related errors

2 System characteristics to be evaluated

• 2.1 MT system-specific characteristics

  • 2.1.1 Translation process models

    • 2.1.1.1 Methodology

      • 2.1.1.1.1 Rule-based models

      • 2.1.1.1.2 Statistically-based models

      • 2.1.1.1.3 Example-based models

      • 2.1.1.1.4 Translation memory models

    • 2.1.1.2 MT Models

      • 2.1.1.2.1 Direct MT

      • 2.1.1.2.2 Transfer-based MT

      • 2.1.1.2.3 Interlingua-based MT

  • 2.1.2 Linguistic resources and utilities

    • 2.1.2.1 Languages

    • 2.1.2.2 Dictionaries

    • 2.1.2.3 Wordlists, glossaries

    • 2.1.2.4 Comparable and parallel corpora

    • 2.1.2.5 Grammars

  • 2.1.3 Characteristics of process flow

    • 2.1.3.1 Translation preparation activities

    • 2.1.3.2 Post-translation activities

    • 2.1.3.3 Interactive translation activities

    • 2.1.3.4 Dictionary updating

    • 2.1.3.5 Process management

• 2.2 System external characteristics

  • 2.2.1 Functionality

    • 2.2.1.1 Suitability

      • 2.2.1.1.1 Target-language only

        • 2.2.1.1.1.1 Readability (or: fluency, intelligibility, clarity)

        • 2.2.1.1.1.2 Comprehensibility

        • 2.2.1.1.1.3 Coherence

        • 2.2.1.1.1.4 Cohesion

      • 2.2.1.1.2 Cross-language / contrastive

        • 2.2.1.1.2.1 Coverage of corpus-specific phenomena

        • 2.2.1.1.2.2 Style

    • 2.2.1.2 Accuracy

      • 2.2.1.2.1 Fidelity

      • 2.2.1.2.2 Consistency

      • 2.2.1.2.3 Terminology

    • 2.2.1.3 Wellformedness

      • 2.2.1.3.1 Punctuation

      • 2.2.1.3.2 Lexis / lexical choice

      • 2.2.1.3.3 Grammar / syntax

      • 2.2.1.3.4 Morphology

    • 2.2.1.4 Interoperability

    • 2.2.1.5 Compliance

    • 2.2.1.6 Security

  • 2.2.2 Reliability

    • 2.2.2.1 Maturity

    • 2.2.2.2 Fault tolerance

    • 2.2.2.3 Crashing frequency

    • 2.2.2.4 Recoverability

    • 2.2.2.5 Reliability compliance

  • 2.2.3 Usability

    • 2.2.3.1 Understandability

    • 2.2.3.2 Learnability

    • 2.2.3.3 Operability

    • 2.2.3.4 Documentation

    • 2.2.3.5 Attractiveness

    • 2.2.3.6 Usability compliance

  • 2.2.4 Efficiency

    • 2.2.4.1 Time behaviour

      • 2.2.4.1.1 Pre-processing time

        • 2.2.4.1.1.1 Pre-editing time

        • 2.2.4.1.1.2 Code-set conversion

        • 2.2.4.1.1.3 Preparation time

      • 2.2.4.1.2 Input-to-output translation speed

      • 2.2.4.1.3 Post-processing time

        • 2.2.4.1.3.1 Post-editing time

        • 2.2.4.1.3.2 Code-set conversion

        • 2.2.4.1.3.3 Update time

    • 2.2.4.2 Resource utilisation

      • 2.2.4.2.1 Memory

      • 2.2.4.2.2 Lexicon

      • 2.2.4.2.3 Clean-up

      • 2.2.4.2.4 Program size

  • 2.2.5 Maintainability

    • 2.2.5.1 Analyzability

    • 2.2.5.2 Changeability

      • 2.2.5.2.1 Ease of upgrading multilingual aspects of system

      • 2.2.5.2.2 Improveability

      • 2.2.5.2.3 Ease of dictionary updating

      • 2.2.5.2.4 Ease of modifying grammar rules

    • 2.2.5.3 Stability

    • 2.2.5.4 Testability

    • 2.2.5.5 Maintainability compliance

  • 2.2.6 Portability

    • 2.2.6.1 Adaptability

    • 2.2.6.2 Installability

    • 2.2.6.3 Portability compliance

    • 2.2.6.4 Replaceability

    • 2.2.6.5 Co-existence

  • 2.2.7 Cost

    • 2.2.7.1 Introduction cost

    • 2.2.7.2 Maintenance cost

    • 2.2.7.3 Other costs