Abstract

The unavailability of parallel training corpora in resource-poor languages is a major bottleneck in cost-effective and rapid deployment of statistical machine translation (SMT) technology. This has spurred significant interest in active learning for SMT to select the most informative samples from a large candidate pool. This is especially challenging when irrelevant outliers dominate the pool. We propose two supervised sample selection methods, viz. greedy selection and integer linear programming (ILP), based on a novel measure of benefit derived from error analysis. These methods support the selection of diverse and high-impact, yet relevant batches of source sentences. Comparative experiments on multiple test sets across two resource-poor language pairs (English-Pashto and English-Dari) reveal that the proposed approaches achieve BLEU scores comparable to the full system using a very small fraction of all available training data (ca. 6% for EP and 13% for ED). We further demonstrate that the ILP method supports global constraints of significant practical value.

Date

2011

Authors

Sankaranarayanan Ananthakrishnan, Shiv Vitaladevuni, Rohit Prasad, Prem Natarajan

Conference

Proceedings of 5th International Joint Conference on Natural Language Processing

Pages

819-827