ISI ReWrite Decoder User's Manual Version 1.0.0a for Public Release 1.0.0a

Related Papers

• Germann, U., Jahr, M., Knight, K., Marcu, D., and Yamada, K. 2001
  Fast Decoding and Optimal Decoding for Machine Translation. Proceedings of ACL-01. Toulouse, France. Best Paper award.
• Germann, U. 2003.
  Greedy Decoding for Statistical Machine Translation in Almost Linear Time Proceedings of HLT-NAACL-2003.. Edmonton, AB, Canada.

What's New?

• IBM Model 3 is no longer supported, only Model 4.
• Therefore, training must be performed with GIZA++. (The original GIZA release won't work! It does not support IBM Model 4, and this version of the decoder supports only Model 4.)
• The decoder now expects input in XML format. Plain text input won't work any more.
• The decoder is much faster.
• You can now force the decoder to translate items in a pre-defined way by providing the desired translation with the input.
• Swapping respects phrase boundaries (if they are provided with the input).
• You can "turn off" swapping for individual phrases.

What you need

• GIZA++ to train the translation model.
• The CMU-Cambridge Statistical Language Modeling Toolkit v.2 for training the language model.
• A list of zero-fertility words, with one word per line. You can create a list manually, or use this script to create one automatically from the translation model files.
• Obviously, training data. See the documentations for GIZA++ and the CMU-Cambridge LM toolkit for details on the required input format for TM and LM training.
• Some shared libraries that are included in this release. Make sure you put their path in your LD_LIBRARY_PATH environment variable!

Notes on Language Model Training

• When compiling the LM toolkit on "little-endian" machines (as a rule of thumb, PCs are little-endian, Sun workstations are not), you MUST set the BYTESWAP_FLAG correctly; see the README file that comes with the toolkit. When compiled with the wrong setting, the toolkit will appear to be working fine but the decoder will crash.
• The decoder expects by default the markers <s> for 'begin sentence' and </s> for 'end sentence'. If you use different tags in the LM training data, you must specify these explicitly. See the documentation that comes with the LM toolkit for details. Section ... describes how to set custom sentence boundary markers.

The Translation Model Config File

The Translation Model Config file contains information about the Translation Model. It contains one parameter-value pair per line. Parameters and values are separated by ‘=’. Comments are separated by ‘#’.

The following example also explains the meaning of the various (mandatory!) parameters.

# This is a sample config file for the translation model for a French (fra) to English (eng) system

TM_RawDataDir = /the/path/to/the/translation/model/files
# TM_RawDataDir points to the location where Giza put its output. 
# If missing, the decoder assumes that the files are in the same
# directory as the config file.

TTable           = fra-eng.t3.final  # translation table
InverseTTable    = fra-eng.ti.final  # inverse translation table
NTable           = fra-eng.n3.final  # fertility table
D3Table          = fra-eng.d3.final  # distortion table for model 3
D4Table          = fra-eng.D4.final  # distortion table for model 4
PZero            = fra-eng.p0_3.final # contains p0 for the model 
Source.vcb       = fra-eng.src.vcb    # maps from words to word IDs
Target.vcb       = fra-eng.trg.vcb    # maps from words to word IDs
Source.classes   = /location/of/the/tokenized/corpus/eng.vcb.classes 
# a path starting with '/' overrides the default path specified with TM_RawDataDir
Target.classes   = /location/of/the/tokenized/corpus/fra.vcb.classes
FZeroWords       = ZeroFert           # file with (English) zero fertility words
                                      # file format: one word per line (anything after the
                                      # first word (space separated) on each line is ignored)

Notes:

• In order to speed up loading of the various probability tables, the decoder writes binary versions of the tables to the hard disk. By default, these files are written to the same directory as the ‘raw’ data. A different directory can be specified either with the --tm-bindatadir switch or with the parameter TM_BinDataDir in the Decoder Config File (see below).
• GIZA++ does not automatically create a list of zero fertility words. You can either construct one manually, or extract it form the training output with this script.

Input Format

The decoder expects the input in XML format. Minimally, an input file looks like this:

Note that the input must be proper XML! Something like <s id=1> is not a proper XML tag --- attribute values must be quoted.

For your convenience, the distribution contains a small xml-wrapper for one-sentence-per-line input.

Additional Markup

Between the <s> tags, the sentence can be marked up with other XML tags; if the markup amounts to a tree structure (e.g. <a><b>one two</b><c>three four</c></a>), only siblings within the tree will be swapped, that is "one" and "two", "three" and "four", and "one two" and "three four" in the example, but not, for example, "two" and "three".

With the attribute english you can tell the decoder how to translate certain items. <ENAMEX english="Kohl">Kohl<ENAMEX>, for example, will force top translate Kohl (the former German Chancellor) as Kohl (instead of cabbage) in English; the actual tag name is irrelevant. Multiple options can be provided by seperating them with a vertical bar, e.g. 'english="April 7|1 Apr|04/07"'. The decoder then picks the option that best fits the given language model context.

The attribute allow_swapping="no" prevents the swapping of the immediate children of the respective node in the XML structure. The children of those children can be swapped with one another. This feature can be used to preserve parentheses: <paren allow_swapping="no"><lp english="(">(</lp><middle> this is a parenthesis</middle><rp english=")">)</rp></paren> will keep "(" and ")" as brackets around the content of the parenthesis, while the words of the parenthesis can be swapped around within the scope of the parenthesis.

Decoder Config File (optional)

Suppose, for example, that you want to run the decoder always with the same translation model and print out the three best translations for each input sentence together with various probability measures (Language Model Probability, Translation Model Probability, and various combinations of these two probabilities). In this case, the file decoder.cfg might look like this:

# This is a sample decoder config file (~/decoder.cfg)

LanguageModelFile = ~/language-models/english.binlm
# this saves us the --lmfile switch

TranslationModelConfigFile = ~/decoders/fra-eng.1.tm-config
# this saves us the --tmfile switch

PrintAlignment     = true
PrintProbabilities = true

Running a Translation Server

Decoder Options