Evan Martin (evan) wrote in evan_tech,
Evan Martin

NIST results

NIST 2005 Machine Translation Evaluation Official Results. As I mentioned before, Franz & co. (congrats again, hawk!) totally rocked it.

But it's worth noting that (at least according to Franz's papers from before Google; I don't know much about what they're actually doing here) part of his approach is use the BLEU score as the objective function in their learning. This does make sense: the BLEU score was designed to correlate with human translation quality and so it's a reasonable function to optimize. And the sentences they were given to translate must have been entirely separate from all available training data. But still, it feels a little weird to me that you'd optimize on the metric used to judge; it means you can make "simple" translation mistakes (at least to a human observer) but still get a good score as long as the scoring function doesn't account for those sort of mistakes.
Tags: google, linguistics, papers

  • memcache job offers

    I get occasional recruiter spam that specifically calls out "my work on memcached". This is pretty funny because all I did was make some trivial…

  • application stack

    "Put yourself in 1995. I'm going to tell the you of 1995 that in 2010, there will be a software platform with the following properties:" Luis Villa…

  • socks5 proxying flash via ssh

    Suppose you're in Germany and want to watch some Flash-based videos that are IP-limited to the US for whatever reason. At first you'd think you could…

  • Post a new comment


    default userpic
    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.