03:07 pm, 22 Aug 05
NIST results
NIST 2005 Machine Translation Evaluation Official Results. As I mentioned before, Franz & co. (congrats again, 
hawk!) totally rocked it.
But it's worth noting that (at least according to Franz's papers from before Google; I don't know much about what they're actually doing here) part of his approach is use the BLEU score as the objective function in their learning. This does make sense: the BLEU score was designed to correlate with human translation quality and so it's a reasonable function to optimize. And the sentences they were given to translate must have been entirely separate from all available training data. But still, it feels a little weird to me that you'd optimize on the metric used to judge; it means you can make "simple" translation mistakes (at least to a human observer) but still get a good score as long as the scoring function doesn't account for those sort of mistakes.
hawk!) totally rocked it.But it's worth noting that (at least according to Franz's papers from before Google; I don't know much about what they're actually doing here) part of his approach is use the BLEU score as the objective function in their learning. This does make sense: the BLEU score was designed to correlate with human translation quality and so it's a reasonable function to optimize. And the sentences they were given to translate must have been entirely separate from all available training data. But still, it feels a little weird to me that you'd optimize on the metric used to judge; it means you can make "simple" translation mistakes (at least to a human observer) but still get a good score as long as the scoring function doesn't account for those sort of mistakes.
I do wonder if they also plan on automatically incorporating feedback from their user community -- that is, having a form on the results page where you can click a number between 1 and 10, and feed that back in to reinforce the translations that people like, and deinforce the things that it doesn't. (With, possibly, another form where you correct the translation, and then say how sure you are that the new translation is correct.)
I suspect that if the ability is present, people will tend to use it well. (Or, at least, as well as they can, which is why you need to normalize against the mean and SD of a user's scores, and give them the oppurtunity to say how sure they are of their suggested translations.)
Well, no, not really: that was what I was trying to post about. The point of making a benchmark is to quantitatively measure performance. The benchmark is designed (and quite a bit of work put into it) to be a way of deciding whether you're doing well or not. So it's the natural thing to use when training a system.
Thanks for the link to the thread.
I did a SMT project as ed.ac.uk last year, and I've still got the code kicking around. I want to see how well I can perform against google :)
[I managed 0.2315 on English->German]
Optimizing for BLEU is a good idea, as it allows for faster feedback. Humans are too slow :/.
Aside: The NIST test package does other MT metrics apart from BLEU.