Ever since machine translation became neural machine translation in 2016, big tech has been trying to persuade us that it has reached human-level quality. Today, it has gone one step further. The nerds over in Silicon Valley are now trying to convince us that machine translation outperforms human-level translation. Are they right? Not on your nelly!
What exactly are the claims?
Anyone with knowledge of the industry knows that human translation has always been superior to machine translation in just about every way apart from speed and price. Big tech knows it, too. That’s why its claims are made with some interesting caveats.
The study only looks at whether a deep learning translation system called CUBITT can produce translations that are comparable to those produced by humans in the domain of ‘news’ translation, which accounts for a tiny proportion of the whole translation sector. The study also claims that machine translation only outperforms human translation in the metric of ‘adequacy’, but what exactly does that mean?
The sentence-level translation Turing test
The tests compared the work of CUBITT against human translators on almost 8,000 sentences across 53 documents. 15 human evaluators were then asked to rate the translations for ‘adequacy’.
The definition of adequacy for the purpose of the test was ‘adequately expressing the source text’s intended meaning in the target language’.
The human evaluators found that 52 percent of CUBITT’s sentences scored higher than the human translations when measured against this metric, while just 26 percent of CUBITT’s translations scored lower than sentences translated by humans. Impressive, right?
Here’s the catch
When compared to the work of human translators, the evaluators observed that CUBITT made fewer errors in addition of meaning, shift of meaning and omission of meaning. However, it made significantly more errors when it came to cross-sentence context and readability, with the work of the human translators rated as more fluent.
Interestingly though, another test compared 100 pairs of sentences that were translated by CUBITT and Google Translate. In this case, the evaluators found that CUBITT’s translations were less likely to be identified as the work of a machine when compared to those produced by Google Translate.
Better than Google Translate but still worse than humans
Big tech’s claims that CUBITT can outperform human translators seems to be more of a newspaper headline than a representation of reality, given that the metric of ‘adequacy’ is one that has been devised specifically for this test.
However, CUBITT can outperform fellow machine translation tool Google Translate, and that is something to celebrate. Machine translation undoubtedly has an important part to play in the industry, so any improvements are welcomed, but the truth is, there’s still no substitute for the work of a professional human translator.
Human translation from a leading UK team
At Linguistica International, our professional team of human translators can help you reach your customers in more than 250 languages. Get in touch to discuss your translation project today on 02392 987 765 or email info@linguistica-international.com.
