AI
arXiv paper finds most translation quality-estimation systems prefer literal over idiomatic renderings of idioms
A preprint submitted on 26 October 2026 introduces a parallel dataset pairing idiom-bearing English sentences with literal and idiomatic translations in nine target languages, and reports that most benchmarked quality estimation metrics and LLM judges rate literal translations above idiomatic ones. Only MetricX-24-Hybrid (XL, XXL) and Gemma-4-31B consistently preferred idiomatic renderings, the authors report.
A preprint submitted to arXiv on 26 October 2026 introduces what its authors describe as the first parallel dataset pairing idiom-bearing English sentences with both literal and idiomatic translations across nine target languages, augmented into multi-sentence segments and a graded literality continuum. Using that dataset, the paper benchmarks eleven conventional quality estimation (QE) metrics from the COMET and MetricX families and ten LLMs from the Qwen-3, Gemma-4 and OLMo-3 families used as GEMBA-style judges.
The paper reports that most systems, especially older metrics and smaller models, display a marked literality bias, rating literal translations of idioms above idiomatic ones. Higher parameter counts and newer training sets appear to reduce this bias to an extent, according to the abstract. Nevertheless, of all the benchmarked systems, only MetricX-24-Hybrid (XL, XXL) and Gemma-4-31B consistently rate idiomatic translations of idioms above literal ones, and only Gemma-4-31B reliably ranks intermediate degrees of idiomaticity and literality.
The authors caution against deploying standard QE metrics in domains rich in figurative language, where higher scores might be rewarding undesirable literalism. The abstract frames the question as whether QE systems have kept pace with large language models' improving handling of idioms and can reliably score idiomatic renderings above faulty literal ones.
Analysis: For people who build AI agents, the practical risk is in retry loops and output gates that rely on these scorers. If a scorer prefers the flat reading, an optimization loop will push toward it. The paper tested idioms only, so whether the same bias appears on other figurative language is untested here.
Claims and evidence
What the author claims, and what backs it up. Key claims are the ones the story depends on.
-
The paper introduces the first parallel dataset pairing idiom-bearing English sentences with both literal and idiomatic translations across nine target languages.
Key claim
Supported
-
quote
https://arxiv.org/abs/2610.35261“We introduce the first parallel dataset pairing idiom-bearing English sentences with both literal and idiomatic translations across nine target languages, augmented into multi-sentence segments and a graded literality continuum.”
-
quote
-
The study benchmarks eleven conventional QE metrics from the COMET and MetricX families and ten LLMs from the Qwen-3, Gemma-4 and OLMo-3 families used as GEMBA-style judges.
Key claim
Supported
-
quote
https://arxiv.org/abs/2610.35261“Using this dataset, we benchmark eleven conventional QE metrics from the COMET and MetricX families and ten LLMs from the Qwen-3, Gemma-4 and OLMo-3 families used as GEMBA-style judges.”
-
quote
-
Most systems, especially older metrics and smaller models, display a marked literality bias, rating literal translations of idioms above idiomatic ones.
Key claim
Supported
-
quote
https://arxiv.org/abs/2610.35261“We find that most systems, especially older metrics and smaller models, display a marked literality bias, rating literal translations of idioms above idiomatic ones.”
-
quote
-
Only MetricX-24-Hybrid (XL, XXL) and Gemma-4-31B consistently rate idiomatic translations of idioms above literal ones.
Key claim
Supported
-
quote
https://arxiv.org/abs/2610.35261“nevertheless, of all the benchmarked systems, only MetricX-24-Hybrid (XL, XXL) and Gemma-4-31B consistently rate idiomatic translations of idioms above literal ones”
-
quote
-
Only Gemma-4-31B reliably ranks intermediate degrees of idiomaticity and literality.
Key claim
Supported
-
quote
https://arxiv.org/abs/2610.35261“and only Gemma-4-31B reliably ranks intermediate degrees of idiomaticity and literality.”
-
quote
-
The authors caution against deploying standard QE metrics in domains rich in figurative language.
Key claim
Supported
-
quote
https://arxiv.org/abs/2610.35261“Our findings caution against deploying standard QE metrics in domains rich in figurative language, where higher scores might be rewarding undesirable literalism.”
-
quote
-
The preprint was submitted on 26 October 2026.
Supported
-
quote
https://arxiv.org/abs/2610.35261“[Submitted on 26 Oct 2026]”
-
quote