Automatic Subtitling Translation Quality Assessment

Let’s suppose that it is a snowy day in the middle of winter season, and you lay on with a blanket beside the fireplace but you postponed watching the snow because you have something more exciting to watch. Perhaps you deserved this moment after a long busy day or perhaps you did nothing to deserve it, instead, you just wanted to live this moment. Let’s suppose a new season of a foreign series that you have been waiting for eagerly has come out that night or a movie that you have been waiting for years has released. You take a seat on your most comfortable coach with your favourite food and
beverages, pressed the start button, you sat back, and you are sure that you will watch a production with high pleasure… However, you are constantly faced with a translation error or subtitle shift while you should be so peaceful… You are drifted away from the moment of joy you dreamed of and with disappointment slowly closed the TV show/film due to translation errors that constantly distracted you… Unfortunately, such things happen in life.

What if I told you that there is a contract of illusion designed so that your viewing pleasure will never be spoiled?

The idea of contract of illusion was first created by Romero Fresco in 2011, whose work was aimed at improving intralingual subtitle translations. As to Fresco, dubbing or subtitling should be so perfect that it should make the audience forget the original one and not feel that they are facing a translated content. Nowadays, in parallel with the increasing consumption of audio-visual content, the need to examine the quality of translations of intralingual or interlingual subtitles of these contents is also increasing. Therefore, it is possible to consider the quality assessment of subtitle translations as a new field of study. More studies are carried out on this subject every year and it is aimed to improve the quality of subtitle translations.

Jan Pedersen is another researcher who set out with the ‘contract of illusion’, which has become the motto of the researchers in both intralingual and interlingual subtitle quality evaluation studies, throughout the years. The FAR model, developed by Pedersen at Stockholm University in 2017, and the title of his work is ‘The FAR model: assessing quality in interlingual subtitling’. The Model has been used as a subtitle quality assessment tool since the year it was designed and has been used to check subtitle translations between languages in various countries. However, I can say that these studies have just started to gain importance in Türkiye and are still insufficient. I would like to introduce the FAR model, which I take as a basis for my automatic subtitle translation quality assessment project I am carrying out with my consultant Asst. Prof. Muhammed BAYDERE, which is supported by TÜBİTAK [The Scientific and Technological Research Institution of Turkey] as a 2209-A project.

Now, I would like to paraphrase the FAR model by summarising Pedersen’s article which I mentioned above.

The FAR model, which was created to evaluate the quality of interlingual subtitle translations, is based on three basic principles: functional equivalence, acceptability, and readability. The initials of these three principals form the name of the model. In this model based on error analysis, a penalty score system is used to show the evaluation at what points need change and improvement in the subtitle translation. According to the penalty score system, the errors detected in the subtitles are categorized in three different ways: minor, standard, and serious. Minor errors are evaluated with 0.25 points, standard errors with 0.5 points, and serious errors with 1 point to form a table. The total score obtained gives the assessment a very concrete result on the quality of the subtitle. Minor errors are defined as errors that are hard to spot and so minor that only very attentive viewers can break the contract of illusion.

Standard errors have been defined as errors that tend to break the contract and ruin the subtitle for viewers. Serious errors, on the other hand, are defined as errors that make it difficult to understand not only the sentence they are in but also the previous or next sentences, and may even completely eliminate intelligibility.

Different error titles were identified in each of the three main principles of the model. Functional equivalence errors are divided into two as semantic and stylistic errors. Semantic errors cause loss of meaning and expression disorder in the subtitle and are the most critical of the error groups. At the same time, the absence of any sound (crying, screaming, clicking, squeaking, etc.) in the subtitle is also included in this error group. For this reason, they score higher than other error groups in the scoring system. Minor semantic errors are scored 0.5 points, standard semantic errors are scored 1 point, and serious semantic errors are scored 2 points. Stylistic errors do not pose as serious a problem as semantic errors and cause a loss of nuance rather than incomprehensibility. Errors detected in the context of acceptability are divided into three as grammatical errors, spelling errors, and idiomatic errors. Grammar errors are detected in subtitles that violate the grammar rules of the target language. Misspelling a single letter in the subtitle and typos that make the word almost unreadable are also considered spelling errors. Idiomatic errors, on the other hand, include not only the use of incorrect idioms but also word combinations that disrupt the natural flow of the language. These mistakes expose the viewer to language that feels out of line and break the contract. Errors detected in line with the principle of readability are divided into three subheadings: segmentation and spotting, punctuation, and finally reading speed and line length. Segmentation and spotting errors can be characterized as ‘subtitle drift’ which was caused by bad synchronization by speech. Within punctuation errors, besides punctuation marks, there is also the use of italics to emphasize the sounds that do not belong to the characters, such as telephone, radio, voice recording. Reading speed and line length errors can be evaluated as subtitles that pass fast enough to make the viewer want to stop the content they are watching or the subtitles that the viewer has already read but still remain on the screen for seconds.

The FAR model can be used to evaluate every interlingual subtitle translation. In my project, I used the FAR model to evaluate the quality of automatic subtitle translations in English-Turkish language pair used on Intergovernmental Panel on Climate Change (IPCC) YouTube channel. IPCC is a panel about climate change, and it is connected to the United Nations. In the YouTube channel, there are informative videos about this critic issue. One of the main motivations of this project is to bring the FAR model, which has not been the subject of a study in Türkiye before, to the Turkish literature and to enhance a
new perspective for future studies in the field of subtitle translation quality assessment.

Thanks to the contract of illusion, which was secretly signed between the subtitle translator and the audience, I wish you good days when you can sit on a comfortable couch and watch the eagerly awaited films without any trouble.

References

IPCC. Intergovernmental Panel on Climate Change. Accessed 20 April 2023. https://www.ipcc.ch/

Pedersen, J. (2017). “The FAR model: assessing quality in interlingual subtitling”. Journal of Specialised Translation, (28), 210-229.

Romero Fresco, Pablo. (2011). Subtitling through speech recognition: respeaking. Manchester: St Jerome.