In neural machine translation (NMT), BPE is usually used to limit vocabularies since a large number of vocabularies have a negative impact on training. Use of BPE could limit the number of vocabularies to some extent, but word alignment by attention mechanism cannot be done correctly since vocabularies are not properly segmented into word stems.
In neural machine translation, a huge number of vocabularies from the corpus have a negative impact on its training, and the longer the length of sentences which reflects the number of tokens is, the longer the training time gets. Therefore, it is important to perform good tokenization for minimizing the number of vocabularies and the length of sentences.
Kim Tong Gwon, a researcher at the Institute of Information Technology, proposed a stem and ending based tokenization (SEBT) method for improving the accuracy of word alignment by applying to tokenization a stem and ending dictionary which was made by segmenting BPE tokens into stems and endings.
This method has four advantages:
First, it is possible to reduce the number of vocabularies by correctly segmenting the stems and endings of Russian and Korean.
Second, it can improve learning efficiency by relieving the sparse word problem.
Third, it can improve the alignment accuracy between Russian and Korean tokens in the attention mechanism.
Fourth, it can reduce the training time since the sentence length is not increased after tokenization.
The experiments showed that this approach improves the performance of Russian-Korean NMT by improving the alignment between Russian and Korean tokens during the training of Russian-Korean neural machine translation and by reducing the number of Russian and Korean vocabularies.
You can find more information in his paper “SEBT: Word Stem and Ending based Tokenization in Russian-Korean Neural Machine Translation” in “2025 10th International Conference on Intelligent Information Technology (ICIIT 2025)”.