Алгоритм токенізації та стемінгу для текстів українською мовою
Loading...
Date
2017
Authors
Глибовець, Андрій
Точицький, Володимир
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
У статті досліджено проблематику застосування алгоритмів токенізації (лематизації) та
стемінгу для обробки документів українською мовою як початкових етапів для аналізу текстових
документів. Аналіз текстів – доволі поширена задача, що виникає в різноманітних завданнях, і на
першому етапі початкової обробки завжди використовується стемінг або лематизація. Для
більшості мов цю проблему давно розв’язано й існує багато різноманітних реалізацій. Але для
української мови немає реалізації цих алгоритмів. Для написання лематизатора потрібно мати
багато додаткових інструментів, оскільки він спирається на лексикографічний аналіз, тоді як
реалізація стемера не потребує додаткових бібліотек. У цій статті ми зосередилися на створенні
стемера для української мови. Було проаналізовано наявні алгоритми і запропоновано адаптацію
та реалізацію для української мови. Результатом є готовий модуль мовою PHP, що перебуває
у вільному доступі та може використовуватися в проектах, пов’язаних з аналізом текстів
українською мовою.
The article focuses on the problem of adaptation of tokenization and stemming algorithms for documents written in the Ukrainian language and finding an optimal approach to the initial stage of automated text analysis. Intelligent Text Analysis (IAT) is the direction of Data Mining and Artificial Intelligence, whose purpose is to retrieve information from collections of text documents based on the application of effective methods of machine learning and the processing of natural language. Every year, progress in this area is growing at a rapid pace. Modern software increasingly needs ready-made solutions for word processing and enhancement of their systems. The greatest difficulty is encountered when working with natural language or with texts without a clear content structure. The key tasks of IAT: categorization of texts, information search, processing of changes in text collections, and development of means of representation of information for the user. In order to work comfortably with the text, one needs to translate it into the presentation of the programming language and process it for further work. For the Ukrainian language, there are some unfinished works (for example, “Free algorithm for stigma for the Ukrainian language” http://www.senyk.poltava.ua/projects/projects.html) or complex systems, but without the necessary functional (for example, “LanguageTool” https://www.languagetool.org/development/). Therefore, there is a need to implement their own libraries for processing and analyzing texts. After analyzing the classic approaches to the text mining text analysis, we adapted Porter’s algorithm and implemented a freely available PHP language module for tokenization and emulation of the Ukrainian language. Practical testing has shown that for tokenization, a simple breakdown into spaces with the subsequent normalization of tokens works well. For stemming, we used an approach that cuts the end and suffix, bringing words to its root as closely as possible. A better result can be achieved through lemmatization, but such a solution requires a separate study of the language and large resources. The resulting initial processing of the text in this article allows one to work with tokens to obtain semantic links in texts written in Ukrainian. The stemmer designed as a module that can be quickly integrated into the project and available for use at the link https://packagist.org/packages/tochytskyi/ukrstemmer
The article focuses on the problem of adaptation of tokenization and stemming algorithms for documents written in the Ukrainian language and finding an optimal approach to the initial stage of automated text analysis. Intelligent Text Analysis (IAT) is the direction of Data Mining and Artificial Intelligence, whose purpose is to retrieve information from collections of text documents based on the application of effective methods of machine learning and the processing of natural language. Every year, progress in this area is growing at a rapid pace. Modern software increasingly needs ready-made solutions for word processing and enhancement of their systems. The greatest difficulty is encountered when working with natural language or with texts without a clear content structure. The key tasks of IAT: categorization of texts, information search, processing of changes in text collections, and development of means of representation of information for the user. In order to work comfortably with the text, one needs to translate it into the presentation of the programming language and process it for further work. For the Ukrainian language, there are some unfinished works (for example, “Free algorithm for stigma for the Ukrainian language” http://www.senyk.poltava.ua/projects/projects.html) or complex systems, but without the necessary functional (for example, “LanguageTool” https://www.languagetool.org/development/). Therefore, there is a need to implement their own libraries for processing and analyzing texts. After analyzing the classic approaches to the text mining text analysis, we adapted Porter’s algorithm and implemented a freely available PHP language module for tokenization and emulation of the Ukrainian language. Practical testing has shown that for tokenization, a simple breakdown into spaces with the subsequent normalization of tokens works well. For stemming, we used an approach that cuts the end and suffix, bringing words to its root as closely as possible. A better result can be achieved through lemmatization, but such a solution requires a separate study of the language and large resources. The resulting initial processing of the text in this article allows one to work with tokens to obtain semantic links in texts written in Ukrainian. The stemmer designed as a module that can be quickly integrated into the project and available for use at the link https://packagist.org/packages/tochytskyi/ukrstemmer
Description
Keywords
text mining, токенізація, лематизація, стемінг, семантичний аналіз текстів, стаття, tokenization, lematization, stemming, semantic text analysis, стаття
Citation
Глибовець А. М. Алгоритм токенізації та стемінгу для текстів українською мовою / Глибовець А. М., Точицький В. В. // Наукові записки НаУКМА. Комп'ютерні науки. - 2017. - Т. 198. - С. 4-8.