Retrieval Augmented Generation for Ukrainian Government Services: A Comparative Evaluation of the Approaches
Loading...
Date
2025
Authors
Маринич, Антон
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Retrieval Augmented Generation or RAG is a method that is used to improve the quality of retrieval for LLMs, to avoid hallucinations and be aware of all the changes in the data. This approach integrates LLMs with external data source by building a vector index. This thesis presents a comprehensive study on how different RAG approaches perform in Ukrainian Governmental Services domain. I establish a non-RAG baseline using GPT-4.1-mini model and iteratively perform tests on different configurations of RAG approaches. I have also created a dataset with 500 open questions about Ukrainian Governmental Services using GPT-o4-minihigh model. My best results comparing to the baseline are 13.25% improvement in LLM Judge Score using CRAG with Hypothetical Document Embedding and Reranking and 10% improvement on Factual Correctness using CRAG with Reranking.
Description
Keywords
Retrieval Augmented Generation (RAG), Large language models (LLMs), hallucinations, Ukrainian Governmental Services domain, bachelor`s thesis