Development of data pipeline for processing of streaming data on a cloud platform
Loading...
Date
2026
Authors
Materynskyi, Volodymyr
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Air quality monitoring is an important part of public health infrastructure worldwide, and in Ukraine it has become particularly important due to the combined impact of wartime fires and industrial emissions. Existing notification systems for Ukrainian cities operate on hourly readings with limited geographic coverage. This thesis presents the design, implementation, and evaluation of a streaming data pipeline that continuously ingests satellite readings from major Ukrainian cities, detects pollution events and visualises conditions on a live dashboard. The pipeline is organised as five loosely coupled stages spanning data ingestion, stream processing, aggregate storage, real-time visualisation, and email alerting. Load testing demonstrated sustained throughput of 414,000 records per minute with sub-minute latency, and an effective ceiling of approximately 1.79 million records per minute. The work contributes a reproducible reference implementation of streaming infrastructure for environmental monitoring, addressing the coverage and latency limitations of existing Ukrainian systems.
Description
Keywords
streaming data pipeline, real-time visualization, stream processing, environmental monitoring, scalable architecture, bachelor`s thesis