Final Thesis: Continuous Performance Benchmarking and Optimization of an ETL Data Pipeline

Abstract: Software engineering teams rely on development data for productivity measurement, transfer pricing, and impact assessment. Tools for this analysis typically extract the raw data from platforms like GitHub, GitLab, or Jira. The MECOIS project’s ETL pipeline handles the analysis of the data and follows the medallion architecture with Bronze, Silver, and Gold layers. Over time, the pipeline has grown without performance instrumentation, leaving regressions and inefficiencies unattributable to specific changes. To close the gap, this thesis introduces a benchmarking framework and demonstrates it by optimizing an identified bottleneck. The framework uses three measurement tiers of varying granularity to capture pipeline performance at different scopes. Eight benchmarks built on these tiers record each measurement against a stored baseline, attributing performance changes to specific code revisions. Alongside the framework, the Bronze-to-Silver transition’s identity resolution adopts GraphQL-based alias batching and Spark partition-level execution. This speeds up the transition by 2.93× and reduces external SortingHat requests by 99%. Across the full pipeline, this yields a 1.07× end-to-end speedup, with the framework’s measurements identifying extraction, rather than identity resolution, as the dominant runtime cost. Beyond the demonstration, the framework runs in the project’s CI to detect regressions before they reach production.

Keywords: None

PDF: Master Thesis

Reference: Lennart Heimbs. Continuous Performance Benchmarking and Optimization of an ETL Data Pipeline. Master Thesis. Friedrich-Alexander-Universität Erlangen-Nürnberg: 2026.