Data Warehouse, Medallion Architecture, PostgreSQL, DBT, Airbyte, Python, Dagster, Power BI, SharePoint Automation, Monitoring, Grafana, Prometheus, Docker, ETL, Data Orchestration

Building a Modern Data Warehouse from the Ground Up: A Telecommunications Success Story

Morice Nouvertne

When I first joined this innovative German telecommunications provider, the landscape was familiar yet fragmented: an organization rich in data but lacking a coherent structure to harness it. Specializing in fiber infrastructure and offering end-to-end technical solutions, the company needed not just reports—but actionable insights delivered reliably and efficiently. My mission: to turn scattered data into unified intelligence.

The Challenge: Fragmented Data, Siloed Insights

Initially, data lived across disconnected systems: an internal CMS, Jira, various public APIs, and specialized telecom endpoints. Every department had its preferred tools, leading to redundancy, inefficiencies, and limited visibility into crucial operational metrics. The need for a robust, scalable solution was clear—a central data warehouse that could unify all these sources into a single source of truth.

My Approach: Architectural Innovation

I took ownership of the project end-to-end, starting with careful stakeholder discussions to identify pain points and key reporting needs. Leveraging my expertise in Python and SQL, I chose PostgreSQL as the warehouse’s database engine—proven, open-source, and powerful. To organize the data logically and efficiently, I implemented a medallion architecture (bronze-silver-gold layers), using DBT (Data Build Tool) to handle transformations seamlessly within a maintainable, version-controlled pipeline. This layered approach made raw data easier to manage and transformations more transparent.

Connecting the Pieces: Robust ETL Pipelines

Data doesn’t move itself. To reliably fetch, transform, and load data from diverse sources—including Jira tickets, internal content management systems, public endpoints, and telecom-specific APIs—I combined two powerful tools:

  • Airbyte, a modern open-source data integration platform, to handle standardized sources with ease.
  • Custom Python scripts to handle niche telecom data, ensuring no critical insights slipped through the cracks.

Together, these tools streamlined the previously complex and manual ETL workflows into a robust, automated pipeline that effortlessly refreshed the warehouse.

From Airflow to Dagster: A Modern Orchestration Evolution

Initially, Apache Airflow drove the orchestration—but as the data grew, so did the complexity. To tackle this, I migrated the data pipeline to Dagster, a cutting-edge orchestration tool built specifically for data-intensive workloads. Dagster’s advanced logging, scalability, and easy monitoring significantly improved the reliability and maintainability of our pipelines, ensuring stakeholders consistently received fresh and accurate data.

Reporting Excellence: Introducing Power BI & Automated Delivery

With the foundational data warehouse established, the next step was enhancing the company’s analytical capabilities. I introduced Power BI as the reporting and visualization layer, creating enterprise-grade dashboards to deliver clear, actionable insights across the organization. Additionally, I automated these reports’ delivery to stakeholders through direct integration with SharePoint, ensuring key decision-makers had immediate and continuous access to critical operational data.

Cross-Functional Leadership and Collaboration

As the project matured, my role naturally evolved into that of a technical lead. Beyond mere implementation, I regularly engaged with stakeholders across the business—from engineering and operations to marketing and customer support—guiding architectural decisions and ensuring that our solutions directly addressed organizational needs. Today, the data warehouse I built serves as the company’s single source of truth, powering strategic decisions and operational excellence through a scalable, maintainable, and future-proof architecture.

Tech stack

  • PostgreSQL
  • DBT (Data Build Tool)
  • Airbyte
  • Python 3.11
  • Dagster
  • Power BI
  • SharePoint automation
  • Grafana
  • Prometheus
  • Docker
  • Git & CI/CD pipelines

What I learned

  • End-to-end deployment (test & production) entirely solo, including setup of CI/CD and infrastructure as code.
  • Implementing robust monitoring and alerting with Grafana and Prometheus.
  • Designing and enforcing a medallion architecture for clear data lineage.
  • Building maintainable, version-controlled transformations using DBT.
  • Migrating orchestration from Airflow to Dagster for improved observability.
  • Automating report delivery via Power BI and SharePoint to streamline stakeholder access.
  • Coordinating cross-functional requirements into scalable data solutions.