Integrating heterogeneous data sources with varying schemas and update frequencies
Developed a metadata-driven ETL architecture with Azure Data Factory
A metadata-driven data platform for consolidating operational sources into analytics-ready workflows.
Context
The project was about creating a reliable foundation rather than a flashy front end. Multiple data sources needed to move through a coherent platform where quality, lineage, and update patterns could be reasoned about.
Build
I helped shape Azure Data Factory and Databricks workflows around metadata-driven ETL, data quality checks, and maintainable orchestration. The outcome was a platform pattern that could scale with additional sources instead of becoming a one-off pipeline.
Engineering
Developed a metadata-driven ETL architecture with Azure Data Factory
Implemented advanced data quality rules and monitoring
Created optimized Databricks workflows for large-scale data processing
Built a self-service data catalog for business users
Outcome
Stack