About the role
- We are looking for an experienced Senior Azure Data Engineer with 8+ years of hands-on experience in designing, developing, and maintaining scalable cloud-based data engineering solutions. The ideal candidate will have strong expertise in Azure Synapse Analytics, Azure Data Factory, Azure Databricks, Azure Data Lake Storage Gen2, Python, PySpark, Spark SQL, and Power BI.
- The role involves building robust ETL/ELT pipelines, implementing incremental and CDC-based data processing, optimizing distributed data workloads, and developing enterprise data models across Development, QA, and Production environments.
- The candidate should be capable of independently owning technical solutions, collaborating with architects and business stakeholders, contributing to Agile delivery, and mentoring other engineers.
Job Description
- Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Storage, Azure Key Vault, Azure Functions, Logic Apps, and ADLS Gen2.
- Build data ingestion and transformation solutions using Python, PySpark, Spark SQL, and SQL for large-scale batch and distributed data processing.
- Configure and manage Azure Data Factory components, including linked services, datasets, integration runtimes, pipelines, triggers, parameters, and activities.
- Develop data solutions using Synapse SQL pools, Synapse Spark pools, Databricks clusters, notebooks, Azure Functions, and Logic Apps.
- Implement incremental data-loading frameworks using Delta Lake, MERGE operations, upserts, deduplication, and Change Data Capture (CDC).
- Design reusable and metadata-driven pipeline frameworks to support multiple data sources and business domains.
- Develop Python-based database integrations with SQL Server, Oracle, and MySQL, including stored procedure execution and dataset processing.
- Design and implement enterprise data warehouse and lakehouse solutions using dimensional modeling, Star Schema, Snowflake Schema, and Slowly Changing Dimensions (SCD).
- Implement data-quality checks, exception handling, logging, monitoring, alerting, and operational support mechanisms.
- Optimize Spark workloads, SQL queries, pipeline execution, cluster configurations, and cloud resources for performance and cost efficiency.
- Develop and maintain Power BI semantic models, dashboards, and reports using Power Query and DAX.
- Implement CI/CD processes for promoting data engineering code across Development, QA, and Production environments using GitHub or Azure DevOps.
- Ensure source control, branching, pull requests, code reviews, testing, and deployment best practices.
- Support data governance and security through Databricks Unity Catalog, role-based access control (RBAC), and data-access policies.
- Participate in Agile/Scrum ceremonies, conduct code reviews, mentor engineers, and contribute to engineering standards.
- Collaborate with data architects, business stakeholders, BI developers, and engineering teams to translate requirements into scalable technical solutions.
- Own technical deliverables independently from solution design and development through testing, deployment, and production support.
Qualifications
- Relevant educational qualifications in Computer Science, Information Technology, Engineering, Data Science, or a related discipline.
- 8+ years of hands-on experience in Data Engineering, with strong expertise in Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage Gen2, and Azure Storage.
- Strong experience designing, developing, and maintaining scalable ETL/ELT pipelines and enterprise data engineering solutions.
- Proficiency in Python, PySpark, Spark SQL, and SQL, with experience using Pandas, NumPy, and Python database connectivity libraries.
- Hands-on experience integrating Python applications with SQL Server, Oracle, and MySQL, including stored procedure execution and returned dataset processing.
- Strong understanding of Azure Data Factory components, including linked services, integration runtimes, datasets, pipelines, activities, triggers, and parameters.
- Experience working with Azure Databricks clusters, notebooks, Spark workloads, and distributed data processing.
- Experience with Azure Synapse Spark pools and SQL pools.
- Hands-on experience implementing incremental loads, Delta Lake processing, MERGE/upsert operations, deduplication, and CDC.
- Experience designing enterprise data warehouse and lakehouse solutions, including dimensional modeling, Star Schema, Snowflake Schema, fact and dimension tables, and SCD.
- Strong experience with GitHub or Azure DevOps and CI/CD-based code promotion across Development, QA, and Production environments.
- Experience developing Power BI semantic models, dashboards, and reports, with strong knowledge of Power Query and DAX.
- Experience implementing data-quality checks, logging, monitoring, exception handling, and pipeline alerting.
- Experience managing credentials and secrets securely using Azure Key Vault.
- Exposure to Databricks Unity Catalog, data lineage, RBAC, and enterprise data governance practices.
- Experience optimizing Spark workloads, SQL queries, data pipelines, cluster configurations, and cloud resource utilization.
- Experience working in Agile/Scrum environments, conducting code reviews, establishing engineering standards, and mentoring junior and mid-level engineers.
- Cloud & Azure Technologies: Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Storage, Azure Functions, Logic Apps, and Azure Key Vault.
- Programming & Data Processing: Python, PySpark, Spark SQL, SQL, Pandas, NumPy, and Python database connectivity libraries.
- Databases: SQL Server, Oracle, MySQL, Azure SQL, Synapse SQL, stored procedures, and database integrations.
- Data Engineering: ETL/ELT pipelines, Delta Lake, Change Data Capture (CDC), incremental loads, MERGE/upsert operations, deduplication, data ingestion, data transformation, and data-quality validation.
- Data Modeling & Architecture: Dimensional Modeling, Star Schema, Snowflake Schema, fact and dimension tables, Slowly Changing Dimensions (SCD), Data Warehousing, and Medallion Lakehouse Architecture.
- Business Intelligence & Analytics: Power BI, semantic models, dashboards, reports, Power Query, and DAX.
- DevOps & CI/CD: GitHub, Azure DevOps, CI/CD pipelines, source control, branching, pull requests, code reviews, automated testing, and deployment management.
- Governance & Security: Databricks Unity Catalog, RBAC, Azure Key Vault, data lineage, data governance, and data-access policies.
- Performance Optimization: Spark job optimization, SQL query tuning, pipeline performance, cluster configuration, distributed processing, and cloud cost optimization.
- Methodologies: Agile, Scrum, sprint planning, backlog refinement, daily stand-ups, sprint reviews, and retrospectives.
- Leadership & Collaboration: Independent technical ownership, solution design, stakeholder collaboration, code reviews, technical mentoring, production troubleshooting, and engineering best practices.
- Key Competencies: Azure Data Engineering, Databricks, Azure Synapse Analytics, Azure Data Factory, Python, PySpark, SQL, ETL/ELT, MLOps-ready Data Pipelines, Data Modeling, Power BI, CI/CD, Data Governance, and Cloud Optimization.
Perks & Benefits
Medical Insurance
Choose from Health Savings Plan, HRA, or PPO, whichever best fits your healthcare needs and budget.
Parental leave
Take the time you need to welcome and care for your growing family, with leave designed to support you through this milestone.
Training and development
Access structured, project-specific training and continuous learning opportunities that help you grow in your role and career.
Two-way transport
Enjoy stress-free commuting with two-way cab pick-up and drop-off facilities for every shift.
