Role: Azure Data Engineering Lead

• Experience: 7-10 Years

• Location: Hyderabad

• We need immediate joiners only (Max. 15 days)

• This is work from office role (5 days)

Mandatory Skills:

• Experience in Python and PySpark programming language- At least 4 Years.

• Experience in Azure Data Lake Storage Gen2 (ADLS) and Azure DevOps.

• Experience with Azure Databricks, Apache Spark, and Delta Lake- At least 4 Years.

• Experience in Azure Data Engineering with data pipeline development, architecture, and system optimization- At least 7 Years.

About company: We provides companies with innovative technology solutions for everyday business problems. Our passion is to help clients become intelligent, information-driven organizations, where fact-based decision-making is embedded into daily operations, which leads to better processes and outcomes. Our team combines strategic consulting services with growth-enabling technologies to evaluate risk, manage data, and leverage AI and automated processes more effectively. With deep, big four consulting experience in business transformation and efficient processes.

Role Overview:

We are looking for an accomplished and dynamic Data Engineering Lead to join our team and drive the design, development, and delivery of cutting-edge data solutions. This role requires a balance of strong technical expertise, strategic leadership, and a consulting mindset.

As the Lead Data Engineer, you will oversee the design and development of robust data pipelines and systems, manage and mentor a team of 5 to 7 engineers, and play a critical role in architecting innovative solutions tailored to client needs.

You will lead by example, fostering a culture of accountability, ownership, and continuous improvement while delivering impactful, scalable data solutions in a fast-paced, consulting environment.

 Key Responsibilities:-

1. Architecture & Delivery:

• Lead the end-to-end architecture of data pipelines leveraging Azure Databricks, Delta Lake, and Apache Spark.

• Design and implement the medallion architecture (Bronze, Silver, Gold) to structure scalable data lakes

• Oversee data ingestion from diverse sources, transformation pipelines, and delivery of analytics-ready datasets

• Define best practices for real-time and batch data processing patterns

2. Technical Leadership:

• Mentor and guide a team of data engineers; perform code reviews and enforce engineering best practices

• Collaborate with solution architects, DevOps, data scientists, and business stakeholders

• Set and maintain coding standards, data governance policies, and deployment strategies

3. Platform Engineering:

• Architect CI/CD pipelines for data projects using Azure DevOps and Git

• Optimize Spark jobs, Delta Lake performance, and storage cost management

• Implement observability, data quality checks, and job orchestration across the stack

4. Governance & Security:

• Implement RBAC, Key Vault, managed identities, and other security controls

• Ensure compliance with data privacy regulations (GDPR, HIPAA, etc.)

• Promote data cataloging and lineage through Azure Purview or equivalent tools.

Required Qualifications

• 7 to 10 years of experience in data engineering with hands-on expertise in data pipeline development, architecture, and system optimization.

• 4+ years of hands-on experience with Azure Databricks, Apache Spark, and Delta Lake.

A. Core Competency:

1. Azure Databricks Mastery:

• Expert in Apache Spark (via PySpark, SQL, and optionally Scala).

• Deep experience with Delta Lake: schema evolution, time travel, merge operations, optimization.

• Develop complex notebooks, jobs, and ML pipelines in Databricks.

• Hands-on with Databricks Job Workflows, cluster configurations, and job orchestration.

2. Advanced Azure Ecosystem Proficiency:

• Azure Data Factory (ADF): building robust and reusable pipelines, parameterization, integration with Databricks.

• Azure Synapse Analytics: working knowledge of data movement, analytical workloads.

• Azure Data Lake Storage Gen2 (ADLS): hierarchical namespace, lifecycle management, access policies.

• Azure DevOps: Implementing CI/CD pipelines for notebooks, configuration-as-code.

• Azure Key Vault, Managed Identities, and RBAC: for securing pipeline access and secrets.

3. Data Architecture & Design:

• Designing and implementing Medallion Architecture (Bronze, Silver, Gold layers).

• Dimensional modelling and denormalization for BI/reporting-ready datasets.

• Implementing data lakehouse principles using Delta.

• Designing for scalability, incremental ingestion, and idempotent pipelines.

B. Engineering & Development Skills:

1.Programming & Frameworks:

• Strong with Python and PySpark.

• Skilled in SQL (including analytical functions, optimization).

• Optional: Scala or Java for Spark-based enhancements.

2.Data Quality & Observability:

• Implement data validation frameworks.

• Monitoring with logs, custom alerts, data profiling.

• Experience with data observability tools.

3.Performance Optimization:

• Partitioning, Z-ordering, and caching techniques in Delta.

• Spark tuning and Cost-aware design (e.g., autoscaling, spot instances).

C. Security & Governance:

1.Enterprise Data Governance:

• Metadata cataloging and lineage using Azure Purview.

• Data masking, row-level security in Gold layer.

• Implementing audit trails for access and usage.

D. Leadership & Strategic Thinking:

1.Project Ownership & Team Leadership:

• Driving data strategy and platform choices.

• Mentoring junior engineers; enforcing code quality & best practices.

• Conducting code reviews and architecture walkthroughs.

• Balancing delivery speed with data reliability and governance.

2.Cross-functional Collaboration:

• Working closely with Product Managers, Architects, Data Scientists.

• Translating business requirements into scalable data solutions.

• Strong documentation and communication skills.

E. Nice to Have Skills:

• MLflow for model management and experiment tracking.

• Familiarity with BI tools like Power BI, Tableau.

• DataOps and test automation for pipelines.

Education:

• Bachelor’s or Master’s degree in computer science, Data Engineering, or a related field.

• Certifications such as Azure Data Engineer, Databricks Certified Data Engineer Professional is a plus.

Why Join Us?

• Lead mission-critical data projects with enterprise impact.

• Collaborate with a high-performing team in a cloud-native environment.

• Influence data architecture and engineering strategy across the organization.

• Access to continuous learning, leadership programs, and certification support.