We are hiring a Data Engineer to work on building and maintaining scalable data pipelines for our client Incedo in Chennai. The role involves working with large datasets, developing ETL/ELT pipelines, and ensuring smooth data flow across systems. You will use tools like SQL, Python, Azure Databricks, and PySpark to process and transform data for analytics and business use cases.

Key Skills Required

• Strong SQL (complex queries, joins, aggregations)

• Python for data manipulation and processing

• Azure Databricks

• PySpark

• Data Engineering fundamentals

• ETL / ELT pipeline design and development

• Data pipeline development

• REST API integration

• Basic understanding of Azure / cloud platforms

• Azure Data Factory (ADF) is good to have.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and real-time processing

• Build and optimize ETL/ELT workflows using Azure Databricks and PySpark

• Work with large datasets to ensure data quality, reliability, and performance

• Develop efficient SQL queries for data extraction, transformation, and analysis

• Integrate data from multiple sources using REST APIs and other ingestion methods

• Collaborate with data scientists, analysts, and business teams to deliver data solutions

• Implement and support data engineering best practices across the platform

• Monitor, troubleshoot, and optimize data pipelines for performance and cost efficiency

• Ensure secure and compliant handling of data across cloud environments (Azure)

• Support continuous improvement of data architecture and engineering processes

Shift: 1 PM – 10 PM