We are looking for a Senior Data Engineer with a deep understanding of Apache Spark (Scala & PySpark), Kafka Streams (Java), AWS services, Snowflake, Apache Iceberg, Tableau, and Data Lake architectures.
As a senior member of our team, you will be responsible for leading the design, implementation, and optimization of large-scale data systems, real-time streaming solutions, and cloud-based data platforms.
You will work with other engineers to deliver high-quality data solutions, mentor junior team members, and collaborate closely with cross-functional teams to solve complex business problems.

Key Responsibilities:

• Lead the design and development of scalable, high-performance data architectures on AWS, leveraging services such as S3, EMR, Glue, Redshift, Lambda, and Kinesis.

• Architect and manage Data Lakes for handling structured, semi-structured, and unstructured data.

• Design and build complex data pipelines using Apache Spark (Scala & PySpark), Kafka Streams (Java), and cloud-native technologies for batch and real-time data processing.

• Optimize these pipelines for high performance, scalability, and cost-effectiveness.

• Develop and optimize real-time data streaming applications using Kafka Streams in Java.

• Build reliable, low-latency streaming solutions to handle high-throughput data, ensuring smooth data flow from sources to sinks in real-time.

• Manage Snowflake for cloud data warehousing, ensuring seamless data integration, optimization of queries, and advanced analytics.

• Implement Apache Iceberg in Data Lakes for managing large-scale datasets with ACID compliance, schema evolution, and versioning.

• Design and maintain highly scalable Data Lakes on AWS using S3, Glue, and Apache Iceberg.

• Ensure data is easily accessible, stored in optimal formats, and well-integrated with downstream analytics systems.

• Work with business stakeholders to create actionable insights using Tableau.

• Build data models and dashboards that drive key business decisions, ensuring that data is easily accessible and interpretable.

• Continuously monitor and optimize Spark jobs, Kafka Streams processing, and other cloud based data systems for performance, scalability, and cost.

• Implement best practices for stream processing, batch processing, and cloud resource management.

• Lead and mentor junior engineers, fostering a culture of collaboration, continuous learning, and technical excellence.

• Ensure high-quality code delivery, adherence to best practices, and optimal use of resources.

• Work closely with Data Scientists, Product Managers, and DevOps teams to understand business needs and deliver impactful data solutions.

• Participate in technical discussions, from system design to data governance.

• Ensure that data pipelines, architectures, and systems are thoroughly documented and follow coding and design best practices.

• Promote knowledge-sharing across the team to maintain high standards for quality and scalability.
  
  Required Skills & Qualifications:
  Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or related field (or equivalent work experience).
  
  Experience:
  • 5+ years of experience in Data Engineering or a related field, with a proven track record of designing, implementing, and maintaining large-scale distributed data systems.
  • Proficiency in Apache Spark (Scala & PySpark) for distributed data processing and real-time analytics.
  • Hands-on experience with Kafka Streams using Java for real-time data streaming applications.
  • Strong experience in Data Lake architectures on AWS, using services like S3, Glue, EMR, and data management platforms like Apache Iceberg.
  • Proficiency in Snowflake for cloud-based data warehousing, data modeling, and query optimization.
  • Expertise in SQL for querying relational and NoSQL databases, and experience with database design and optimization.
  
  Technical Skills:
  • Strong Experience in building ETL pipelines using Spark(Scala & Pyspark) and maintain them.
  • Proficiency in Java, particularly in the context of building and optimizing Kafka Streams applications for real-time data processing.
  • Experience with AWS services (e.g., Lambda, Redshift, Athena, Glue, S3) and managing cloud infrastructure.
  • Expertise with Apache Iceberg for handling large-scale, transactional data in Data Lakes, supporting versioning, schema evolution, and partitioning.
  • Experience with Tableau for business intelligence, dashboard creation, and data visualization is a plus.
  • Knowledge of CI/CD tools and practices, particularly in data engineering environments.
  • Familiarity with containerization tools like Docker and Kubernetes for managing cloud based services.
  
  Soft Skills:
  • Excellent problem-solving skills, with a strong ability to debug and optimize large-scale distributed systems.
  • Strong communication skills to engage with both technical and non-technical stakeholders.
  • Proven leadership ability, including mentoring and guiding junior engineers.
  • A collaborative mindset and the ability to work across teams to deliver integrated solutions.
  
  Preferred Qualifications:
  • Experience with stream processing frameworks like Apache Flink or Apache Beam.
  • Knowledge of machine learning workflows and integration of ML models in data pipelines.
  • Familiarity with data governance, security, and compliance practices in cloud environments.
  • Experience with DevOps practices and infrastructure automation tools such as Terraform or CloudFormation.