Chief Manager - Data Engineering

KEY RESPONSIBILITIES:

• Architect, Design, develop, and maintain large-scale data pipelines that can handle large datasets from multiple sources.
• Expertise in real-time data replication and batch processing of data using distributed computing platforms like Spark, Kafka, etc.
•  Optimize performance of data processing jobs and ensure system scalability and reliability.
•  Collaborate with DevOps teams to manage infrastructure, including cloud environments like AWS
•  Collaborate with data scientists, analysts, and business stakeholders to develop tools and platforms that enable advanced analytics and reporting.
•  Lead and mentor junior data engineers, providing guidance on best practices, code reviews, and technical solutions.
• Evaluating and implementing new frameworks, tools for data engineering
• Strong analytical and problem-solving skills with attention to detail.
• To maintain a healthy working relationship with the business partners/users and other MLI departments
• Responsible for overall performance, cost and delivery of technology solutions

KeyTechnical competencies/skills required

• Hands-on experience with AWS services such as S3, DMS, Lambda, EMR, Glue, Redshift,RDS (Postgres) Athena, Kinesics, etc.
• Expertise in data modelling and knowledge of modern file and table formats.
• Expertise in data replication tool like Qlik replicate, AWS DMS
• Proficiency in programming languages such as Python, PySpark, SQL/PLSQL for implementing data pipelines and ETL processes.
• Experience data architecting or deploying Cloud/Virtualization solutions (Like Data Lake, EDW, Mart ) in enterprise
• Knowledge of modern data stack and keeping the technology stack refreshed.
• Knowledge of DevOps to perform CI/CD for data pipelines.
• Knowledge of Data Observability, automated data lineage and metadata management would be an added advantage. 
• Cloud/hybrid cloud (preferable AWS) solution for data strategy for Data lake, BI and Analytics 
• Set-up logging, monitoring, alerting, dashboards for cloud solution and data solution
• Experience with data warehousing concepts.