Monitor & Support Data Pipelines: Oversee daily ETL workflows and orchestration using AWS Glue and AWS Step Functions. Ensure smooth execution of scheduled jobs and proactively identify failures or performance issues.
Incident Management & Troubleshooting: Perform initial triage and root cause analysis for pipeline failures. Collaborate with data engineers, DBAs, and application teams to resolve issues quickly.
Automation & Scripting: Develop Python scripts for operational tasks, data validation, and automation. Enhance monitoring and ing systems for proactive issue detection.
API & Integration Support: Manage and maintain REST endpoints for exposing data to downstream systems. Ensure secure and efficient data delivery through APIs.
Documentation & Continuous Improvement: Document recurring issues and contribute to knowledge base articles. Recommend improvements for reliability, scalability, and performance.
Mandatory Skills :
Hands-on experience with AWS services: Glue, Step Functions, EMR, EC2.
Strong Python scripting skills for automation and troubleshooting.
Experience in production support and incident management.
Familiarity with REST API concepts and integration.
Solid understanding of SQL for data validation and debugging.
Preferred Skills :
Experience with FastAPI for building and managing APIs (added advantage).
Knowledge of job scheduling tools (Airflow, Control-M, Autosys). Experience with monitoring tools (CloudWatch, Splunk, Grafana).
Exposure to CI/CD pipelines and DevOps practices.
Understanding of data governance and quality frameworks.
Soft Skills Strong analytical and problem-solving abilities.
Excellent communication and documentation skills.
Ability to work independently and in a collaborative team environment.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full time