Consultant - Sr. Data Engineer
- Affine Analytics
- 5 - 9 years
- Bengaluru
- 1 year ago
- Email to a friend
- Report this job
Job Description
- Design and develop data ingestion frameworks for batch and real-time data processing using distributed systems.
- Create, implement, and maintain streaming data solutions leveraging technologies like Kafka, Apache Spark, and Hadoop.
- Develop data transformation pipelines, ensuring high performance, scalability, and reliability of data systems.
- Work closely with business analysts and stakeholders to understand requirements and translate them into technical ETL solutions.
- Build, deploy, and support application code and analytical models across various environments.
- Integrate open-source data tools to enhance analytics capabilities and operationalize them for enterprise usage.
- Use tools like Python, Java, Hive, Presto, and Scala to prototype and build data engineering solutions.
- Ensure adherence to data governance policies by implementing or validating data lineage, data quality checks, and data classification.
- Participate in on-call support rotation, ensuring high availability and incident resolution.
- Align solutions with company-wide software development lifecycle (SDLC) and DevOps best practices.
- Provide recommendations for process improvements and develop innovative solutions for complex data engineering challenges.
- Collaborate with cross-functional teams to support delivery of critical data projects and enterprise data platforms.
- Bachelors degree in computer science, Engineering, or a related field; masters degree preferred.
- 5+ years of experience in data engineering or a related role.
- Proficiency in Apache Airflow for workflow scheduling and management.
- Strong experience with Hadoop ecosystems, including HDFS, MapReduce, Hive, Presto.
- Expertise in Apache Spark/Scala, Kafka for large-scale and real time data processing.
- Proficient in Python
- Advanced SQL skills for data analysis and reporting.
- Experience with cloud platforms (e.g., AWS, Google Cloud, Azure) preferably GCP is a plus.
- Strong analytical and problem-solving skills.
- Excellent communication and collaboration abilities.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: Affine AnalyticsLocation(s): Bengaluru
Keyskills: Data Engineering Apache Airflow GCP Scala Hadoop Kafka HDFS AWS Apache Spark Python MapReduce
Affine Analytics
SmarTek21 stands at the forefront of innovation, leveraging Generative AI to power advanced data ecosystems and industry-leading conversational interfaces that transform customer interactions. Established in 2006, we seamlessly blend human acumen with cutting-edge artificial intelligence to tackle t...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in