Databricks - Senior Engineer @ Iris Software

• Apache Spark (Core, SQL, Structured Streaming)
• PySpark
• Databricks (AWS/Azure)
• Advanced SQL

DevOps & CI/CD

• Jenkins
• Git/GitHub/Bitbucket

Programming

• Python
• SQL
• Java/Scala (Nice to have)

Cloud & Infrastructure

• AWS (S3, EMR, EC2, IAM, CloudWatch)/Azure- Preferred
• Databricks Runtime & Cluster Management- Preferred

Streaming & Integration

• Apache Kafka
• Snowflake integration (preferred)(Nice to have)
• Airflow (preferred)(Nice to have)

Preferred Qualification

• Experience in financial services, regulatory reporting, or enterprise data platforms.
• Hands-on experience in Delta Lake optimization and incremental processing strategies.
• Experience with Snowflake data warehousing.
• Databricks Certification (Professional level preferred).
• Strong understanding of distributed computing principles.

Key Responsibilities

• Design and develop scalable batch and near-real-time ETL/ELT pipelines using Databricks (AWS) and Apache Spark (PySpark, Spark SQL, Structured Streaming).
• Modernize legacy SQL/Hive/stored procedure workflows into distributed Spark-native architectures.
• Perform Spark performance tuning, including:
• Build structured streaming pipelines using Kafka and Spark Structured Streaming.
• Design dimensional data models (Fact/Dimension, SCD Type 2).
• Orchestrate pipelines using Databricks Workflows / Apache Airflow.
• Integrate CI/CD pipelines using Jenkins, Git, Bitbucket/GitHub for automated deployment across DEV/UAT/PROD.

• Data Science and Machine Learning - Data Science and Machine Learning - Databricks
• Programming Language - Python - Apache Airflow
• DevOps/Configuration Mgmt - DevOps/Configuration Mgmt - GitLab,Github, Bitbucket
• Development Tools and Management - Development Tools and Management - CI/CD
• DevOps/Configuration Mgmt - DevOps/Configuration Mgmt - Jenkins
• Data Science and Machine Learning - Data Science and Machine Learning - Apache Spark
• Cloud - AWS - Tensorflow on AWS, AWS Glue, AWS EMR, Amazon Data Pipeline, AWS Redshift
• Cloud - AWS - AWS S3, S3 glacier, AWS EBS
• Cloud - AWS - Amazon IAM, AWS Secrets Manager, AWS KMS, AWS Cognito
• Database - Database Programming - SQL
• Middleware - Message Oriented Middleware - Messaging (JMS, ActiveMQ, RabitMQ, Kafka, SQS, ASB etc)
• Beh - Communication and collaboration.