Lead Software Engineer
- Virtusa
- 7 - 12 years
- Hyderabad
- 1 month ago
- Email to a friend
- Report this job
Job Description
Job Description
Job Description Data Engineer (Mid-Level, AWS & Databricks)
Role Overview
As a Mid-Level Data Engineer, you will develop and maintain the data pipelines that form the backbone of our AWS Data Lakehouse. Your mission is to translate business requirements into efficient PySpark code and ensure our SQL Server data is accurately migrated and synchronized with Databricks.
Key Responsibilities
ETL Development Develop and debug data pipelines in Databricks using PySpark and SparkSQL.
Data Ingestion Implement data movement from SQL Server to Amazon S3 using efficient batch and incremental loading techniques.
Database Operations Query and manage SQL Server environments to validate data consistency during the migration phase.
Modernization Refactor existing SQL-based transformations into modular Python scripts or Scala functions.
Required Skills & Qualifications
4-6 years of experience in Data Engineering.
Hands-on Experience Building production pipelines in Databricks on AWS.
Core Competencies Strong Python, PySpark, and complex SQL (T-SQL preferred).
AWS Basics Familiarity with S3 bucket management and basic IAM roles.
Adaptability Ability to work across Python and SQL comfortably, with an interest in learning/using Scala for performance tuning.
Nice to Have
Prior exposure to SSIS (this is a plus, not a requirement).
Knowledge of AWS Glue or AWS Lambda for lightweight orchestration.Role & responsibilities
Preferred candidate profile
Perks and benefits
Job Classification
Industry: Financial ServicesFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Technical LeadEmployement Type: Full timeContact Details:
Company: VirtusaLocation(s): Hyderabad
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in