Data Engineer
- Valethi Technologies
- 5 - 10 years
- Noida, Gurugram
- 7 days ago
- Email to a friend
- Report this job
Job Description
Senior Data Platform Engineer (Full-Time
Employment)
Role Overview
We are looking for a highly skilled Senior Data Platform Engineer / Software Engineer (Data Platform) to
design, build, and scale a next-generation, metadata-driven data infrastructure platform.
This is not a traditional ETL development role. Instead of creating isolated pipelines, you will contribute to
a generic, reusable platform where a single framework powers multiple ingestion and transformation
workloads across the organization. Every architectural decision and code contribution should be scalable,
maintainable, and capable of supporting future use cases.
We are seeking engineers with a strong software engineering mindset who enjoy building frameworks,
reusable components, and enterprise-grade data platforms.
Employment Type: Full-Time (Permanent)
Key Responsibilities
Platform & Framework Engineering
Design and enhance scalable, metadata-driven data ingestion frameworks.
Build reusable platform components that support multiple ingestion patterns rather than one-on one solutions.
Develop highly configurable orchestration logic where behavior is controlled through metadata and
configuration instead of code duplication.
Ensure all implementations follow abstraction frst design principles and can be leveraged across existing and future pipelines.
Advanced Python Engineering
Develop production-grade Python applications following SOLID and DRY principles.
Design, publish, and maintain reusable internal Python packages and shared libraries.
Configure and manage enterprise package distribution using private PyPI or internal artifact repositories.
Manage package versioning, dependency resolution, and environment consistency across multiple projects.
Build maintainable project structures using setup.py and pyproject.toml
PySpark & Databricks Engineering
Build and optimize large-scale distributed data processing pipelines using PySpark.
Work extensively with Delta Lake, Delta Live Tables (DLT), and Databricks platform capabilities.
Optimize Spark jobs through partitioning strategies, ecient joins, JDBC tuning, and execution
planning.
Understand Delta Lake features including ACID transactions, schema evolution, optimization, and time travel.
Design scalable ingestion strategies for high-volume enterprise datasets.
Git & Software Engineering Excellence
Maintain exceptional Git hygiene with disciplined branching, pull request management, and merge strategies.
Perform clean rollback and recovery operations using advanced Git commands when required.
Participate in peer code reviews while ensuring production-quality standards.
Follow modern software engineering practices throughout the development lifecycle.
Infrastructure & Data Governance
Work with Infrastructure-as-Code using Terraform for provisioning cloud resources and services.
Understand the distinction between infrastructure provisioning and data governance responsibilities.
Work within Databricks Unity Catalog to manage data assets, permissions, and governance policies.
Quality Engineering & DevOps
Build automated testing strategies including Unit, Integration, and End-to-End testing.
Implement monitoring, validation, and data quality frameworks.
Support CI/CD pipelines for automated deployments and reliable release management.
Collaborate closely with architecture and platform teams to continuously improve engineering standards.
Required Technical Skills
Python
Advanced Core Python programming
Strong understanding of object-oriented design
Internal package creation and lifecycle management
Private package repositories (PyPI/Artifact Registry)
Dependency management and packaging best practices
PySpark & Databricks
PySpark optimization for distributed workloads
Delta Lake
Delta Live Tables (DLT)
Databricks platform expertise
Performance tuning and troubleshooting
Software Engineering
SOLID principles
Design patterns
Metadata-driven architecture
Framework development
Code modularization and reusability
Version Control
Cloud & DevOps
SQL
Preferred Qualifications
6 to 12 years of experience in Data Engineering or Data Platform Engineering.
Strong experience building enterprise-scale reusable frameworks rather than project-specific pipelines.
Hands-on experience with Databricks Lakehouse architecture.
Experience contributing to internal developer platforms or shared engineering libraries.
Familiarity with enterprise software development practices and Agile methodologies.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: OtherRole Category: OtherRole: OtherEmployement Type: Full timeContact Details:
Company: Valethi TechnologiesLocation(s): Noida, Gurugram
Keyskills: Python
Similar positions
Valethi Technologies
About us We help companies remove process chaos and fast-track business outcomes. We have been helping small to mid-sized and enterprise companies across industries from distribution and manufacturing to consumer goods and healthcare achieve greater process efficiency, intelligent and faster de...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in