Engineering Expert - AI & Data Products, Runtime Platform
- Sapital Recruitments
- 4 - 8 years
- Bengaluru
- 21 days ago
- Email to a friend
- Report this job
Job Description
Software-Design and Development Engineering Expert - AI Data Products, Runtime Platform As an Engineering Expert, you will be instrumental in the expansion of our Foundation Services team in Bangalore. Your profound expertise in distributed data processing, Spark optimization, data transformation pipelines, data engineering architecture principles, AI Agent development, and LLM-driven engineering will drive scalable data solutions and intelligent automation, reinforcing SAP Data Analytics' pivotal role in SAP's Data AI strategy. This is a product expert role requiring deep hands-on expertise across the modern cloud-native stack.
What you'll build - Lead the design and execution of optimized data processing solutions at scale.
- Define and enforce data engineering architecture principles including data lineage, schema evolution, idempotency, fault tolerance, data partitioning strategies, and separation of concerns across processing layers.
- Spearhead Spark optimization strategies using PySpark to maximize performance and scalability in data processing.
- Architect and refine pluggable data transformation pipelines for efficient processing of large datasets.
- Establish architectural standards for data lakehouse patterns, batch vs. streaming paradigms, data contracts, and metadata-driven pipeline design.
- Design and develop AI Agents, MCP (Model Context Protocol) Servers, and MCP Tools to enable intelligent, autonomous workflows.
- Craft and optimize advanced prompts using prompt engineering techniques (chain-of-thought, few-shot, Zero-shot prompting) to drive reliable and high-quality LLM outputs in product workflows.
- Contribute to LLM fine-tuning and training efforts - including data curation, training dataset preparation, RLHF (Reinforcement Learning from Human Feedback), evaluation benchmarking, and model alignment for domain-specific use cases.
- Build and maintain microservices architectures that support modular, scalable product capabilities.
- Implement GitOps practices using Argo CD and Helm to advance CI/CD pipelines and operational efficiency on Kubernetes.
- Apply advanced SQL skills to refine data transformations across vast datasets.
- Integrate AI ML technologies into high-performance data solutions, staying informed on emerging trends in agentic AI and LLM-based tooling.
- Utilize SAP HANA Spark to drive innovation in data engineering processes.
- Drive architectural decisions around data quality, observability, scalability, and reliability across the platform.
- Mentor junior engineers, fostering a culture of continuous improvement and technical excellence.
- Collaborate effectively with global stakeholders to achieve successful project outcomes.
- Extensive experience in data engineering, distributed data processing, and expertise in SAP HANA or similar databases.
- Deep understanding of data engineering architecture principles - including data pipeline architecture, ELT/ETL patterns, Delta tables and their optimization (Z-ordering, compaction, vacuum, partitioning, liquid clustering), schema-on-read vs. schema-on-write, and pipeline orchestration best practices.
- Proficiency in Python (PySpark) is essential.
- Hands-on experience in AI Agent development, MCP Server development, and MCP Tool development.
- Strong prompt engineering skills - experience designing, testing, and iterating on prompts for LLMs (GPT-4, Claude, Gemini, or similar) across diverse use cases including code generation, data transformation, and agentic reasoning.
- Strong expertise in microservices architecture and development.
- Deep knowledge of Kubernetes, Helm, and Argo CD for container orchestration and GitOps-driven deployments.
- Advanced understanding of Spark optimization and scalable data processing techniques.
- Proven experience in architecting data transformation pipelines with a focus on reusability, extensibility, and maintainability.
- Strong understanding of AI ML technologies, LLMs, and industry trends in agentic systems.
- Knowledge of data governance, data quality frameworks, and observability in distributed data systems.
- Familiarity with LLM frameworks and tooling such as LangChain, LlamaIndex, and LangGraph.
- Effective communication skills within a global, multi-cultural environment.
- Proven track record of leadership in data processing, platform initiatives, and product engineering.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: DBA / Data warehousingRole: Database Architect / DesignerEmployement Type: Full timeContact Details:
Company: SAPLocation(s): Bengaluru+ View Contact
Keyskills: kubernetes cd continuous integration python data processing ci/cd pyspark engineering data engineering helm artificial intelligence microservices sql spark container orchestration data transformation sap hana etl communication skills architecture
Fraud Alert to job seekers!
₹ Not Disclosed
Sapital Recruitments
Sapital Recruitments specializes in IT and Healthcare hiring. We take a practical, transparent approach to connect companies with right talent. Founded on 6th February 2025, Sapital Recruitments is a dynamic and performance-driven placement and HR solutions firm dedicated to building strong workfor...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in