Vage

Senior Data Engineer

S&P GlobalHyderabad, TelanganaPosted 18 November 2026

Data & AISenior & Lead

Sign in free to apply on the company's careers page.

Skills mentioned

  • SQL
  • LLMs
  • Prompt Engineering
  • Spark
  • Databricks
  • AWS
  • CI/CD
  • Serverless
  • IAM
  • Incident Response
  • Compliance
  • Stakeholder Management

About the job

About the Role:

Grade Level (for internal use): 11 Key Responsibilities Data Pipeline Transition and Platform Delivery • Partner with the core Databricks team to support the transition of existing data pipelines to the target enterprise data platform. • Provide hands-on technical guidance for enhancing, stabilizing, and optimizing pipelines after transition, with a focus on performance, scalability, maintainability, operational reliability, and secure AWS deployment patterns. • Implement and promote repeatable engineering patterns for ingestion, transformation, testing, deployment, monitoring, and support across onboarded datasets. • Ensure pipelines are designed and operated in a way that supports long-term platform consistency, reliability, observability, and ease of maintenance. • Support the design and operation of cloud-native data pipelines using AWS services such as: • Amazon S3 for durable data lake storage • AWS Glue for data integration, cataloging, and processing • AWS Lambda for event-driven processing • Amazon Kinesis for streaming use cases • AWS Lake Formation for governed data lake controls • AWS IAM for secure access control

• Promote the use of encryption, environment-level controls, access policies, and platform guardrails to support secure access to platform resources and data products. • Apply lakehouse technologies and concepts such as Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, and governed data access patterns.

AI-Assisted Engineering and Context Engineering • Use AI-assisted engineering tools such as  Claude, GitHub CoPilot, LLMs, and similar platforms  to accelerate data engineering delivery while maintaining enterprise quality and security standards. • Apply AI tooling to support common engineering activities, including: • Code generation • Code refactoring • SQL and PySpark optimization • Unit test generation • Documentation creation • Pipeline analysis • Troubleshooting and debugging • Data quality rule suggestions • Runbook and operational support documentation

• Apply  prompt engineering and context engineering  practices to improve the accuracy, relevance, and reliability of AI-generated outputs. • Structure technical context for AI-assisted work, including requirements, schemas, data contracts, architecture constraints, coding standards, logs, error messages, acceptance criteria, and governance requirements. • Create or contribute to reusable prompt templates, context packs, and engineering playbooks for common data platform activities such as Databricks migration, AWS data engineering, pipeline troubleshooting, and test generation. • Review, validate, and refine AI-generated code, recommendations, documentation, and analysis before applying them to production-grade engineering work. • Ensure responsible use of AI tooling by following enterprise standards for security, privacy, sensitive data protection, compliance, and human review.

Data Onboarding and Asset-Agnostic Enablement • Support asset-agnostic data onboarding patterns that can accommodate a broad range of data assets, domains, and source systems without requiring bespoke platform redesign for each use case. • Work with platform, data engineering, architecture, governance, and business-aligned teams to simplify and standardize how data is ingested, transformed, governed, and published to the enterprise platform. • Help offshore and global teams adopt common onboarding frameworks, technical standards, and delivery practices that improve speed, consistency, and reduce friction. • Establish reusable ingestion and processing patterns across batch, streaming, and event-driven use cases. • Create or contribute to reusable technical assets such as: • Design patterns • Reference implementations • Onboarding templates • Pipeline frameworks • Technical documentation • Operational runbooks • AI-assisted engineering templates

• Support metadata-driven and configuration-driven pipeline patterns to improve scalability, consistency, and maintainability.

Data Mastering Platform Integration • Support integration of platform pipelines and datasets with the enterprise data mastering platform. • Collaborate with upstream and downstream stakeholders to ensure mastered data can be consumed reliably through standardized interfaces and governed data flows. • Help implement data quality controls, reconciliation processes, metadata alignment, and stewardship workflows required to support trusted mastered data in the platform. • Contribute to issue resolution and continuous improvement related to mastering-related ingestion, enrichment, and distribution workflows. • Support data mastering capabilities aligned with platforms such as  NeoXam DataHub , including: • Data acquisition • Cleansing • Enrichment • Mastering • Reconciliation • Golden copy generation • Downstream distribution of trusted data products

• Work with business and platform stakeholders to support golden record / golden copy generation and maintenance for key financial datasets, including reference data, instruments, business entities, market data, investment data, risk data, and related mastered domains.

Technical Leadership and Engineering Excellence • Serve as a senior technical individual contributor for data platform engineering, with hands-on involvement across pipeline migration, lakehouse architecture, AWS-native data engineering, governance, AI-assisted engineering, and mastering integrations. • Provide technical guidance to engineers, delivery partners, and platform stakeholders without direct people management responsibility. • Contribute to architecture discussions, design reviews, code reviews, implementation planning, technical standards, and production readiness reviews. • Translate architectural direction into practical engineering tasks, implementation patterns, and reusable technical assets. • Promote engineering best practices including: • Version control • Automated testing • CI/CD • Infrastructure as code • Release automation • Monitoring and alerting • Incident response • Documentation • Production support readiness

• Help establish cloud engineering standards for infrastructure automation, release management, and environment promotion using tools and services such as AWS CodePipeline, AWS CodeBuild, and infrastructure automation frameworks. • Drive operational discipline across production data pipelines, including observability, logging, telemetry, support models, service ownership, and incident management.

Collaboration, Governance, and Platform Standards • Partner effectively with global platform, architecture, governance, security, Databricks-aligned, and data mastering teams to ensure offshore delivery aligns with enterprise standards. • Act as a technical bridge between platform direction and engineering execution. • Support governance requirements through appropriate controls around: • Data lineage • Schema consistency • Data quality • Metadata • Retention • Access control • Encryption • Auditability • Secure data distribution

• Ensure monitoring and operational health practices are in place using logging, alerting, telemetry, dashboards, and AWS-native operational tooling where appropriate. • Help improve consistency and reusability across data onboarding and pipeline delivery by contributing to platform guardrails, standards, engineering playbooks, and reusable implementation patterns.

Required Qualifications Professional Experience • 6–8+ years of experience in data engineering, data platforms, cloud data architecture, software engineering, or related engineering domains. • Demonstrated experience operating as a senior individual contributor, technical lead, senior data engineer, lead data platform engineer, or equivalent technical role. • Proven ability to support technical initiatives and influence engineering outcomes in a complex, execution-focused environment. • Experience partnering with global or distributed teams to deliver platform and pipeline initiatives across time zones. • Ability to provide technical guidance and improve delivery quality without direct people management authority.

Technical Expertise • Strong hands-on experience with modern data engineering and pipeline development, including batch and/or streaming data workflows. • Experience working with Databricks-based data platforms and supporting migration or transition of pipelines into a lakehouse-oriented architecture. • Strong familiarity with AWS cloud-native data engineering, including services such as: • Amazon S3 • AWS Glue • AWS Lambda • AWS Lake Formation • Amazon Kinesis • AWS IAM • AWS-native monitoring, logging, and security capabilities

• Working knowledge of technologies and concepts such as: • Delta Lake • Apache Iceberg • Databricks Unity Catalog • Metadata-driven pipelines • Data lake governance • Lakehouse architecture • Catalog-driven processing

• Experience supporting data integration patterns involving mastering, MDM, reference data, market data, investment data, risk data, or trusted data distribution workflows. • Solid understanding of data quality, schema management, lineage, metadata, access control, encryption, and operational support for production data pipelines. • Experience with AWS IAM, data lake governance, policy-based access controls, and secure platform operations. • Familiarity with engineering best practices such as version control, automated testing, CI/CD, infrastructure automation, release management, and monitoring. • Experience implementing data platform observability and reliability practices, including alerting, monitoring, telemetry, dashboards, and production support procedures.

AI Tooling and LLM-Enabled Engineering • Practical experience using AI tooling such as  Claude, GitHub CoPilot, LLMs, or similar AI-assisted development platforms  in an engineering context. • Experience applying LLMs to data engineering or software engineering workflows, including code generation, refactoring, test creation, documentation, debugging, troubleshooting, and pipeline analysis. • Understanding of prompt engineering and context engineering, including how to structure technical context, examples, constraints, schemas, logs, coding standards, and acceptance criteria to improve AI-generated outputs. • Ability to use AI-assisted workflows to support Databricks development, AWS data engineering, SQL, PySpark, pipeline migration, data quality analysis, and operational support activities. • Ability to evaluate, validate, and refine AI-generated code or recommendations before applying them to production-grade engineering work. • Understanding of responsible AI usage in enterprise environments, including security, privacy, compliance, human review, and protection of sensitive or regulated information.

Collaboration and Execution • Strong communication and stakeholder management skills, with the ability to work effectively across offshore and global teams. • Ability to translate architectural direction into actionable technical deliverables, implementation plans, and reusable engineering patterns. • Comfortable operating in a role that blends hands-on engineering, technical guidance, and delivery enablement. • Strong problem-solving skills and ability to navigate ambiguity in complex enterprise data platform environments. • Ability to partner with engineers, architects, governance stakeholders, and business-aligned data teams to drive practical outcomes.

Preferred Qualifications • Experience supporting enterprise data onboarding frameworks across multiple business domains, asset classes, or source system types. • Familiarity with enterprise data cataloging, metadata management, business glossaries, lineage, operational metadata platforms, or semantic data layers. • Knowledge of  semantic modeling  is a plus, particularly for supporting consistent business definitions, governed data consumption, data discoverability, an

Collected from the employer's own careers page by Vage. Always apply on the company's site; Vage never charges you to apply.