About the Role

The Data Engineer I role supports a GenAI-powered insights assistant initiative by building and scaling ingestion and embedding pipelines for unstructured knowledge bases. This position involves building batch/streaming data pipelines, implementing data consistency checks, and developing cross-domain metadata services to enhance AI-driven insights.

Key Responsibilities

• Build batch and streaming data pipelines using Spark and AWS streaming services
• Implement automated checks to ensure data consistency across different data types
• Define and maintain data contracts with source teams to keep schemas consistent
• Develop cross-domain metadata services linking structured and unstructured data catalogs
• Create APIs and event-driven workflows integrating AI insights with business tools

Required Skills & Qualifications

Must Have:

• 1+ years of data engineering experience
• Experience with data modeling, warehousing and building ETL pipelines
• Experience with one or more query language (e.g., SQL, PL/SQL, DDL, MDX, HiveQL, SparkSQL, Scala)
• Experience with one or more scripting language (e.g., Python, KornShell)

Nice to Have:

• Experience with big data technologies such as: Hadoop, Hive, Spark, EMR
• Experience with any ETL tool like, Informatica, ODI, SSIS, BODI, Datastage, etc.
• Familiarity with RAG (Retrieval-Augmented Generation) principles
• AWS experience: Lambda, S3, SageMaker, Bedrock Knowledge Bases