AI Data Engineer

About the role

We are looking for an AI Data Engineer to design, build, and operate data pipelines that power AI/ML and agentic applications. You will own the end‑to‑end data flow, from ingestion through to feature‑ready, retrieval‑ready datasets, ensuring reliability and performance at scale.

Key responsibilities

• Develop, maintain, and scale batch and streaming data pipelines used by multiple teams.
• Design and optimise Spark/Databricks jobs, handling skew, shuffle, partitioning and memory constraints.
• Manage object‑store data layers (S3, ADLS, MinIO) with appropriate partitioning, file formats (Parquet, Delta, Iceberg) and compaction strategies.
• Integrate heterogeneous data sources—databases, APIs, event streams, and files—into coherent data flows.
• Process unstructured and multimodal content, parsing and normalising it for AI consumption.
• Collaborate with AI squads to deliver business‑driven use cases, including feature engineering for ML models.
• Participate in risk management, governance, and regulatory compliance activities.

Required profile

• Proven experience building production‑grade data pipelines at scale.
• Strong understanding of system design principles such as throughput, latency, consistency and cost.
• Ability to troubleshoot, perform backfills, manage schema changes and provide on‑call support.
• Excellent communication skills and a collaborative attitude.

Required skills

• SQL
• Python
• Apache Spark
• Databricks (or equivalent distributed compute platform)
• Azure and/or AWS fundamentals
• Data‑warehouse and data‑lake concepts
• Distributed systems design
• Flink (preferred)
• S3, ADLS, MinIO object stores
• Parquet, Delta Lake, Iceberg file formats
• Kafka, Kinesis, Pulsar streaming platforms