AI Data Engineer

About the role

The AI Data Engineer will design, build, and operate the data pipelines and infrastructure that power AI/ML and agentic applications. You will own the end‑to‑end flow from ingestion to retrieval‑ready, feature‑ready data, working closely with product and engineering squads.

Key responsibilities

• Develop and maintain production‑grade batch and streaming pipelines that serve multiple teams.
• Design, tune, and operate Spark/Databricks jobs, handling skew, shuffle, partitioning, and memory optimisation.
• Manage object‑store backbones (S3, ADLS, MinIO) with expertise in partitioning, file formats (Parquet, Delta, Iceberg) and cost‑performance trade‑offs.
• Integrate heterogeneous data sources – databases, APIs, event streams, and files – into reliable, coherent flows.
• Parse, chunk, and normalise unstructured and multimodal content for AI consumption.
• Make system‑design decisions around throughput, latency, consistency, idempotency, and cost.
• Collaborate on feature engineering for ML, ensuring offline/online consistency.
• Contribute to governance, risk management, and stakeholder communication within the AI squad.

Required profile

• Proven experience building and operating large‑scale data pipelines in production.
• Strong understanding of lakehouse platforms (Databricks, Delta Lake, Unity Catalog).
• Ability to design robust data integration across diverse sources.
• Excellent communication skills and a collaborative attitude.
• Experience leading or mentoring a team is a plus.

Required skills

• Spark
• Databricks (or equivalent distributed compute)
• Delta Lake / Unity Catalog
• S3, ADLS, MinIO
• Parquet, Delta, Iceberg file formats
• Kafka, Kinesis, Pulsar
• Flink