🎯 Your mission will be to ensure that IOL’s data flows reliably, efficiently, and at scale from its sources to its consumers — analysts, models, dashboards, and APIs — while meeting the quality, traceability, and performance standards the business needs.
You will also play a multiplier role within the team: we expect you to raise technical standards, reduce technical debt, and introduce AI capabilities where they can generate real value.
🚀 Responsibilities:
Design and maintain batch and streaming data pipelines on Databricks (Delta Lake, Spark Structured Streaming, and Delta Live Tables).
Evolve the Medallion Architecture (Bronze / Silver / Gold).
Implement data quality, governance, and lineage practices using tools such as Unity Catalog and automated testing.
Monitor and optimize pipelines, queries, and storage.
Define and track Data & Analytics KPIs.
Build data models that enable teams to explore and leverage data autonomously and securely.
Incorporate AI and emerging technologies where they can generate value.
📌 Requirements:
4+ years of experience in Data Engineering and production data pipelines.
2+ years of hands-on experience with Databricks, Delta Lake, and Spark.
Advanced proficiency in Python, PySpark, pandas, and SQL.
Experience with at least one cloud platform: Azure, AWS, or GCP.
Experience implementing AI/ML solutions in production.
Degree in Systems Engineering, Computer Science, Mathematics, Statistics, or a related field, or equivalent technical experience.
Experience leading complex data projects or mentoring less experienced team members.
🙏🏻 Nice to have: Databricks Certified Data Engineer Associate/Professional or ML Associate certifications.
