Senior Technical Lead

Job Summary

Role

• Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
• Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
• Carry out data modelling in Databricks.
• Expose the Delta tables to the visualisation layer.
• Ensure that the data pipelines are optimised.

Skillsets

• Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
• Fluent in PySpark and Databricks SQL.
• Proven experience building Databricks architecture patterns.

Familiarity with Unity Catalog for data governance and access control.

Key Responsibilities

Role

• Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
• Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
• Carry out data modelling in Databricks.
• Expose the Delta tables to the visualisation layer.
• Ensure that the data pipelines are optimised.

Skillsets

• Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
• Fluent in PySpark and Databricks SQL.
• Proven experience building Databricks architecture patterns.

Familiarity with Unity Catalog for data governance and access control.

Skill Requirements

Role

• Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
• Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
• Carry out data modelling in Databricks.
• Expose the Delta tables to the visualisation layer.
• Ensure that the data pipelines are optimised.

Skillsets

• Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
• Fluent in PySpark and Databricks SQL.
• Proven experience building Databricks architecture patterns.

Familiarity with Unity Catalog for data governance and access control.

Other Requirements

Role

• Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
• Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
• Carry out data modelling in Databricks.
• Expose the Delta tables to the visualisation layer.
• Ensure that the data pipelines are optimised.

Skillsets

• Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
• Fluent in PySpark and Databricks SQL.
• Proven experience building Databricks architecture patterns.

Familiarity with Unity Catalog for data governance and access control.