JOB DESCRIPTION
1. Primary Objective:
- Design and build scalable batch and real-time ETL/ELT data pipelines
- Design data models and DataMarts (Customer Profile, Journey, Event) per business requirements
- Operate, monitor, and optimize the data platform on Microsoft Azure and Google Cloud
- Integrate source systems and provide technical support to other teams in the Data Department
- Ensure data quality and adherence to data governance standards across pipelines
- Provide technical guidance, code review, and knowledge sharing within the team
2. Main Responsibilities
- Build and optimize ETL/ELT pipelines using SQL/Python from trading, CRM, and APIs; directly operate CDC flow via Debezium on GKE Autopilot to Azure Databricks
- Design DataMarts using Medallion/Data Vault architecture; collaborate with stakeholders and vendors to clarify requirements and integrate CDP/CRM
- Manage, operate, and monitor resources/costs on Azure (Databricks, ADF, Synapse/Fabric, ADLS) and Google Cloud (GKE Autopilot, Pub/Sub)
- Monitor pipelines, troubleshoot incidents, automate alerting, and continuously improve system performance
- Implement automated data quality checks, reconciliation, and defect handling (using Great Expectations or equivalent)
- Integrate core trading, CRM, and accounting source systems; lead data migrations/upgrades with zero data loss or downtime
- Implement IAM access control, encryption, and data masking; perform backup/recovery and establish CI/CD workflows via Git and Azure DevOps
- Support Analytics, Data Science, and Governance teams in data usage; mentor Data Engineers and review their code
3. Other Responsibilities
- Maintain technical documentation, research new data technologies, and handle ad-hoc data requests
- Perform other tasks as assigned by the Data Engineering Manager
JOB REQUIREMENTS
1. Educational level
- Bachelor’s degree in information technology, Computer Science, Data Science or a related field
2. Professional Experience
- 5+ years of experience as a Data Engineer with strong ETL/ELT pipeline development and large-scale data modeling
- Advanced SQL proficiency and hands-on experience with cloud data platforms (Azure Databricks, BigQuery, Redshift, Synapse, Snowflake, Spark)
- Familiarity with CDP/CRM ecosystems and data integration with marketing platforms (Facebook Ads, Google Analytics, Criteo, RTB House)
- Practical experience with workflow orchestration and transformation tools (dbt, Airflow, Azure Data Factory)
- Strong Python skills for data processing, automation, and API integration (Scala/Java for Spark is a plus)
- Experience with real-time event streaming technologies (Kafka, Pub/Sub, Event Hub)
- Knowledge of Data Governance (data quality, lineage, catalog) and frameworks like Great Expectations or Soda
- Experience with CI/CD for data engineering (Git, GitHub Actions, Azure DevOps) and exposure to Docker/Kubernetes
- Prior involvement in enterprise-scale CDP/CRM integration projects
3. Professional Competencies/Knowledge
- Data Classification and Archiving
- Data Management Process Understanding
- Data Analysis Knowledge
- CRM System Knowledge
- Data Architecture Knowledge
- Database Management Knowledge
- Data Recovery Knowledge
- Information Technology Knowledge
- Project Management Knowledge
