What you’ll do:
- Design and build batch and streaming data pipelines.
- Monitor pipelines, troubleshoot failures, and fix root causes.
- Optimize Spark/SQL jobs and run backfills safely.
- Ensure data quality, including audited financial data.
- Use AI to automate repetitive work.
Requirements
Must-have:
- 3+ years of experience in Data Engineering, including at least 1 year building and operating production pipelines.
- Strong SQL and Python skills.
- Hands-on experience with Spark, including performance tuning and troubleshooting OOM errors and data skew.
- Experience with Airflow or a similar orchestration tool.
- Understanding of the infrastructure around pipelines: storage, compute, and logging.
- Proficiency with Git and code review workflows.
- Regular use of AI tools in daily work, with concrete examples of automation.
- Disciplined and detail-oriented; willing to respond to critical issues outside office hours.
Nice-to-have:
- Experience with Kafka/CDC streaming.
- Familiarity with Trino, Iceberg, Hive/HDFS, and S3.
- Experience with Scala.
- Experience working with audited or financial data.
- Experience with game data.
