Design and develop big data processing engines with a focus on ETL pipeline, including complex logic and geo-location analysis.
Description
We are looking for a Big Data / Data Engineer to join our growing team. In this role, you will be responsible for the design and development of our big data processing engine a heavy ETL pipeline handling complex logic including geo-location analysis, running across both cloud and on-prem environments.
Requirements:
- 4+ years of experience in Data Engineering
- 2+ years of hands-on experience with Scala (must) or Java (optional)
- 2+ years of experience with SQL and NoSQL DBs
- Hands-on experience with Spark & PySpark (must)
- Experience with Airflow (must)
- Experience with AWS and/or GCP
- Experience with Kubernetes (k8s) and on-prem deployments
- Solid understanding of distributed systems design, pipeline architecture, and partitioning strategies
- Deep understanding of Agile principles, practices and values
- Solid understanding of design patterns, software development techniques, and clean code practices
Advantage:
- Experience with Apache Iceberg, Helm Charts, or data catalog tools
- Experience with Spring Framework
- Experience with microservice architecture
- Experience working with DevOps and CI/CD practices
- Customer-facing experience or comfort working directly with clients
- Bachelor's, BSc or Master's degree in Computer Science or Engineering
Similar Jobs
Digital Media • Gaming • Information Technology • Software • Sports • Esports • Big Data Analytics
Lead the design and execution of performance testing strategies, develop scalable testing frameworks, and mentor engineering teams while ensuring system resilience and reliability.
Top Skills:
C#GatlingGoJavaJmeterK6LocustPython
Aerospace • Information Technology • Software • Cybersecurity • Design • Defense • Manufacturing
Responsible for developing avionics systems, creating analytical models, performing validation and verification, and collaborating with cross-functional teams, while ensuring compliance with system requirements.
Top Skills:
Arp4754AIbm DoorsMatlabMbseSimulinkSysml
Aerospace • Information Technology • Software • Cybersecurity • Design • Defense • Manufacturing
The Manufacturing Engineer supports fabrication processes, resolves technical issues, ensures quality, collaborates with stakeholders, and mentors others while applying Lean principles.
Top Skills:
CappCatiaCdsEnoviaErpIvtPdm
What you need to know about the Vancouver Tech Scene
Raincouver, Vancity, The Big Smoke — Vancouver is known by many names, and in recent years, it has gained a reputation as a growing hub for both tech and sustainability. Renowned for its natural beauty, the city has become a magnet for professionals eager to create environmental solutions, and with an emphasis on clean technology, renewable energy and environmental innovation, it's attracted companies across various industries, all working toward a shared goal: advancing clean technology.
