Design, develop, and maintain data engineering solutions across the SDLC using Python (primary) and Scala (secondary). Build distributed data pipelines on Hadoop/Spark ecosystems, extend data processing platforms (Spark SQL, Hive, Starburst), manage CI/CD, and deploy on Kubernetes/OpenShift. Collaborate with stakeholders in Agile teams and ensure maintainable, reusable code.
Qualifications:
- Strong communication skills
- Experience of Agile development and scrums
- Banking and securities domain knowledge would be an added advantage
Skills Required:
- Strong experience working across the entire SDLC lifecycle
- Programming experience in one or more application or systems languages, Python - Primary, Scala - Secondary (basic knowledge)
- Strong experience working with Python concepts and libraries such as Jupyterhub, Airflow, Pandas, NumPy etc.
- Good experience with classes based OOP and design patterns
- Distributed Systems Design Experience - including understanding of distributed systems concepts and principles
- Knowledge and understanding of Kerberos and authentication
- Hadoop Ecosystem of Tools (Spark, Hive, Impala, etc).
- Experience extending and implementing core functionality and libraries in data processing platforms (Spark / Spark SQL, Hive, Starburst, etc)
- Strong experience working with the CI/CD pipeline and tools like Jenkins , Harness/Tekton, Udeploy/Ansible,Bitbucket, Jira
- Strong experience in cloud platforms like Kubernetes, OpenShift4
- Ability to deal with multiple stakeholders and follow through on open issues.
- A commitment to writing understandable, maintainable, and reusable software.
- Willingness to learn new languages and methodologies.
- Experience working with business partners and engineers to gather, understand, and bridge definitions and requirements.
- An innate desire to deliver and a strong sense of accountability for your work.
Education:
- Bachelor’s degree/University degree or equivalent experience
Photon Vancouver, British Columbia, CAN Office
900-885 W Georgia Street , , Vancouver, BC , Canada, V6C 3H1
Similar Jobs
Agency • Information Technology
Design, develop, test, and deploy high-performance Apache Spark/Scala data processing applications and ETL pipelines on Cloudera (CDH). Optimize Spark and platform performance, build data pipelines using HDFS, Hive, Impala, HBase, and Kafka, ensure data integrity and security, collaborate with data scientists and analysts, and implement version control and CI/CD for deployments.
Top Skills:
SparkCi/CdCloudera (Cdh)FlumeGitGitlabHbaseHdfsHiveImpalaJenkinsKafkaNifiOoziePostgresScalaSpark SqlSqoop
Artificial Intelligence • Enterprise Web • Software • Design • Generative AI
Lead strategy and execution for enterprise identity, access control, and governance features. Partner with go-to-market, engineering, and design to build scalable RBAC, audit, and compliance capabilities for large customers and evangelize principles across the product.
Top Skills:
AIAudit LoggingCompliance FrameworksIdentity And Access Management (Iam)Role-Based Access Control (Rbac)Webflow
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Lead enterprise commercial enablement for PC1: own proposal/SOW strategy, Salesforce and SFMC platform ownership, sprint-driven product delivery, AI enablement, digital budget management, and cross-functional digital project leadership to drive scalable commercial and marketing outcomes.
Top Skills:
AIMicrosoft TeamsSalesforceSalesforce Marketing CloudSAPSharepoint
What you need to know about the Vancouver Tech Scene
Raincouver, Vancity, The Big Smoke — Vancouver is known by many names, and in recent years, it has gained a reputation as a growing hub for both tech and sustainability. Renowned for its natural beauty, the city has become a magnet for professionals eager to create environmental solutions, and with an emphasis on clean technology, renewable energy and environmental innovation, it's attracted companies across various industries, all working toward a shared goal: advancing clean technology.
