Data Platform Engineer 🔥
We are looking for a Devops Engineer with a focus towards data platforms to join the Data Team at 1Password.
As a Data Platform Engineer, you will work with other engineers and technical resources from cross-functional teams to build a fully automated Data infrastructure that powers business automation, data science & analytics functions.
You have a passion for designing, implementing, and operating stable, scalable, and efficient solutions to flow data from production systems into the data lake, and from the lake to various supporting tools. You are curious about all the nuances in the source data systems & always make an effort to assert data quality checks and improve documentation.
This is a Remote opportunity based in Canada.
What we're looking for:
- Deep knowledge of data-relevant AWS services such as RDS, S3, Kinesis, Glue, Redshift & Athena
- Proficiency with Terraform
- Experience setting up containerized applications with Kubernetes
- Experience setting up and managing Apache Spark clusters and programming in spark
- Experience with DAG management frameworks, e.g. Airflow or Luigi.
- Experience running & monitoring batch & micro-batch workloads in production
- Proficiency in SQL & modern MPP data platforms, e.g. redshift, snowflake, prestodb (We use AWS Athena)
- Proficiency in programming with Python & Scala
- Ability and willingness to learn new technologies quickly.
- Proficiency with VSC Systems (we use Git), setting up & maintaining CI pipelines (we use gitlab)
Bonus points if you have:
- Experience setting up change data capture & streaming analytics.
- Experience integrating data to/from SaaS tools such as Salesforce, Marketo, etc.
- Experience with DBT
What you can expect:
- Build out and maintain our Data Platform on AWS using infrastructure-as-code (Terraform)
- Set up the components of our data platform: AWS S3, AWS Aurora, AWS Redshift, AWS Athena, Kubernetes, AWS Kinesis, Apache Airflow, Spark
- Set up change data capture from our application database to our data platform.
- Set up and manage internal and external API Gateways, load balancers and container orchestration on Kubernetes
- Build a streaming data pipeline connecting our various SaaS tools with our data platform & other internal API endpoints.
- Define and deploy a logging, monitoring and alerting infrastructure using AWS CloudWatch, DataDog & Elasticsearch