Data Engineer – HIRING ASAPStart date: ASAP
Duration: 12 Months
Location: Hybrid Working in London officeRate: £700-£768 per day inside IR35SummaryThe International Engineering team is responsible for building and running the UK and Canadian apps. As a Data Engineer, you will own engineering efforts to provide partner teams within the organisation with data sources that build our AI/ML user experience and help run our client robustly and at scale.Responsibilities:
Build and maintain scalable data pipelines for BigQuery using Cloud Composer and Airflow;
Provide observability and monitoring using Monte Carlo and Looker as well as operational tools such as NewRelic and Splunk, driving reliability, data quality, data quality and robustness;
Build critical data quality checks to ensure that the data we process and provide to partner teams meets security and compliance requirements;
Participate in code and technical design reviews;
Scope and plan work together with product engineering, data science and the business; support other teams for all tasks related to data tracking, collection, ingestion, quality monitoring and alerting.
Skills:
3+ years of experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt;
Working knowledge of Python, SQL and shell scripting;
Experience with Docker/k8s and “infrastructure as a code” tools such as Terraform, Salt or similar;
3+ years of significant professional experience developing applications written in a modern programming environment – preferably Scala/Java and/or Python;
Professional experience across the entire software development life cycle, from gathering requirements to production deployment using CI & CD;
Confident at articulating complex technical content to peers and partners across the organization via Technical Design Documents.
Bonus Skills:
Experience optimising complex OLAP queries in a DW system such BigQuery, SnowFlake, Druid, etc;
Experience with Looker;
Experience with the MLOps life cycle;
DevOps experience with Google Cloud Platform, AWS, or Azure;
Data quality experience especially using & improving internal tools to automatically detect issues.
Expected salary
£700 – 768 per day
Location
London
Job date
Wed, 11 Sep 2024 03:53:21 GMT
To help us track our recruitment effort, please indicate in your email/cover letter where (hiring-jobs.com) you saw this job posting.