Lead the technical direction and hands-on evolution of a large-scale data platform, solving complex Spark, Iceberg, Kafka, and StarRocks challenges while mentoring engineers and improving reliability.
Staff Software Engineer - Data Infrastructure
📜 Description
- Develop data infrastructure for autonomous-driving and ADAS products.
- Adapt the data ecosystem to the requirements of modern ML development.
- Build efficient pipelines to ingest data from vehicle fleets.
- Design and operate cloud-based distributed systems that process and transform large volumes of data.
- Mentor junior engineers and shape long-term product strategy.
🛠️ Requirements
- Extensive experience in Python (or another object-oriented language).
- Experience building reliable, distributed applications for Data Processing or similar areas.
- Working with RPC protocols such as gRPC/protobuf.
- Hands-on experience developing cloud applications (e.g. AWS, GCP, Azure).
- 5+ years of professional software engineering experience.
- Experience writing testable and modular code.
- Experience working in a fast-paced environment, collaborating across teams and disciplines.
- Experience designing, deploying, and maintaining distributed systems.
- Data pipelines, data platforms, workflow orchestration, batch processing.
✨ Benefits
- Health coverage
- Wellness coverage
- Dental coverage
- Vision coverage
- Pension
- Flexible vacation policy
- Family planning and care benefits
- Partner with computer vision and machine learning engineers on data and development challenges.
Full job description
Team
Our data platform team is working on accelerating autonomous driving by providing access to petabytes of data collected by our fleet of autonomous and non-autonomous vehicles. Efficient, fast and cost-effective access to data at large scale is key to tackle the hardest problems in AD/ADAS, from developing the Machine Learning (ML) models for perception and prediction of human driving patterns, to increasing the sophistication of our validation and simulation by identifying rare and interesting real-world driving situations. The data ecosystem developed by the London team is a key building block for developing and testing modern AD/ADAS products that will impact millions of customers.
Our ML and Data pipelines are built on-top of the open-source Flyte orchestration framework and are deployed to AWS. Pipeline code is written in Python. We use SQS and Kafka to automate data connections and leverage BigQuery and ElasticSearch for data storage. We believe strongly in automation and testing to ensure delivery of robust and correct systems. We are a distributed team, working in the UK and US.
Who are we looking for?
The London Data Infrastructure team is looking for engineers who are passionate about and enable the next generation of automotive software development. The right candidate will have excellent communication skills, solid coding skills, broad knowledge of software development across areas such as Data Infrastructure and Warehouses, Data Ingestion, Compute Frameworks, Observability and Build Infra.
Responsibilities:
Work on high-impact projects and innovate new solutions to problems in the self-driving space.
Work with Computer Vision and Machine Learning engineers on high-impact projects and innovate new solutions to problems in the self-driving space.
Understand the complex data requirements of modern ML development and tailor our data ecosystem to these needs.
Build efficient data pipelines for ingestion from the vehicle fleet.
Work on distributed systems that serve, process and transform large quantities of data in the cloud
Mentor junior engineers in their day to day work
Design the long term strategy for several of our products
Minimum Qualifications:
Extensive experience in Python (or another object-oriented language).
Experience building reliable, distributed applications for Data Processing or similar areas.
Working with RPC protocols such as gRPC/protobuf.
Hands-on experience developing cloud applications (e.g. AWS, GCP, Azure).
Preferred Qualifications:
5+ years of professional software engineering experience.
Experience writing testable and modular code.
Experience working in a fast-paced environment, collaborating across teams and disciplines.
Experience designing, deploying, and maintaining distributed systems.
Data pipelines, data platforms, workflow orchestration, batch processing.
Similar jobs
Search more Data Engineer jobsLead Forward Deployed Engineer - Migration
Leads customer migration engagements by setting technical direction and architecture, building Snowflake data solutions, and guiding teams through modernization, optimization, and AI initiatives.
Staff Software Engineer, Infrastructure Data Engineering
Build and operate reliable infrastructure data products—including pipelines, models, and services—to support usage and cost insights, planning, attribution, and budgets while shaping technical direction across teams.
Data Engineering Lead
Leads data engineering delivery and technical teams, designing and operating scalable Databricks-based platforms while advising enterprise customers on architecture, modernization, and solution design.
Leads the technical direction of a cloud-based data platform, scaling ingestion, governance, and machine learning infrastructure while delivering cross-functional initiatives and mentoring engineers.
