Build and govern Databricks data pipelines for migration reconciliation and reporting, expose validated results through an API, and orchestrate production workflows with data quality controls.
Software Engineer (SE / Sr SE), Data Infrastructure
📜 Description
- Design high-throughput C++ systems for recording vehicle sensor and runtime data, capturing events, and collecting telemetry within resource constraints.
- Build Python services and libraries for data transfer, ingestion, validation, format conversion, metadata extraction, cataloging, and storage lifecycle management.
- Evolve versioned vehicle-data formats and C++/Python APIs, including indexing, serialization, compression, schema evolution, and backward compatibility.
- Improve columnar and lakehouse data representations and access APIs for fleet-scale mining, replay, and analytics.
- Follow Quality Management System requirements and contribute to continuous improvement.
🛠️ Requirements
- Programming experience in modern C++ and working proficiency in Python for services, libraries, and tooling
✨ Benefits
- Package, including 401(k) plan.
- BS, MS, or PhD in Computer Science or a related technical field, or equivalent practical experience
- Linux systems knowledge, including concurrency, memory and resource management, filesystem and storage I/O, networking, debugging
- Self-driven with a strong sense of ownership: a quick learner who is eager to take responsibility and drive projects forward end to end
- Familiarity with protobuf, gRPC, compression or media codecs, and native C++/Python bindings
- Experience with autonomous vehicles, robotics, ROS or ROS 2, robotics log formats such as rosbag or MCAP
- Experience with columnar data formats or lakehouse table technologies such as Parquet, Delta Lake, or Apache Iceberg
Full job description
Our autonomous-driving fleet generates petabyte-scale sensor and system data. In this role, you will own critical parts of the path that makes this data reliable and useful—from high-throughput recording on the vehicle to transfer, validation, cataloging, and efficient access for replay, analytics, and machine learning. You will work primarily in modern C++ and Python, solving systems challenges across resource-constrained vehicle computers and large-scale offline infrastructure. We welcome engineers from systems, robotics, storage, or data-infrastructure backgrounds; prior autonomous-vehicle experience is helpful but not required.
We are open to candidates at either the Software Engineer or Senior Software Engineer level. Level will be determined by experience, technical depth, scope of ownership, and demonstrated impact. You do not need experience with every technology in our stack; we value strong fundamentals, ownership, and the ability to learn.
Responsibilities:
-
Design and evolve high-throughput C++ systems that continuously record vehicle sensor and runtime data, selectively capture events, and collect system telemetry while operating safely under CPU, memory, disk, and network constraints
-
Build reliable Python data-plane services and libraries for vehicle-data transfer, ingestion, validation, domain-specific format conversion, metadata extraction, cataloging, and storage lifecycle management
-
Evolve versioned vehicle-data formats and their C++ and Python APIs, including indexing, serialization, compression, schema evolution, and backward compatibility
-
Evolve columnar and lakehouse representations and access APIs for vehicle data, improving scalability and performance for fleet-scale mining, replay, and analytics
-
Ensure that your work is performed in accordance with the company’s Quality Management System (QMS) requirements and contribute to continuous improvement efforts
Required Skills:
BS, MS, or PhD in Computer Science or a related technical field, or equivalent practical experience
Programming experience in modern C++ and working proficiency in Python for services, libraries, and tooling
Linux systems knowledge, including concurrency, memory and resource management, filesystem and storage I/O, networking, debugging, and performance profiling
Self-driven with a strong sense of ownership: a quick learner who is eager to take responsibility and drive projects forward end to end
Preferred Skills:
Familiarity with protobuf, gRPC, compression or media codecs, and native C++/Python bindings
Experience with autonomous vehicles, robotics, ROS or ROS 2, robotics log formats such as rosbag or MCAP, or other high-throughput recording and telemetry systems
Experience with columnar data formats or lakehouse table technologies such as Parquet, Delta Lake, or Apache Iceberg
Similar jobs
Search more Data Engineer jobsAs a Senior Data Engineer, you will design and implement scalable data pipelines, ensuring data governance and supporting Headway's mission to improve access to mental healthcare.
Engenheiro de Dados e Plataforma de Agentes Sênior (AWS)
Engenheiro de Dados Sênior responsável por construir pipelines e governar dados, ferramentas e memória para uma plataforma de IA generativa multiagente em AWS, com foco em segurança, observabilidade e resiliência.
Build and maintain real-time, batch, and streaming data pipelines, warehouse infrastructure, and dbt models while supporting analytics and data quality across business teams.
Build and maintain real-time, batch, and streaming data pipelines and warehouse infrastructure, while supporting analytics and developing dbt models that improve data accessibility, quality, and decision-making.
