Lead the design, training, evaluation, and deployment of large-scale machine learning models, owning training systems and technical decisions from model architecture through production.
Senior / Staff ML Training Optimization Engineer
📜 Description
- Build standardized distributed training frameworks for research and production, improving stability and efficiency.
- Profile model runtime and memory to identify performance bottlenecks.
- Evaluate technologies for training and inference, including CUDA kernels, quantization-aware methods, and compilation and deployment techniques.
- Collaborate with researchers and ML engineers on best practices for resource usage.
- Create and improve tools and dashboards to support broad adoption.
🛠️ Requirements
- MS/PhD or Bachelors degree with a minimum of 4 years of industry experience in Computer Science, Robotics and/or similar technical field(s) of study.
- Solid coding proficiency in a variety of coding languages including Python, C++ or Rust.
- Experience in deep learning frameworks such as PyTorch or Jax.
- Skilled in profiling CPU and GPU code using tools such as PyTorch Profiler and NVIDIA Nsight.
- Open-minded and collaborative team player with willingness to help others.
- Passionate about self-driving technologies, solving hard problems, and creating innovative solutions.
- Experience in identifying when custom CUDA kernels are needed, and implementing them.
- Experience in Bazel in a monorepo environment, and integrating third party packages into dev environments.
- Experience with Kubernetes-based training platforms.
✨ Benefits
- Competitive compensation
- Equity awards
- Medical coverage
- Dental coverage
- Vision coverage
- Unlimited vacation
- Flexible hours and work-from-home support
- Daily drinks, snacks, and catered meals when in office
Full job description
- Build standardized distributed training frameworks for research and production, drive our training towards new levels of stability and efficiency.
- Comprehensively profile model runtime and memory to pinpoint performance bottlenecks.
- Identify and evaluate emerging technologies that can be adopted into Waabi’s training and inference frameworks. Examples include designing new CUDA kernels, quantization-aware training and inference, and compilation/deployment techniques.
- Work with researchers and ML engineers on best-practices for optimal resource usage.
- Create and improve tooling and dashboards to ensure broad adoption of your work.
- MS/PhD or Bachelors degree with a minimum of 4 years of industry experience in Computer Science, Robotics and/or similar technical field(s) of study.
- Solid coding proficiency in a variety of coding languages including Python, C++ or Rust.
- Experience in deep learning frameworks such as PyTorch or Jax.
- Skilled in profiling CPU and GPU code using tools such as PyTorch Profiler and NVIDIA Nsight.
- Open-minded and collaborative team player with willingness to help others.
- Passionate about self-driving technologies, solving hard problems, and creating innovative solutions.
- Experience in identifying when custom CUDA kernels are needed, and implementing them.
- Experience in Bazel in a monorepo environment, and integrating third party packages into dev environments.
- Experience with Kubernetes-based training platforms.
Similar jobs
Search more Machine Learning Engineer jobsSenior Machine Learning Engineer - Applied AI & LLMs (x/f/m)
Build and lead production-scale AI systems for patient-facing healthcare products, developing retrieval pipelines, LLM and VLM applications, and agentic systems while guiding technical standards and product decisions.
Build and productionize machine-learning models that analyze ad visuals, audio, and text across languages, improving brand safety, moderation, creative generation, and advertising outcomes at scale.
Lead the development and deployment of production-grade machine learning systems, shaping architecture and engineering standards while advising clients and mentoring engineers.
Design, research, and deploy machine learning systems for real-time data security, improving NLP and LLM capabilities while building scalable MLOps and production services.
