Staff Software Engineer, Model Infrastructure
As a Staff Software Engineer on the Model Infrastructure team, you'll lead the design and development of systems that ensure high availability and operational excellence for AI inference at Harvey.
ngrok is an all-in-one cloud networking platform that secures, transforms, and routes traffic to services running anywhere. Instead of cobbling together nginx, NLBs, VPNs, model routers, and oodles of other tools, developers solve every networking problem with one gateway. Doesn’t matter if they’re sharing localhost or running AI workloads in production.
We're trusted by more than 9 million developers at companies like GitHub, Okta, HashiCorp, and Twilio. What started as a way to put your local app on a public URL has grown into a universal gateway for API delivery, AI inference, device fleets, and site-to-site connectivity. It’s the same ngrok that millions of developers have loved and leaned on every day for years, now with the power to run production traffic at scale.
A few things you should know:
Still reading? Good. There's more below worth your time.
Our AI Gateway team builds the systems that define how AI traffic is identified, controlled, and understood as it passes through ngrok.
We own the AI-specific control plane at the gateway layer: policies, usage tracking, and enforcement that sit directly on live customer traffic. Our systems must behave correctly under real-world conditions—traffic spikes, unexpected model behavior, misconfigured policies, and customers asking, “Why was this blocked?” or “Where did my tokens go?”
Extra credit if you’ve worked on:
ngrok runs entirely on AWS. Engineers develop by using remote development tools and/or ssh to connect to remote EC2 environments that run a full Kubernetes cluster of the ngrok stack, closely mirroring production. The codebase is primarily Go and TypeScript. We use Postgres for persistence, Kafka for streaming, Protobuf for service boundaries, and Kubernetes, Terraform, Helm, and Buildkite to operate and ship reliably. React is used for user interfaces, and GitHub supports our development workflows and remembers everything.
This is a remote position for candidates outside of the Bay Area and a hybrid role for candidates within commuting distance to San Francisco. Our Bay Area employees commute to the office on Tuesdays and Wednesdays.
All candidates must be US-based, and legally authorized to work in the United States.
At this time, ngrok is unable to provide visa sponsorship for this position. Applicants must be authorized to work in the United States on a permanent, ongoing basis without the need for current or future sponsorship.
Senior Software Engineer
Software Engineer III
Job level and actual compensation will be evaluated based on factors including, but not limited to, qualifications objectively assessed during the interview process (including skills and prior relevant experience, potential impact, and scope of role), internal equity with other team members, market data, and specific work location. We provide an attractive mix of salary and equity.
#LI-Hybrid
As a Staff Software Engineer on the Model Infrastructure team, you'll lead the design and development of systems that ensure high availability and operational excellence for AI inference at Harvey.
As a Staff Software Engineer on the Model Infrastructure team, you'll design and develop systems that ensure high availability and operational excellence for AI requests at Harvey.
As a Senior Software Engineer on the Security Rules team, you will design and maintain software systems for Cloudflare's Application Security products, leading complex projects and mentoring fellow engineers.
As a Staff Production Engineer, you will own high-risk technical domains, write production software, and collaborate closely with product teams to enhance reliability at scale.
As a Senior Software Engineer for Quicksilver, you will enhance a globally distributed key-value store, tackling complex challenges while ensuring high performance and reliability.