🔥 22 hours ago
Own the architecture, ingestion, transformation, and production operations of a multi-source data platform, delivering reliable, current company intelligence through event-driven and batch workflows.
Senior Data Engineer
🕒 yesterday
Data EngineeringAirflowDbtGoogle Cloud Platform
📜 Description
- Design a multi-source platform for concurrent ingestion, event-driven processing, and orchestrated batch workflows.
- Build ingestion from provider APIs, webhooks, files, and web scraping, with rate limits and robust failure handling.
- Orchestrate scheduled workflows and historical backfills using Airflow, Dagster, or Prefect.
- Develop event-driven systems with messaging or task queues, including retries, dead-letter queues, and controlled replay.
- Store original payloads in cloud object storage and transform them with dbt into raw, standardized, and curated layers.
- Implement data quality controls and optimize PostgreSQL schemas and loading patterns for downstream use.
🛠️ Requirements
- At least 5 years building and operating production data platforms across multiple sources; 8 or more years is preferred.
- Production experience with workflow orchestration, especially Airflow, and hands-on experience with dbt and Google Cloud Platform.
- Experience building event-driven systems and asynchronous processing with messaging or task queues.
- Strong SQL, PostgreSQL, and Python skills, including data modeling, incremental loading, asynchronous HTTP, and concurrent processing with tools such as asyncio, aiohttp, or HTTPX.
- Experience with web scraping, including rate limiting and failure handling, is valuable.
- Entity resolution, data enrichment, BigQuery, Dataform, or language model-based structured extraction experience is a plus.
- Comfort working with broad ownership and changing priorities in a startup or small engineering team.
✨ Benefits
- Visa sponsorship is not available.
- Location
Full job description
About the Role
Build and operate a data platform that turns information from multiple sources into reliable company intelligence. You will own the platform end to end, from architecture and ingestion through production operations, supporting products that depend on trusted, up-to-date records.
What You'll Do
Design a multi-source platform for concurrent ingestion, event-driven processing, and orchestrated batch workflows.
Build ingestion for provider APIs, webhooks, files, and web scraping, with rate limits and robust failure handling.
Orchestrate scheduled workflows and historical backfills using tools such as Airflow, Dagster, or Prefect.
Develop event-driven systems with messaging or task queues, including retries, dead-letter queues, and controlled replay.
Store original payloads in cloud object storage and transform them with dbt into raw, standardized, and curated layers.
Implement idempotent upserts, safe retries, data contracts, automated validation, pipeline tests, and monitoring.
Design and optimize PostgreSQL schemas and loading patterns for downstream use.
Integrate data enrichment and structured extraction using language models.
What We're Looking For
At least 5 years building and operating production data platforms across multiple sources; 8 or more years is preferred.
Production experience with workflow orchestration, especially Airflow, and hands-on experience with dbt and Google Cloud Platform.
Experience building event-driven systems and asynchronous processing with messaging or task queues.
Strong SQL, PostgreSQL, and Python skills, including data modeling, incremental loading, asynchronous HTTP, and concurrent processing with tools such as asyncio, aiohttp, or HTTPX.
Experience with web scraping, including rate limiting and failure handling, is valuable.
Entity resolution, data enrichment, BigQuery, Dataform, or language model-based structured extraction experience is a plus.
Comfort working with broad ownership and changing priorities in a startup or small engineering team.
Compensation & Benefits
Visa sponsorship is not available.
Location
Remote.
Similar jobs
Search more Data Engineer jobs🔥 22 hours ago
Own the architecture, ingestion, transformation, and production operations of a multi-source data platform, delivering reliable, validated data for downstream products and company intelligence.
🕒 3 days ago
Scale the data platform by building reliable pipelines, improving data models and quality, and enabling lower-latency analytics while shaping engineering practices as the team grows.
🕒 2 days ago
Build and scale reliable data pipelines, establish dbt standards and quality controls, and create self-service tools and AI agents that help teams access and use data independently.
🕒 5 days ago
Build customer-facing data products for the single-family home market, architect reliable data models and workflows, and shape the data platform’s architecture and engineering standards.