🕒 yesterday
Own the data platform end to end, designing and operating ingestion, event-driven processing, orchestration, and transformation systems that deliver reliable, current data to downstream products.
Senior Data Engineer
🕒 yesterday
Data EngineeringGCPDbtAirflow
📜 Description
- Design and build a multi-source platform for concurrent ingestion, event-driven processing, and batch workflows.
- Develop intake pipelines for APIs, webhooks, files, and web scraping, with rate limiting and failure handling.
- Orchestrate scheduled workflows and historical backfills using tools such as Airflow, Dagster, or Prefect.
- Build queue-based systems with retries, dead-letter queues, and controlled replay.
- Store original payloads in Google Cloud Storage and transform data with dbt into raw, standardized, and curated layers.
- Implement idempotent upserts, data contracts, validation, pipeline tests, monitoring, and optimized PostgreSQL loading; integrate enrichment providers and LLM-based extraction.
🛠️ Requirements
- At least 5 years building and operating production data platforms, with ownership spanning architecture, implementation, deployment, and ongoing operations.
- Production experience with GCP and dbt, plus workflow orchestration such as Airflow, Dagster, Prefect, or Temporal, including dependency management, retries, and recovery.
- Strong Python skills, including asynchronous HTTP and concurrent processing with tools such as asyncio, aiohttp, or HTTPX.
- Strong SQL and PostgreSQL skills in data modeling, query optimization, and reliable incremental loading.
- Experience with event-driven systems, queues, data contracts, raw and curated storage, retention policies, automated testing, and pipeline monitoring.
- Experience in a startup or small engineering team is useful. Web scraping, entity resolution, company or people data enrichment, and LLM-based extraction are also relevant.
Full job description
About the Role
Join a data infrastructure team building a platform that turns information from multiple sources into reliable, up-to-date company intelligence. You will own the platform end-to-end, from architecture and ingestion through production operations, helping power downstream products with trusted data.
What You'll Do
Design and build a multi-source data platform for concurrent ingestion, event-driven processing, and orchestrated batch workflows.
Develop intake pipelines for provider APIs, webhooks, files, and web scraping, with rate limiting and robust failure handling.
Orchestrate scheduled workflows and historical backfills using tools such as Airflow, Dagster, or Prefect.
Build event-driven systems with messaging or task queues, including retries, dead-letter queues, and controlled replay.
Preserve original payloads in Google Cloud Storage and transform data with dbt into raw, standardized, and curated layers.
Implement idempotent upserts, data contracts, automated validation, pipeline tests, and monitoring for freshness and failures.
Design and optimize PostgreSQL schemas and loading patterns, and integrate enrichment providers and LLM-based structured extraction.
What We're Looking For
At least 5 years building and operating production data platforms, with ownership spanning architecture, implementation, deployment, and ongoing operations.
Production experience with GCP and dbt, plus workflow orchestration such as Airflow, Dagster, Prefect, or Temporal, including dependency management, retries, and recovery.
Strong Python skills, including asynchronous HTTP and concurrent processing with tools such as asyncio, aiohttp, or HTTPX.
Strong SQL and PostgreSQL skills in data modeling, query optimization, and reliable incremental loading.
Experience with event-driven systems, queues, data contracts, raw and curated storage, retention policies, automated testing, and pipeline monitoring.
Experience in a startup or small engineering team is useful. Web scraping, entity resolution, company or people data enrichment, and LLM-based extraction are also relevant.
Compensation & Benefits
Annual base salary: $80,000 to $120,000 USD. Visa sponsorship is not available.
Location
Fully remote, candidates must be based in Latin America.
Similar jobs
Search more Data Engineer jobs🕒 yesterday
Build and own data pipelines, entity resolution, and pricing models that turn noisy marketplace listings into reliable comparable items and valuation signals, while monitoring data quality and partnering with computer vision teams.
🕒 yesterday
Own the architecture, ingestion, and production operations of a multi-source data platform, delivering reliable, current company intelligence through orchestrated and event-driven pipelines.
🕒 2 days ago
Build customer-facing data products for the single-family home market, architect reliable data models and workflows, and shape the data platform’s architecture and engineering standards.
🕒 2 days ago
As a Senior Data Engineer, you will design and implement AI-optimized data platforms and collaborate with clients to solve complex data engineering challenges across various industries.