🕒 5 days ago
Desenvolve e mantém pipelines de dados em AWS, Databricks e Cloudera, otimiza processamento distribuído com Spark e apoia áreas de negócio na implementação de soluções de dados.
Senior Data Engineer
📅 Jul 28
DatabricksApache SparkPythonSQL
📜 Description
- Architect and deliver enterprise data platforms from project inception through production deployment.
- Own technical delivery across client engagements and define solution architectures, roadmaps, and implementation strategies.
- Lead client discovery workshops, architecture sessions, and technical planning; present recommendations to stakeholders and executives.
- Develop scalable Databricks pipelines and Lakehouse solutions for batch and real-time data processing.
- Establish engineering practices through architecture and code reviews, and mentor Data Engineers.
- Build secure, reliable cloud platforms with governance, CI/CD, infrastructure automation, and operational monitoring.
🛠️ Requirements
- Bachelor's or Master's degree in Computer Science, Data Engineering, Software Engineering, or a related technical discipline.
- 8+ years of experience designing and building enterprise-scale data platforms.
- 5+ years of hands-on experience with Databricks and Apache Spark.
- Proven experience leading enterprise data engineering projects from architecture through production delivery.
- Strong expertise in Python, SQL, and Spark for large-scale data processing.
- Deep understanding of Delta Lake, Lakehouse architecture, and modern data platform design.
- Experience working with AWS, Azure, or Google Cloud Platform.
- Strong knowledge of ETL/ELT frameworks, distributed computing, and data modeling.
- Experience implementing CI/CD pipelines and Infrastructure-as-Code.
- Strong understanding of data governance, security, metadata management, and data quality practices.
Full job description
About ShyftLabs
At ShyftLabs, we live and breathe data. Since 2020, we’ve been helping Fortune 500 companies unlock growth with cutting-edge digital solutions that transform industries and create measurable business impact. We’re growing fast and we’re looking for passionate problem-solvers who are ready to turn big ideas into real outcomes.
The Team
The Customer Data Platform team builds the systems that turn first-party data into unified customer profiles and audiences that marketers can act on. Our platform processes customer, event and transaction data from enterprise warehouses, resolves identities, and delivers audiences to the major advertising and engagement platforms. The work directly shapes how leading retailers and brands reach their customers, and every decision must respect privacy.
The Opportunity
As a Senior Data Engineer, you will own the core data pipelines of the platform from end to end. You will set technical direction for how we ingest, model and activate customer data at scale on Snowflake and BigQuery. You will partner with product, engineering and client teams to turn business needs into reliable, cost-efficient data systems.
What You'll Be Doing
Design, build and own batch and incremental pipelines that process millions of customer profiles and billions of events per day across Snowflake and BigQuery.
Define the data models for unified customer profiles, identity resolution and audiences, and keep them correct, fast and cost-efficient.
Lead the design of activation pipelines to external advertising and engagement platforms, with strong guarantees for retries, incremental syncs and audience accuracy.
Own data quality end to end: data contracts, automated tests, freshness SLAs, monitoring and incident response.
Build privacy and governance into the platform: PII protection, consent, deletion requests and multi-tenant data isolation.
Raise the engineering bar through design docs, code reviews and mentorship, and turn client and business needs into a clear technical roadmap.
What You'll Bring
Bachelor’s degree in Computer Science, a related technical field, or equivalent practical experience.
5+ years of experience in building and running production data pipelines.
Experience with advanced SQL and data modelling on Snowflake, BigQuery or a similar cloud data warehouse.
Experience in programming with Python or a similar language.
Strong knowledge of ELT patterns on Snowflake (Snowpipe, Streams and Tasks, dbt) and of data modelling for large warehouse tables.
Nice to Have
Experience with customer data platforms, identity resolution or data activation in marketing or retail.
Experience with query optimization and cost management on Snowflake and BigQuery at large scale.
Experience with workflow orchestration (e.g., Airflow, Dagster, Temporal) and streaming systems (e.g., Kafka, Pub/Sub).
Experience with LLM-based tools or agents that work over structured data.
Knowledge of privacy regulations (PIPEDA, GDPR, CCPA) and privacy-by-design practices.
Experience with cloud infrastructure on AWS or GCP, with Docker, Kubernetes and Terraform.
A record of technical leadership and mentorship.
Strong understanding of business intelligence and reporting tools (Power BI, Tableau, Looker).
Salary Range
Similar jobs
Search more Data Engineer jobs
Senior Data Engineer – Informatica CDI & SAS Migration
🕒 3 days ago
NucleusTeq👥 201 - 500 employees🏢 Information Technology And Services
🕒 3 days ago
Designs and optimizes Informatica CDI data pipelines while migrating legacy SAS workflows to cloud-native architectures, preserving business logic and validating data fidelity across modern data platforms.
🕒 2 days ago
Design and maintain ETL/ELT pipelines, modernize legacy data systems and SAS workflows, and build reliable data integrations and database structures for analytical and operational use.
Senior Data Engineer, Ads Data (AI Native)
🕒 2 days ago
Life360👥 501 - 1000 employees🏢 Individual & Family Services
🕒 2 days ago
Own and improve Life360’s Databricks ads data platform, building reliable pipelines, curated datasets, and trusted metrics while reducing runtime and cost and enabling analysts, ad strategists, and sales teams to self-serve insights.
🕒 2 days ago
Build and operate production data pipelines connecting CRM, Data Lake, and Adobe Experience Platform, while ensuring data quality, LGPD compliance, and reproducible engagement metrics.
📍 🇧🇷 Brazil - Remote💼 Full-Time🎸 Senior🛢️ Data Engineer📢 🇧🇷🇵🇹 Portuguese Required📢 🇬🇧 English Required