🕒 2 days ago
Own the full lifecycle of data products, from analysis and design through deployment, while building PySpark programs and aligning technical solutions with evolving business needs.
Senior Data Developer, Brazil
🔥 22 hours ago
DatabricksPysparkDelta LakeUnity Catalog
📜 Description
- Build Databricks ingestion pipelines using bronze, silver, and gold medallion layers, Auto Loader, and Delta Lake.
- Develop before-and-after reconciliation pipelines with record counts, field comparisons, version mapping, and validation rules.
- Model gold-layer reporting for approval and rejection rates, outcome reasons, and cluster- and object-level views.
- Expose results to a web application through an API and document its contract.
- Configure Unity Catalog permissions, lineage, and auditing.
- Orchestrate workflows and jobs, add data quality checks, and manage version control with Databricks Asset Bundles.
🛠️ Requirements
- Data engineering experience, including at least 2 years using Databricks, PySpark, Delta Lake, Unity Catalog, and Workflows.
- Advanced SQL and Python proficiency.
- Experience with data reconciliation or validation in migration projects.
- Knowledge of SAP data structures.
- Experience developing reports from system data.
- Advanced English conversation skills.
✨ Benefits
- Health and dental insurance
- Meal and food allowance
- Childcare assistance
- Extended paternity leave
- Partnership with gyms and health and wellness professionals via Wellhub (Gympass) TotalPass;
- Profit Sharing and Results Participation (PLR);
- Life insurance
- Continuous learning platform (CI&T University);
Full job description
At CI&T, we help large enterprises transform the potential of AI into real business impact with AI Deployment, AI-native execution, and tech-integrated business solutions.
With 30 years of experience in technological transformation, we accelerate innovation with expertise in Agentic SDLC, Application modernization, Data & AI, Martech and Business strategy.
We are 8,000 CI&Ters across more than 25 countries, collaborating to build solutions with real impact. AI is already part of how we work, evolve, and innovate every day.
Responsibilities:
- Build ingestion pipelines for Databricks using a medallion architecture (bronze, silver, and gold layers), leveraging Auto Loader and Delta Lake.
- Implement "before-and-after" reconciliation pipelines, including record counts, field-by-field comparisons, key/table mapping between versions, and large-scale validation rule application.
- Model the gold layer for reporting, covering approval/rejection rates, reasons for outcomes, and views segmented by cluster and object.
- Expose results via an API for a web application (using Databricks SQL Warehouse with the Statement Execution API, or a custom-built service) and document the API contract.
- Configure governance within Unity Catalog—including permissions, data lineage, and auditing—which are critical for client production data.
- Orchestrate processes using Workflows/Jobs, incorporate data quality checks (such as DLT expectations or similar), and manage version control via Databricks Asset Bundles.
Requirements:
- Experience in data engineering, with at least 2 years specifically using Databricks (PySpark, Delta Lake, Unity Catalog, Workflows).
- Advanced proficiency in SQL and Python.
- Experience with data reconciliation or validation within migration projects.
- Knowledge of SAP data structures.
- Experience in developing reports based on system data.
- Advanced English for conversation.
#LI-TO1
At CI&T, we help large enterprises transform the potential of AI into real business impact with AI Deployment, AI-native execution, and tech-integrated business solutions.
With 30 years of experience in technological transformation, we accelerate innovation with expertise in Agentic SDLC, Application modernization, Data & AI, Martech and Business strategy.
We are 8,000 CI&Ters across more than 25 countries, collaborating to build solutions with real impact. AI is already part of how we work, evolve, and innovate every day.
Our benefits:
-Health and dental insurance
-Meal and food allowance
-Childcare assistance
-Extended paternity leave
-Partnership with gyms and health and wellness professionals via Wellhub (Gympass) TotalPass;
-Profit Sharing and Results Participation (PLR);
-Life insurance
-Continuous learning platform (CI&T University);
-Discount club
-Free online platform dedicated to physical, mental, and overall well-being
-Pregnancy and responsible parenting course
-Partnerships with online learning platforms
-Language learning platform
And many more!
More details about our benefits here: https://ciandt.com/br/pt-br/carreiras
At CI&T, inclusion starts at the first contact. If you are a person with a disability, it is important to present your assessment during the selection process. See which data needs to be included in the report by clicking here.This way, we can ensure the support and accommodations that you deserve. If you do not yet have the assessment, don't worry: we can support you in obtaining it.
We have a dedicated Health and Well-being team, inclusion specialists, and affinity groups who will be with you at every stage. Count on us to make this journey side by side.Similar jobs
Search more Data Engineer jobs
🕒 2 days ago
Drive Databricks adoption and technical standards while designing and optimizing scalable data pipelines, strengthening governance and observability, and guiding engineers across data engineering initiatives.
🕒 3 days ago
As a Senior Data Engineer, you will build, enhance, and maintain our data platform on Databricks, transforming large datasets into clean, usable data products.
🕒 yesterday
As a Senior Data Engineer, you will design and implement scalable data pipelines, ensuring data governance and supporting Headway's mission to improve access to mental healthcare.
Engenheiro de Dados e Plataforma de Agentes Sênior (AWS)
🕒 yesterday
Leega Consultoria👥 201 - 500 employees🏢 Information Technology And Services
🕒 yesterday
Engenheiro de Dados Sênior responsável por construir pipelines e governar dados, ferramentas e memória para uma plataforma de IA generativa multiagente em AWS, com foco em segurança, observabilidade e resiliência.